The rugnux live analysis of a serial collection took min(receiver_threads,
16 per card), so a site running 16 receiver threads on four cards analysed
on four threads a card. It now runs 16 a card (fewer where a card's memory
does not hold them), as offline rugnux does. Measured on a 10.7 Mpx stills
replay: 8 a card saturates, 16 is the fastest, 32 and 64 are 4-9 % slower.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The vivado/vitis_hls detection, the hls target and the pcie_100g/pcie_8x10g
(and _bd) targets are removed: the firmware is not built from this tree.
docs/FPGA.md keeps the toolset version and the IP licence terms of the
released bitstreams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
As the owner decided: no capping. Where min(stills_prepass_images, images)
is more than half of the image buffer's slots, JFJochReceiverService::Start
throws InputParameterInvalid before the stills receiver is built - and so
before the detector is armed (JFJochServices starts the receiver first). A
synchronous /start answers 400, an asynchronous one reports the error through
/wait_till_done, and the broker goes back to Idle. The message gives the
slots, the slot size, N, the image buffer it needs (2 x N x slot, as
image_buffer_MiB) and the stills_prepass_images that would fit.
CheckStillsPrePassFits (StillsGPUAnalysis.h) is shared with
jfjoch_stills_gpu_replay, which exits with the same message.
The capping in JFJochReceiverRugnuxStills::OnStart and its "the pre-pass
runs on the first K images" note are removed. JFJOCH_BROKER.md and the
stills_prepass_images description say the collection is refused (C++ model,
Python docs and frontend types regenerated).
Tests: JFJochReceiverRugnux_Stills_PrePassDoesNotFit (7 of a 12-slot buffer:
refused, InputParameterInvalid, nothing sent); BufferFullDuringPrePass keeps
200 frames through a 12-slot buffer with a pre-pass of 6, all written;
CancelDuringRun takes a pre-pass of 2 for its 4 slots;
GPUStills_GeometryRefinedOnModules and GPUStills_cancel_during_prepass get an
image buffer of 2 x N slots. Replay: -b 30000 at -P 1000 is refused; with
2020 slots per-frame results as before on both windows checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The live merge's per-image merged vector (rotation: the rugnux run's
MergeStatistics::image_merged; stills: the final StillsLiveMerge) is
put into the receiver plots as the new plot type image_merged and into
the end message, so the broker's master file carries imageMerged and
jfjoch_viewer on a broker fills its image list's Merged column and
filter through /preview/plot (empty, so N/A, until the merge is made,
and on an older broker). Frontend client and Redoc regenerated; the C++
model is unchanged (plot_type is a query parameter). The frontend plot
menu has a "Merged" entry.
--mode scale re-reads the imageIndexed of its input files, so the
report's IMAGES_INDEXED and the indexed column of _plot.txt are filled
(absent / nan for a file without it). The summary line names a lattice
count only where one was counted.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
As the owner decided: how many frames the pre-pass holds no longer depends on
their compression.
- HoldArena (and its test) is gone. A frame waiting for the pre-pass or the
analysis is serialised into a slot of the image buffer of its own and marked
ready, as before 38137e16a: it is an ordinary image-buffer entry, found by the
normal lookup (image routes, preview, jfjoch_viewer), counted in
/image_buffer/status as a sending slot and listed among its images. The slot is
discarded with the last copy of the frame, so the analysed copy is then the only
one there. The receiver's waiting map, GetWaitingImage / GetLastWaitingImage and
the receiver-service fallback are removed.
- A frame takes a slot only while more than half of the slots are free
(GetImageSlot(slots - slots/2)). Nothing is written before the pre-pass, so it
holds exactly slots/2 frames; afterwards frames waiting for the writer count
against the same half. With no slot to take the frame waits; the host never
discards one.
- At the start the pre-pass window is fitted to the buffer: where
min(stills_prepass_images, images) > slots/2, the pre-pass runs on the first
slots/2 frames, and a warning in the log and a note in /rugnux/status say so.
The "held only K of N" path (StartPrePass(full), hold_full) can no longer be
reached and is removed.
- jfjoch_stills_gpu_replay holds frames the same way and prints the same warning.
- Docs: JFJOCH_BROKER.md gives the slot formula (largest compressed image +
4 MiB) and "N frames need N slots, 2N in the buffer"; the stills_prepass_images
description says the same (C++ model, Python docs, frontend types regenerated).
Tests: JFJochReceiverRugnux_Stills_ImagesVisibleDuringPrePass now sees the held
frames as sending slots of the buffer; BufferFullDuringPrePass checks the start
note (12 slots: the pre-pass on the first 6). Replay at -P 1000 with a buffer of
2020 slots: per-frame results identical to the previous HoldArena replays on both
windows checked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The report now states how many images made it through each stage, as
counts beside the indexing rate: IMAGES_INDEXED and LATTICES_INDEXED
(section 2; a stills measure, --developer on rotation like
INDEXING_RATE), IMAGES_INTEGRATED (section 2) and IMAGES_MERGED
(section 3), plus an "Images" summary line and a console line. An image
is integrated on its main lattice only - extra lattices are indexed but
not integrated - so integrated images are also integrated lattices.
The merge records per image whether any of its observations reached
it (MergeStatistics::image_merged): on stills from MergeOnTheFly's
AddImages (past the cell form, scale and --min-image-cc), on rotation
from the disposition ledger (not rejected). It drives the merged column
of _plot.txt (which on stills used to be approximated by "has a scale"),
goes to the _process.h5 as /entry/MX/imageMerged (and through CBOR),
is read back by the reader, replaces MergedCrystals for --mode scale's
INPUT_nn rows, and --mode scale now writes a _plot.txt for its merge.
_plot.txt gains an indexed column at the end, so no column moves.
jfjoch_viewer's image list gets a Merged column and, from the Index and
Merged header context menus, Yes/No/N/A row filters that combine (e.g.
indexed but not merged); a file without imageMerged shows N/A.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The replay held one frame per image-buffer slot (an uncompressed frame's size), so a 2000-frame
pre-pass of a 9M detector needed ~180 GB. It now packs the compressed frames into at most half of
the buffer's slots as JFJochReceiverRugnuxStills does, and starts the pre-pass on what is held when
the hold is full (StartPrePass(true)); -P sets the pre-pass images directly.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
rc.184-rc.186 refined the distance, beam centre and cell on whatever strong frames the pre-pass
found (the fit's own convergence was the only gate; StillsGPUAnalysis applied the result), which is
how a header distance 2.5 % short was recovered live. The 50-strong-frame gate of f25117853 refused
that fit on low-hit windows. The gate is now 6: below it a fit took the distance 1.4 % the wrong way
on 3-4 frames and collapsed the merge; from 6 up the fit matched one on 200.
Replays (8000-frame windows, header distance 2.5 % short): in-house lysozyme N=1000, 14 strong
frames: header distance, CC1/2 0.641 at 1.81 A -> 117.86 mm, CC1/2 0.939 at 1.35 A; R3 chip start
N=250 (27 frames) CC1/2 0.795 -> 0.916, N=500 (35 frames) 0.807 -> 0.872; windows with >= 50 strong
frames identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
Live stills frames already refine their own cell and beam centre per frame, as JFJochReceiverFPGA
does (IndexAndRefine with GeomRefinementAlgorithm::BeamCenter, the cell within 5 % of the reference),
so a wrong header distance still indexes: its error goes into each frame's cell scale. What broke was
the merge: with the given cell it keeps the volume population nearest that cell
(GivenCellPopulation), and at a 2.5 % short distance a stray population near the stated volume won
over the frames refined ~7.5 % smaller - 315 of 352 frames left out, CC1/2 0.01. Where the pre-pass
did not refine the geometry, the live merge now picks the crystals as without a given cell: the
major form by shape (LeaveOutMinorCellForm), merged in the consensus cell. Where it refined, unchanged.
Replays, 8000-frame windows: in-house lysozyme (header ~7 px and 2.5 % off) N=500 CC1/2 0.007 ->
0.697 (1.75 A), N=1000 0.013 -> 0.641 (1.81 A); R3 chip start N=500 0.801 -> 0.807; refined windows
identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
f25117853 skipped DecideStillsGeometry when the pre-pass found fewer than 50 strong frames. The
centre is decided first in every case - the file's centre against the scattered background's
(walk, point-symmetry capture), file kept unless it scores below FILE_KEPT_ABOVE of the best, as
offline - and the 50-frame gate applies only to the geometry refinement: where the trial moved the
centre, the fit is asked again from it. StillsGPUAnalysis takes the beam centre the pre-pass
decided in every case and the distance and cell only where the fit ran; the note says when the
centre came from the background. The header centre stays only where the trial keeps it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The serial pre-pass runs once on the first stills_prepass_images frames (new rugnux live
setting, default 1000, 100-20000, reset to 1000 by POST /initialize) instead of the doubling
stages (500 -> ... -> 2000 frames, stop at 200 geometry frames). One decision: with a cell and at
least RUGNUX_LIVE_GEOMETRY_MIN_FRAMES (50) strong frames among them, the beam centre, distance
and cell are refined on them; otherwise the header geometry is used exactly as given - the
rugnux pre-pass then neither fits nor tries a centre (ProcessConfig::stills_geometry_min_frames),
and StillsGPUAnalysis applies beam/distance/cell only where the fit ran. The X-ray-only steps
(saturation, shadow, defective pixels, polarization, background ring, powder rings) run as before
on the start of the window; the background beam-centre estimate is reported, not applied, when
the geometry is kept. Log and live-status notes say "Geometry refined from K of N frames" or
"Fewer than 50 strong frames in the first N: header geometry used". Where the hold (at most half
of the image buffer) fills before N frames, the pre-pass runs on those held and says so. The
first N images are always analysed under stills_downsample. Rotation is unaffected.
Two fixes in the stills geometry:
- The refit (bundle indexed again at the fitted geometry) was skipped on every modular detector
whenever the modules were not being placed, i.e. always live: placing_modules now also needs
refine_modules. Offline, where the first fit places the modules and so skips its refit, the
fit is run again from the placed geometry, as after a centre the trial moved.
- The local beam-centre grid no longer adopts a chance maximum: it moves the centre only from
one that puts spots on the lattice.
OpenAPI: rugnux_settings/rugnux_status.stills_prepass_images; C++ model, frontend client and
python client docs regenerated; frontend settings field. jfjoch_stills_gpu_replay and
jfjoch_fpga_stills_test take -P <images>.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The "not yet" list named a rotamer search, which the side-chain field does;
the Metal path runs four refinement kernels, not only the density gather;
the ModelScaling.h comment gave k_sol up to 0.8 where the box stops at 0.6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
A new §15.13 of the refinement reference tabulates the free set over the whole run: how it is obtained
(drawn by the merge, inherited from -z, taken from --free-flags, read by --mode refine, closed under the
twin domains, carried into another group), what is fitted on the working reflections and what on the free
ones, every decision the free set takes (model validation, molecular replacement, the refinement's weights,
NCS sigma, displacement model, model as given, twin law) with the rule each applies, what is decided
without it, what only reports it, and how small free sets are met. §10.7, §14.3 and the model/reference,
report and CLI pages link to it instead of repeating it.
Corrected to the code as it is: the placement stage (rigid bodies, B per residue) and the solvent build's
rounds are kept on the work-set target, not R-free (overview, REFINE_R_FREE_PLACEMENT,
REFINE_SOLVENT_BUILD_ROUNDS); REFINE_AS_GIVEN states its actual rule; §15.7's "and nothing else" now names
the twin-law decision; §14.3 names the twin-hypothesis map choice and molecular replacement among the
free-set selections; --free-flags is documented as applied only where the rigid body placed the model.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
fpga/hls stays in the tree as the source of the firmware, but CMake no longer compiles it: the CPU
library JFJochHLSCores (built with -DJFJOCH_HLS_NOSYNTH, linked by nothing since the HLS simulation
went) is gone. Only the vitis_hls synthesis targets remain (hls, pcie_100g, pcie_8x10g, *_bd), defined
only when vivado and vitis_hls are on PATH; synth_hls_function.tcl passes no include path, so Vitis
uses its own ap_int.h/hls_stream.h/hls_burst_maxi.h.
Deleted: fpga/include (the Xilinx ap_* arbitrary-precision headers, etc/ap_private.h, the modified
hls_burst_maxi.h simulation model whose licence terms were never established, and LICENSE.HLS_Headers)
and fpga/hls/parallel_stream.h (CPU stand-in adapted from Vitis hls_stream.h). The dead
JFJOCH_HLS_NOSYNTH branches in hls_jfjoch.h, jf_conversion.cpp, spot_finder.cpp and host_writer.cpp go
with them, so the reference sources show the synthesised path only.
Licences: the three Xilinx HLS rows leave THIRD_PARTY_NOTICES.md, licenses/xilinx-hls-headers.txt and
its COLLECT.sh line go. viewer/resources/third_party_licenses.html is regenerated; it was stale, so the
regeneration also adds six texts already in licenses/ (durin, GPL-2.0, hpcn-uam, rapidjson,
slsDetectorPackage-md5-Apache-2.0, stb_sprintf). docs/THIRD_PARTY_NOTICES.md is left to
update_version.sh.
No host code (receiver, host_library, acquisition_device, common, tools, tests) uses ap_*/hls types;
tools/UDPSimulator.cpp keeps fpga/hls_simulation/sls_packet.h, which is plain C++.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
AbortLargeRun: 1M images of 8 modules, cancelled after a second, ends within
30 s (about 30 s and a timeout before the fix, about 5 s after).
StopCardsWhileStalled: nobody reads, the queue full and the card waiting for
buffers when StopCards() comes; every image the card got (195) is delivered,
in order, then the end message. ReceiverErrorStopsPuller: the writer refuses
the start message; Stop() throws and the card is released (it stayed armed
before the fix). CollectingPusher can refuse the start message for that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
An image whose frames no card got to (all cards finished, every one's fastest
frame before the image's first) is now let go at once, as JFJochReceiverFPGA
does, instead of being assembled from missing modules: an abort early in a
long run took minutes to work through the rest while Stop() waited, and a
silent end with fewer frames than asked for left the receiver without
messages long enough to declare the end message lost.
JFJochReceiverService::FinalizeMeasurement stops the puller even when the
receiver ends with an error, so its threads do not hold the cards' buffers,
and the cards stay armed, until the next run; the receiver's error is
rethrown first.
Nits: WaitForFrame's comment no longer refers to the removed cross-card
give-up; emplace for try_emplace; jfjoch_fpga_stills_test no longer prints
the always-0 dropped count; docs/TOOLS.md names the writer exception (an
analysed image without an image-buffer slot within the wait is not written,
writer_queue_full).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
As the Fable review proposed and the owner decided:
- HoldArena is the only hold: the quarter of single-frame slots of 4c9cc4838
(visible_held, visible_max) is gone. A waiting frame is seen by lookup: the
receiver keeps image number -> (weak_ptr<HeldImage>, the message as it came),
inserted in OnImage, erased when the frame is sent. JFJochReceiver::
GetWaitingImage (default: none) serialises it, its image in the hold, into the
output while it holds the frame; GetLastWaitingImage gives the latest.
JFJochReceiverService::GetImageFromBuffer (and the JPEG and TIFF routes) try
the image buffer first, then the receiver; MaxImage takes the later of the
two. ImageBuffer is unchanged.
- The pre-pass holds up to PREPASS_MAX_IMAGES; it starts when the hold has no
room for another frame (HoldArena::Full), on what is held.
- One behaviour, no IsBlocking(): a frame that finds the hold full waits while
the collection is not cancelled, and the pre-pass and the analysis drain it.
No frame is lost at the hold; a source faster than the analysis loses frames
at the card. An analysed frame that finds no slot to be written from within
2 s is still not written and counted (writer_queue_full), as before.
Tests: ImagesVisibleDuringPrePass sees every waiting frame by lookup (and no
slot in sending); BufferFullDuringPrePass, 200 frames into a 12-slot buffer,
writes them all with no loss; CancelDuringRun gives frames until the pre-pass
starts (the hold full) instead of relying on a six-frame pre-pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
Removed SlowConsumer_Puller, Paced and DroppingNoLockInversion (paths that
no longer exist). SlowConsumer: 300 images, more than the queue and the
mock's buffers hold, none lost, in order, and the mock card waited for host
buffers (pipeline_stalls_host > 0). CancelWithoutConsumer keeps its hang
guard. SlowConsumer_Receiver: no loss behind a slow writer. New
SilentEndDeliversAll: a silent cancel after the card is through still
brings every image to a slow writer. MissingModule ends the cards with
StopCards(). FPGAImagePullerSettings(bool) in the callers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
Removed -r (host pacing and the per-rate loop with its table), -d (the
dropping run), -p (the queue size) and -U with the dummy-module padding; the
blocked fraction, measured from the removed queue-wait time, is gone - the
card's own wait for host buffers (pipeline_stalls_host), printed per card,
says how long the chain held the source. Options: -i -s -N -B -C -S -X -t
-k -m -K -v. docs/TOOLS.md rewritten to match, and docs/JFJOCH_BROKER.md
no longer says a frame finding the stills hold full is dropped: it waits,
and frames are lost only at the card.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The broker ends every run with a silent cancel once the detector is done
(JFJochServices -> JFJochReceiverService::Cancel(true)). With every image
now waiting for the receiver, a full cancel there would cut the images still
on their way. StopCards() stops the cards and lets what they hold through,
then the end message; Cancel() is StopCards() plus "the receiver takes no
more images". The service uses StopCards() for a silent cancel and Cancel()
otherwise. Stop() still cancels first, for a receiver that ended on an error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
Images now always wait for room in the queue to the receiver (128 compressed
images, Queue), holding the cards' buffers, so the cards wait for host
buffers and frames are lost only at the card, as the card's generator and a
detector do. Removed what served only particular tests: the not-blocking
path that dropped an image at a full queue (and the dropped count, which
JFJochReceiverStream now reports as the base's 0), the host pacing of the
source (frame_rate_hz), the queue-wait time, the queue size argument, the
cross-card give-up (FastestFrame) with its late-frame bookkeeping
(given_up, ReleaseLateFrames), and the dummy-module padding of the cards
(CardExperiment, modules_per_card). The device itself gives up on a module
lagging behind its card past ThresholdFramesLost and returns a frame that
comes later; MissingAfterFrames stays beyond that, as the header says.
FPGAImagePullerSettings is reduced to the one bool that sends every
collection through the puller (tests comparing it with JFJochReceiverFPGA).
IsBlocking() stays, now always true: JFJochReceiverRugnuxStills' hold waits
on it, and that wait is changed on its own branch.
The tests and jfjoch_fpga_stills_test are adapted in the commits that follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The start message (and so the master file's detectorSpecific/summation_mode)
says "fpga" where the card sums alone, "cpu" where the host sums alone (a
prime above 256, or CPUSummation) and now "hybrid" where both do (1000 = 250
x 4). The string passes through CBOR and the writer as it is, and nothing
reads it back into a decision, so files with "cpu" or "fpga" read as before.
CBOR.md listed "internal", which the code never wrote: now none|fpga|cpu|hybrid.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The card sums up to 256, above that the largest divisor up to 256 and the
host the rest (hybrid summation); images are 32-bit. jfjoch_fpga_test
already has -S for the card-only case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
For a summation of N frames the card sums d, the largest divisor of N not
above 256 (GetFPGASummation; 256 itself is now summed on the card), and the
host the N/d card images of each image (GetHostSummation): 1000 = 250 x 4, a
prime above 256 = 1 x N. CPUSummation(true) still puts all of it on the host.
The card writes 32-bit where it sums and the host adds to it.
FPGAImagePuller now takes summation on the host, so rugnux live runs on such
collections. Each image thread keeps one int32 module accumulator per module
(ModuleSummation, saturating, error pixels carried, as JFJochReceiverFPGA and
the HLS do), takes the card's images of its image in order, all modules of
each before the next, adds each into the accumulator and returns the card's
buffer; the summed modules then go into the thread's image as before. A
module missing from any card image is missing from the image. The card's
per-frame spots and ROI sums were already not read on this path; the
module statistics are those of the first card image with error, saturated,
packet counts and pixel sum added (ModuleSummation now adds the pixel sum
too). Packets expected per image and the not-blocking give-up distance are
in card images (x N/d).
JFJochReceiverFPGA follows the same split (it has to: the card's summation
is set by the experiment), summing N/d card images instead of N frames; its
end-of-run expected packets no longer multiply the run's total by the
summation a second time.
OpenAPI image_time_us and the broker docs describe the split; C++ model,
frontend client and python client docs regenerated. Tests: the divisor rule,
and N = 256 (card only), 1000 (250 x 4) and 257 (host only) on the mock
through both receivers, pixels against a sum done in the test with error and
saturated pixels and a 32-bit overflow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The card counts the packets of a summed module in 16 bits (axis_completion
packet_count, hls_jfjoch.h), 512 per frame, so from 128 summed frames the
count wraps: at 128 and 256 to exactly 0, which the host took for a module
not collected (IsAnyPacketCollected), so every image was lost; between them
the efficiency came out wrong. Where 512 * card summation exceeds 16 bits,
the host now counts the packets from the module's packet mask (the packets
present in all of its frames) times the summation, and keeps the count in 32
bits (Completion, AcquisitionCounters). Exact for complete modules; with
packets lost in some frames it counts only those present in all. The mock of
the card keeps the card's 16-bit count, so it shows the same wrap (checked:
without this, summation 256 on the mock delivers no image).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
The fixed bound of 18 working reflections per atom on trying a U per atom is
replaced by the bound Hamilton's test itself sets: a U per atom is tried where
the working reflections outnumber the parameters of the model with it counted
unrestrained (nine per atom, the positions counted as Merritt (2012) counts
them) - the end of the range of restraint weights at which the test can be
taken at all. And where the tests are inconclusive and the free set decides,
a model with more displacement parameters needs its R-free lower outright,
not within the paired tie the incumbent is otherwise given: the burden of
the evidence is on the parameters it adds.
On the eight sets the choice turns on (CPU): a U per atom is taken at
1.2-1.4 A with 16-31 reflections per atom and at 2.0 A with 12, where it
lowers R-free; TLS is kept at 2.25 and 2.8 A with 6-8 reflections per atom
(the bound) and at 1.9 A with 13, where the U per atom raised R-free by 0.003
and only -LL free preferred it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The text said the image time is at most 256 frame times. Nothing enforces that:
summation of 2..255 frames runs on the FPGA (IsCPUSummation, MAX_FPGA_SUMMATION),
and anything longer is summed on the host by JFJochReceiverFPGA, which takes no
rugnux live analysis (FPGAImagePullerFor). The description now says so. Text
only; C++ model comment, frontend client and python client docs regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
38137e16a packed every waiting frame into the hold, so a frame was no longer in
the image buffer by its number until analysed (JFJochReceiverRugnux_Stills_
ImagesVisibleDuringPrePass failed). The waiting half of the image buffer is now
split: up to a quarter of the slots hold a frame each, serialised there as
before (the image routes find it), the rest of the half packs frames as their
compressed bytes (HoldArena). The bound is unchanged: at most half of the image
buffer, nothing beside it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The puller had 128 slots, each a whole uncompressed image and a compressed one
(about 40 MB on a 9M, 5 GB, allocated and written at every start, no knob), a
thread per module writing its module into the slot of each image in flight, and
since 0060c734b a further copy of the compressed image.
Now, as JFJochReceiverFPGA: each image thread takes the next image, waits for its
modules in the cards' buffers (WaitForFrame, the same rules), assembles them into
an image of its own, compresses it, and the cards' buffers go back. The cards'
buffers are the window over the images in flight; nothing else holds an image.
Frames of a module given up on that arrive later go back as soon as they do
(ReleaseLateFrames). Dummy modules (CardExperiment) are only returned.
The images compressed and waiting for the receiver are bounded by `queue`
(default 128, -p of the stills test, the old slot count): not blocking, an image
the receiver has no room for is dropped before it is assembled, its frames back
at once; blocking, it waits for room (the time is GetQueueWaitTime_s, the source
blocked). The warning says the receiver's queue was full, not "no free slot".
Per collection on a 9M with N image threads: N x (an uncompressed image + its
compressed bound), 64 threads 2.6 GB where 128 slots were 5.2 GB; the queue
holds compressed images only. Per image: the modules copied once into the
thread's image, compressed once, the compressed bytes copied once into the
output (the receiver keeps them) - as before less the slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
An image not chosen went neither to the analysis nor to the writer. It now
waits with the others and goes to the writer as it came, without its analysis
(HeldImage::analyse): images_sent equals the images collected again; the
analysed and indexed counts and the merge are of the images analysed. POST
/initialize resets stills_downsample to 1, as force_still and refine_solvent.
The OpenAPI text and the broker docs say both, and that the first 2000 images
(the pre-pass's largest sample, StillsGPUAnalysis::PREPASS_MAX_IMAGES) are
all analysed. C++ model, frontend client and python client docs regenerated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Lane l's hold (702b8d4a6) kept a frame that found no free slot as a heap copy
of its compressed bytes, up to the image buffer's size again - up to twice the
configured image memory. Now every waiting frame is held as its compressed bytes
packed into slots of the image buffer (HoldArena): several frames to a slot, at
most half of the slots, the slots kept for the analysis threads to send with
left alone; a slot is reused once its frames are gone, and the slots go back
before the image buffer is checked at the end. A frame that finds the waiting
half full is dropped and counted as before (blocking: waits). The receiver's
image memory is the image buffer (image_buffer_MiB of the broker, -B of the
stills test) and nothing beside it; the broker takes the same receiver.
The frames are no longer serialised into a slot of their own while they wait,
so the broker's image routes show a frame once it is analysed.
HoldArenaTest: half of the slots at most, three frames of a third of a slot
each to a slot, nothing larger than a slot, a freed slot reused, all slots
back after ReleaseSlots. jfjoch_fpga_stills_test prints the process's maximum
resident memory.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The live stills analysis took min(16, threads) threads whatever the cards.
It now takes what offline rugnux's per-image loop takes (ImageWorkers):
16 a card, fewer on a card too small for 16 workers' engines (half its memory,
about 24 bytes a detector pixel each, GPUWorkersPerCard), at most the threads
given; without a GPU, the threads given. Both now come from free functions in
Rugnux.h that the Rugnux members call.
The indexer pool stays the broker's (indexing settings thread_count, default
4): it is shared with the per-image analysis and its GPU indexers are built
once for the broker's life. jfjoch_fpga_stills_test -t defaults to 8, as offline
rugnux on stills (StillsIndexerThreads).
On this host (one GPU) the thread count does not change (16); on a server with
4 cards it is 64, at most -N.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
-v raised the logger to debug, which the devices use for two lines per module
frame: 2.25 M lines and 570 MB in 56 s on the server's four cards, until its
disk filled. -v now prints only the 0.5 s progress lines; the debug log is off.
The completion line of FPGAAcquisitionDevice::ReadWorkCompletionThread goes:
its frame and module are read from the buffer only later, so it printed 0 for
both (WaitForActionComplete's line has them).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
- The compressed image is copied into a buffer of the output's own and the
slot returned at once: slots are held while an image is assembled and
compressed, not while it waits in the queue or in the receiver. The queue
keeps its bound of max_slots images: not blocking, an image that finds it
full is dropped and counted; blocking, it waits for room while the receiver
takes images (until Cancel()). The end message waits the same way. Cancel()
(the receiver stops taking images) is now apart from stopping the cards
(CancelCards), which the puller does at its own end without giving up on
the receiver. No slot deleter runs under a lock any more.
- modules_per_card (FPGAImagePullerSettings): the cards run
FPGAImagePuller::CardExperiment, each card with that many modules - its
modules of the image first, then dummies the puller only returns to the card.
The internal generator runs as fast as its card's modules allow, so a 9M on
4 cards (5, 5, 4, 4) ran at 2.04 and 2.55 kHz on the server and drifted
~500 images/s apart; padded to 5 each the cards send the same bytes. The
expected packets, the efficiency and the image are of the image's modules.
- jfjoch_fpga_stills_test pads by default where the modules do not divide
evenly (-U: not), lists each card's modules and dummies, and loads the
generator card by card with zeros for the dummies; the distinct images
follow the padded count.
Mock, 9M, 4 cards, blocking, 3000 images: padded and -U give the same images
(indexing 0.760, 48.7 spots per image, 25 distinct). The mock cards' speed does
not go with their modules (each is two CPU threads), so on the mock padding does
not keep them in step: that is for the server to show (-v).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
-r holds frame i on the host until i/rate, which makes the card wait for its
buffers: with -d that is blocking again, not a detector at that rate. Dropping
the frames that come early instead is no better: the cards' generators run
free, each at its own speed, so each card would keep a different subset of the
frames. The combination is refused.
With -d, the drops "no free slot" come from those free-running generators
drifting apart: the images the cards ahead have started hold the puller's slots
until the frames of the card behind arrive (it is waited for until max_slots
behind). The receiver is not what holds them: JFJochReceiverRugnuxStills copies
the image into its image buffer (or its own copy) in OnImage and the puller's
output, with the slot, goes when OnImage returns. A detector's modules share a
trigger and do not drift. Usage and docs say to give -p at least the drift -v
shows (mock, 9M, 4 cards, 1500 images: efficiency 0.49 with 128 slots, 0.98
with 512).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The hang of a non-blocking run on four cards (backtrace from the server): a
module thread dropping an image called Emit from GetSlot with images_mutex
held and waited for order_mutex, while a compress thread in Emit, holding
order_mutex, destroyed the last copy of an image - the receiver had already
taken and dropped the copy just queued - and its deleter (ReturnSlot) waited
for images_mutex. Only a dropping run goes through that branch of GetSlot.
- GetSlot emits a dropped image after releasing images_mutex.
- Emit and the flush at the end keep the copies of what they let out until
order_mutex is released, so no slot goes back under it.
- Cancel() sets `cancelled`: an image waiting for a slot in blocking mode is
dropped instead, as after a cancel nobody may take the images holding the
slots (FPGAImagePuller_Blocking_CancelWithoutConsumer hung before, passes now).
- FPGAImagePuller_DroppingNoLockInversion: 4 modules, 2 slots, 8 compress
threads, a consumer that pauses so that images are dropped and frees them at
once; under a time limit. The race window is a few instructions wide: on the
mock it did not hang before the fix either, so it guards the lock order rather
than reproducing the server's hang.
- jfjoch_fpga_stills_test -v: the receiver's and the stills analysis' counts and
whether each card is done, beside the cards' progress, so that a run that stops
shows which part waits.
The 4 h hang of a blocking two-card mock run (2026-11-18 09:29) was the binary
before 4c8af0dbc (late frames of a lagging card not released while a module
waited); the same run now takes 17-45 s (4 repeats).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
As Merritt (2012) and PDB-REDO (Joosten et al. 2012) apply Hamilton's R-ratio
test: each model's restraints are credited as w R observations, w scanned
from 0 to the bound at which they take every parameter they restrain off its
count (Merritt's N_atoms / R for a B per atom, 6 N_atoms / R for a U per
atom), the two models' weights independently, six values each. The larger
model is taken where it passes more than 95% of the tests, the smaller where
it passes fewer than 30%, and in between the free set decides (the paired
R-free and -LL free rule every other free-set choice uses, rather than
PDB-REDO's Z(R-free) and R-free - R gap cut-offs, which are fixed thresholds
on one measure). A U per atom is tried only above 18 working reflections per
refinable atom, PDB-REDO's bound.
Against the single unrestrained test this replaces: a U per atom is taken
again at 1.2-1.4 A with 23-33 reflections per atom (the tests inconclusive,
the free set preferring it) and still not at 2.0-2.8 A with 6-12.5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
dataset_settings.max_spot_count loses its default in OpenAPI and becomes
optional in DatasetSettings. Left out, the broker's per-image indexing
(JFJochReceiverFPGA, JFJochReceiverLite) keeps DEFAULT_MAX_SPOT_COUNT = 250, as
before; rugnux live stills takes rugnux's default, the budget the pre-pass
measures up to 1000. Given, both use it: the pre-pass keeps it as
`rugnux --max-spots` does (no budget measured) and the frames after it are
indexed on it. Before, live stills always took 1000 (lane l, 17bfca6a2),
whatever was given.
C++ model, frontend client and the python client's docs regenerated (the
python-client package itself is built at release by update_version.sh);
OpenAPIConvert sets it only where given; the frontend builds. The stills
replay's -p default is now that of rugnux live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
- rugnux (mx mode) already cleared the file's space group and cell
(ClearStoredCrystal, since rc161); it now says so in the log when the file had a
space group and no -S was given.
- jfjoch_stills_gpu_replay and jfjoch_fpga_stills_test built the collection's
start message from the file and so carried the file's space group into the live
merge as if the user had given it. Now only -S gives one (new in the stills
test); the file's is ignored with a log line.
- The live path is unchanged: the space group of the start message (the dataset
settings) is what the merge uses, as given.
- --mode scale keeps taking the group a _process.h5 was written in: rugnux's own
output, its reflections already indexed in it.
- No battery row depended on the file's space group: the battery runs mx mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
In the displacement-model choice, a candidate with more displacement
parameters than the incumbent (TLS bodies over a B per atom, a U per atom over
either) now also has to lower R-work by more than its extra parameters
account for: R_small / R_large above Hamilton's critical ratio
sqrt(b / (n - m) F(b, n - m, 0.05) + 1), n the working reflections, m the
larger model's displacement parameters (the hypothesis trials hold the
positions), b the extra ones, counted unrestrained. The F point is Paulson's
normal approximation (within 0.3% of the exact value at 5/100 degrees of
freedom and closer above). Each trial records its displacement parameters.
On the 2.0-2.8 A sets where a U per atom was taken on -LL free alone, the
test now refuses it (1-1.3 working reflections per U parameter); where TLS was
taken over the B per atom it passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
A serial collection can be analysed on a fraction of its images, to keep up
with a stream faster than the analysis. Not a stride, which is in step with
periodic patterns of the stream (pump-probe on/off, storage cells, module or
card interleave) and could take only the dark frames: each image is chosen on
its own with probability 1/F from the SplitMix64 finaliser of its number,
salted (StillsImageSelected), reproducible and the same on every thread and
card. The first StillsGPUAnalysis::PREPASS_MAX_IMAGES (2000) images are all
taken, as the pre-pass may use any of them. An image not chosen is counted as
collected and skipped before the analysis, neither analysed nor written - as
one the serial filter leaves out.
- OpenAPI rugnux_settings / rugnux_status: stills_downsample (int64, >= 1,
default 1); C++ model and frontend client regenerated; the Rugnux settings
page gets the field. Kept by POST /initialize, as reserved_cpus.
- JFJochReceiverService::RugnuxLive, JFJochStateMachine::SetRugnuxLive,
JFJochReceiverRugnuxStills take it.
- jfjoch_fpga_stills_test -k <F>, jfjoch_stills_gpu_replay -k <F>.
- StillsSelectionTest: fraction within 5 sigma for F = 2, 3, 4, 10 over 400k
images, every phase of a period-2 and a period-F pattern gets its share,
deterministic, pre-pass images and F = 1 all taken.
Measured (mock, 9M, 6000 images, -k 4): 2984 analysed = 0.497 (expected 0.5
with the first 2000 all in; 0.246 of the rest). Host-only replay of 10000
serial 8M images: 17.0 s -> 10.0 s in all, after the pre-pass 10.6 s -> 3.4 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
- JFJochReceiverFPGA no longer runs rugnux: its live stills (StillsGPUAnalysisFPGA, GPUSink, the merged data
in END) and its pedestal code go; the receiver is the one of rc186 (rc186-old), which differs from what it
replaces only in these two parts.
- JFJochReceiverPedestal back from rc186 (242587c58, 7080109a4) unchanged: raw frames from the devices into
JFPedestalCalc, nothing on any socket. In rc186 it took every PSI pedestal run directly from the devices,
not through the puller; so it does here.
- Dispatch for a PSI detector: pedestal -> JFJochReceiverPedestal; rugnux live on (Standard mode, no host
summation) -> FPGAImagePuller and JFJochReceiverRugnuxStills or ...Rotation; otherwise JFJochReceiverFPGA,
unchanged. With host summation rugnux live is not run (a warning in the status).
- The FPGAImagePullerUse enum goes: FPGAImagePullerSettings(slots, blocking, rate, always), where always (every
collection through the puller, without rugnux to JFJochReceiverLite) is kept for the tests comparing the
puller with JFJochReceiverFPGA pixel for pixel.
- jfjoch_fpga_stills_test loses -F; StillsGPUAnalysisFPGATest goes, the GPU-stills processing tests run the
puller path only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Every frame that arrives before the pre-pass is over waits for the analysis, and each took a slot
of the image buffer, sized for an uncompressed frame plus metadata (24 MB on a 9M detector for a
frame of ~1.5 MB compressed). Not blocking, the frames past the buffer were dropped: on the mock,
9M, 2 cards, 1500 frames at ~140 Hz, a 13-17 s pre-pass, 683 of 1186-1400 collected were sent.
A frame that finds no slot is now held as a copy of its compressed bytes, up to the image buffer's
size again; a slot still comes first (the frame can be seen there while it waits), and the frame
is sent from a fresh slot after its analysis as before. Same mock run: 1462-1489 of 1462-1489
collected sent (the rest are the mock puller's own module drops), peak RSS 27.1 -> 27.7 GB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The other builds link cuFFT as before (dynamic on the server build, static in
the portable ones). jfjoch_fpga_stills_test is copied to the FPGA servers on
its own: its JFJOCH_STATIC_CUFFT property makes the indexer and the beam-centre
FFT, which choose by a generator expression on the linking target, give it
libcufft_static.a, and it gets the device-link step that needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The R-free part of the free-set rule is a paired difference over the free
reflections; the model as given is kept only where it displaces the
refinement and the refinement is not better on both R-work and R-free; and
why -LL free keeps the refinement's own likelihood parameters rather than D
and the variance fitted to the working set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The pre-pass refines the cell beside the distance (their scales trade), and offline indexes every
frame on that refined cell with up to 1000 spots. Live kept the stated cell at the refined distance
(off in scale by as much as the distance moved, 1.7% on one 9M set) and the broker's 250-spot
cap (a third short of a 9M frame's ~380 spots). Both now come from the pre-pass result; the
start message keeps the stated cell, and the merge keeps it as the given cell.
Replay on the first N frames (live before -> after, offline rugnux on the same frames):
9M 100 frames 0.99 -> 1.00 (offline 0.99), merge CC1/2 0.758 -> 0.798; 9M 30 frames 0.967 -> 1.00;
4M 100 frames 1.00 -> 1.00 (offline 1.00); 8M 200 frames 0.320 -> 0.325 (offline with the file's
space group 0.320). Per-image analysis +10% on 9M (16.0 -> 17.6 ms, more spots).
The 8M gap reported (live 0.35 vs offline 0.56) was the comparison: the file carries a space group,
which live takes from it and offline only with -S. Offline with -S gives 0.349 = live; without it,
offline indexes 13-14 more frames of 6-10 spots out of ~40, and merges worse (CC1/2 0.10 vs 0.20).
jfjoch_stills_gpu_replay: -p spots per image; per-frame <prefix>_images.tsv and _spots.tsv to
compare frame by frame with an offline _process.h5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
After a second round whose weight trials held the U the first round's TLS or
U-per-atom choice made, the label was reset to "isotropic" (or "the model's U
held") because the kept trial carried no hypothesis name. It now keeps the
first round's model with ", U held"; and a model whose per-atom U are held
also drops the input's TLS groups, as one that refined them does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
R-free in FreeSetPrefers is now compared as a paired difference: each trial
keeps R-free as one term per free twin group (FreeRTerms, the grouping
TwinNull's paired z uses, now shared as TwinGroups), and a candidate may be
higher than the incumbent by no more than the standard error of the sum of the
groups' differences. The single-model standard error R_free / sqrt(2 N_free)
it replaces was several times wider than the scatter of two trials from one
start on the same reflections, which left -LL free deciding almost alone.
The as-given guard now asks whether the model as given displaces the
refinement on the free set (rather than whether the refinement displaces it),
and never keeps it where the refinement is better on both R-work and R-free:
on one guard set the refinement lowered R-free 0.2408 -> 0.2373 and R-work
0.242 -> 0.215 but raised -LL free by 7, and the model as given came out.
That -LL free is higher there however D and the variance are fitted (on the
free set, on the work set, on both, or crosswise), so it is not a
consequence of fitting the variance on the free set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
A module was given up for a frame once ANY card was MissingAfterFrames (52)
frames further on. The cards run on their own, so a card a little behind
lost whole modules from every image: on the mock, 9M on 2 and 4 cards not
blocking, the cards drift 50-500 frames apart (each mock card is two CPU
threads; none of them ever waited for host buffers, so the drift is the
cards' own speed, not the host) and the efficiency was 0.44-0.61.
Now a module is missing when its OWN card is MissingAfterFrames further on,
as the device itself counts, and behind another card only once that card is
max_slots ahead - past that, its images would be dropped for want of a slot
anyway. A dead card is still noticed, after max_slots frames.
Mock, 9M, 1500 images, not blocking: 2 cards 0.905 (drift up to ~120), 4 cards
with 512 slots 0.978 (drift ~380), 4 cards with 128 slots 0.487 (drift ~500,
now whole images dropped for want of a slot rather than modules missing).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
-d runs the puller as the broker does, dropping what the host has no room
for, to compare with the default blocking run. -v logs each card's slowest
and fastest module frame every 0.5 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The server build linked libcufft.so, so any executable copied to another
machine needed the CUDA toolkit library beside it; only the portable builds
linked libcufft_static.a. Now every build does where CUDA::cufft_static
exists (Linux); Windows, which has none, keeps the DLL as before.
libcufft_static.a needs a device-link step in every executable that links
it. The four per-target CUDA_RESOLVE_DEVICE_SYMBOLS settings (rugnux,
jfjoch_viewer, jfjoch_portable_test, rugnux_refine_test) become one pass at
the end of the top-level CMakeLists over the executables and shared
libraries of the source tree (fetched dependencies left alone).
Executables grow by about 290 MB (rugnux 193 -> 486 MB, jfjoch_broker 488
MB, jfjoch_fpga_stills_test 484 MB); ldd no longer lists libcufft.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
FPGAImagePuller gains two options, off by default (the broker is unchanged):
- blocking: an image waits for a free slot instead of being dropped, and no
module is given up for being behind the others; JFJochReceiverRugnuxStills
then waits for room in the image buffer (ImagePuller::IsBlocking). The cards
keep their frames, host_writer waits for host buffers and the stall reaches
the internal generator through the AXI-stream back-pressure, so nothing is
lost anywhere. The time the module threads waited for a slot is kept.
- frame_rate_hz: image i is taken from the cards no earlier than i / rate
after the start. The firmware has no frame period for the generator, so
this is how it is paced.
GetSlot now looks for the image again after each wait (another module may
have started it).
jfjoch_fpga_stills_test always blocks; -r runs one collection per rate and
reports delivered rate, the fraction of it the source was blocked, sustained
rate (images / wall time), analysis rate after the pre-pass, indexing rate,
and the card's cycles waiting for host buffers; a table at the end. The mock
counts its wait for host buffers as pipeline_stalls_host, like timer_host.
Tests: FPGAImagePuller_SlowConsumer_Blocking (no drop, all images in order),
FPGAImagePuller_Paced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
- Layout from the file: its NXmx module groups, else the modules the gaps of
its pixel mask mark (any placement and orientation of 1030 x 514 modules);
a module without a stated readout direction is read out along its long
side from its box's corner (the multipixels are symmetric, so the frames
are the same). The file's modules also go to the analysis.
- Distinct images per card = (its generator module frames - 1) / its
modules, the run takes the smallest over the cards, at most
FPGA_INTERNAL_GENERATOR_MAX_IMAGES (frame_cell, 128) and within
FPGA_INTERNAL_GENERATOR_MAX_MODULE_FRAMES (mem_cell, 1024). The broker's
64-image limit stays: DetectorSettings::InternalGeneratorImages takes the
maximum as a parameter, defaulting to it. The mapping (cards, modules,
module frames used) is printed at start.
- -K defaults to 64 MiB of mock generator memory = 128 module frames.
- Squeeze round-trip tests for a 4M and a 9M layout and a 16-module layout
with turned and standing modules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
A module waiting for its next frame released the late frames of images it
had given up on only after the wait. With one card more than
MissingAfterFrames behind the others, its buffers filled with such frames,
the card could not write the frame waited for, and the run hung (seen with
two mock cards under load). The release now happens inside the wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
With steps now taken on the work-set target, the trajectory goes on from the
ladder's choice instead of stopping at its best step on the free set, and the
weight halved (tried only at the first step before) is how it can tighten a
weight looser than the data want: on the 19-set sweep (GPU) it took one set
back to its earlier R-free (+0.0038 -> +0.0003) and changed no other, at 3%
more target evaluations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
A separate program (jfjoch_fpga_test is unchanged). The first images of a
stored JUNGFRAU dataset (assembled, 8/36 px gaps; the layout from the image
size) are squeezed back into module frames and loaded into the card's internal
frame generator; the run is made with the conversion off, so the card returns
them verbatim, and the host assembles them as signed 16-bit counts for rugnux
live stills through FPGAImagePuller (-F: JFJochReceiverFPGA's live stills).
Geometry, energy and cell from the file (-C overrides the cell).
The generator holds what its memory and the 64-image generator setting allow
and cycles through them for the whole run, so images repeat: throughput,
latency, spots and indexing rate are what it measures, not the merge. It
prints images collected/sent/dropped, frame rate, CPU use, indexing rate,
spots per image, per-stage times, the pre-pass and its decisions, the
analysis rate, the latency after the last image and rugnux's steps.
-K runs it on MockAcquisitionDevice, whose generator memory is now a
constructor argument (default 32 MiB as before); AcquisitionDevice exposes
the module frames its generator memory holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The inverse of TransferModuleAdjustMultipixels: a raw pixel is the sum of the
2 (multipixel) or 4 (chip corner) assembled pixels its value was split over,
as the split conserves the count; an error pixel among them makes it an error
pixel, a saturated one or a sum past the destination's range a saturated one.
ConvertedToRawGeometry takes one of them instead, which is right for masks and
maps but halves the counts of the split pixels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
FPGAImagePuller (rc186's, as 0d26ecd13 left it, with 6b485e57c: every slot and
its compressed-image buffer made up front) takes the card's frames, assembles
and compresses them and hands them on in image order as an ImagePuller, so a
PSI collection runs through the same stream receivers as a DECTRIS one.
JFJochReceiverService::UseFPGAImagePuller(FPGAImagePullerUse, slots):
- RugnuxLive (default): a PSI collection analysed by rugnux live goes through
the puller to JFJochReceiverRugnuxStills, or JFJochReceiverRugnuxRotation for
a rotation (which the FPGA path had no live analysis for); every other
collection stays on JFJochReceiverFPGA.
- Always: also the others, through JFJochReceiverLite (rc186's switch).
- Never: everything on JFJochReceiverFPGA, its own live stills included (the
rc188 behaviour, kept for comparison).
Pedestal runs, summation on the host and runs without images never take it.
Ported by hand, not cherry-picked: the puller files and its tests are restored
from rc186-old; the service changed since (force_still, refine_solvent,
ClearRugnuxStatus), so its dispatch is written anew around them.
Tests: FPGAImagePullerTest (rc186's 20 cases on the mock, the switch as an
enum, plus FPGAImagePuller_DefaultOnlyWithRugnuxLive); the three GPU-stills
processing tests run on both the default and JFJochReceiverFPGA's live stills.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Describe the free-set rule (free -ln L lower and R-free within its standard
error), the work-set target every other step is taken on, and credit Tickle,
Laskowski & Moss (2000) and PDB-REDO (Joosten et al. 2012, 2014).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The free set now decides only the X-ray weights of the positions and the B
factors, the NCS restraints' sigma, the displacement model (isotropic / TLS
cuts / U per atom), the reading of the model's TLS groups, and whether the
refinement displaces the model as given. Each of these takes a candidate over
the incumbent only where both free measures agree: its -ln L of the free
reflections lower and its R-free not higher by more than the standard error of
the incumbent's, R_free / sqrt(2 N_free) (Tickle, Laskowski & Moss 2000), in
place of the fixed R-free tie of 0.001 (which alone decided most choices). The
approach of requiring free measures to agree follows PDB-REDO (Joosten et al.
2012, 2014).
Every other decision now uses the work-set target - w (-ln L work) + E_geometry
(with NCS at its sigma), plus w_B (-ln L work) + E_B where B is refined - at
the weights the two candidates share, a step taken where it lowers that by
more than 1e-4 relative and the geometry stays within bounds where it was:
the trajectory's steps (which now end where the last step taken ended, not at
the best step on the free set), the final ladder's step, the step after the
displacement choice, the second round, the solvent build's rounds, the
side-chain field's kept test and the population's child. A candidate at other
weights is not compared on a target that changed with them (the positions and
B minimise separate targets, whose sum a weight change can raise while it
lowers each) and is left to the free-set rule. The placement's rigid bodies
and B per residue are kept where they lower -ln L work.
Each weight trial now records -ln L work and the restraint sums
(RefinementWeightTrial::nll_work, e_geometry, e_b) and logs its target.
Not changed: the twin-law choice (a paired free-set z against its null, a
model hypothesis test with its own standard error), MR and model validation.
No k-fold cross-validation yet for free sets under 500 reflections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Since the transpose product reads J from a per-atom ordered copy (jt) and each entry's row from
row_at, an entry holds two float4 and three int (param, entry, row_at) on the card; row_of stays on
the host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The positions' normal matrix a trial holds on its card while it minimises (J and its index, the
CG vectors, the per-residue block inverses, the kept residuals of the reorthogonalisation) was
outside the GPU budget that places the trials: on a 16 GB card a large model placed three trials
by a 12.1 GB count and peaked at 14.2 GB (a run died there with an out-of-memory once the card
had less to spare). RestraintNormalMatrixGPU::DeviceBytes counts what its allocations come to,
from the geometry restraints as the engine's restraint count takes them and the residues'
parameter atoms; RefineOnce adds it to each engine's own bytes, so the slot placement and the CPU
fallback, both decided up front, see it.
Ribosome case (open arm, 16 GB card, -N 16): 0.91 GB per trial; two trials at once instead of
three; predicted peak 11.74 GB (11196 MiB), measured 11348 MiB. Without the count the measured
14198 MiB matches the old placement plus three normal matrices (14124 MiB). With the card
reported as 7 GB the run goes to the CPU before the first trial, with 12 GB it places one trial.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The three components were drawn inside one argument list, whose order of evaluation is unspecified, so
compilers (GCC right to left, Clang/MSVC left to right) built different directions from the same seed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The residuals' reorthogonalisation launched a dot and a projection per earlier residual (about 570
extra launches per solve): a solve took 5.7-10 ms on the small models against 1.5-3.8 ms before
it. All coefficients are now taken at once from the same z (classical Gram-Schmidt: c_j =
z . r_j / rz_j, on the CPU too), in one partial-sum kernel over all kept residuals, one finish and
one projection: 1.9-4.4 ms per solve (myob 3.3, cytc 2.8, 7ris 2.6, 9zlo 1.9, 7l6j 4.4 ms; 4ybb
44 ms, as before the reorthogonalisation).
Memory: the per-residue preconditioner keeps only the lower triangle of each block inverse, and the
kept M^-1 r_j are in float (c_j comes from r_j, kept in pairs, so float's rounding of z_j enters only
as c_j times it, far below the pairs). 4ybb peak GPU memory 14838 -> 14198 MiB (base 12982, without
the reorthogonalisation 14038).
Device vs host still median 3e-14, max 6e-12 (myob, cytc, every solve); a 1e-12 change of the
right-hand side moves the solve by 2e-12.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
HLSSimulatedDevice, HLSDevice (fpga/hls_simulation) and the tests of the HLS kernels themselves go:
FPGANetworkTest, FPGAFrameGeneratorTest, FPGAHostWriterTest, FPGASpotFindingUnitTest, FPGAPTPTest,
FPGAHLSModulesTest, and MockAcquisitionDeviceTest, which compared the mock with the HLS device. The
receiver, the acquisition-device library and jfjoch_test no longer link JFJochHLSSimulation, and
jfjoch_fpga_test loses its -H (HLS simulation) option; without hardware the broker and the tests run on
MockAcquisitionDevice.
Kept for the firmware: fpga/hls (untouched) with its CPU library JFJochHLSCores, now EXCLUDE_FROM_ALL
so the software build does not compile the kernels or the Xilinx headers of fpga/include (the
pcie_* firmware targets build it on demand); fpga/hls_simulation/hls_cores.h and sls_packet.h, which
the HLS testbenches (fpga/hls_tb), sls_detector.cpp and tools/UDPSimulator.cpp include. The licence
notices are unchanged: fpga/include and parallel_stream.h stay, datamover_model.h carried no third-party
notice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
With the mock the generator's 30 frames arrive at once and the run was over before the cancel. The
case now gives the card fewer frames than the pre-pass waits for and no final packet, so the frames
stay held, whatever the card's speed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
A configuration without a pcie entry gets the mock of the card (AcquisitionDeviceGroup::AddMockDevice)
instead of the HLS simulation. The local run of docs/JFJOCH_BROKER.md works as before: the images
uploaded as the internal generator's images are streamed and written (checked end to end with
jfjoch_broker_test.py: 18 images collected, written and equal to the uploaded image outside the pixel
mask); the card's spot finding, ROI and azimuthal results are zero in this mode, as the mock computes
nothing. The docs and CLAUDE.md say so.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
TransposeProduct summed each component over an atom's entries in a loop of its own, so every
entry's index, J and row were gathered once per component, in random sectors. Now the entries
are read once, each component's float-pair sum updated in the same order as before; and J is
copied into the transpose's order once per At() (TransposeJ, beside JacobianKernel in Fill),
with the row of each entry indexed once at Upload, so the product streams J and the rows and
gathers only t. The DiagonalKernel reads the same copy. Every sum keeps its terms and their
order: the products and the diagonal are bit-identical to the old kernel's (all four cases of
RefinementNewton_product_GPU_vs_CPU dumped and compared), at +20 B per entry of device memory.
On a 242k-atom model with 45 NCS copies, 4 x T4, the first ladder (4 trials per card): per
product, xyz without NCS 3.12 -> 0.89 ms, with NCS 11.1 -> 2.88 ms, B unchanged (0.44 ms);
TransposeJ 0.29 ms per At(). The refinement 1116 -> 1065 s, the refined model bit-identical, the
card peak 11.3 GB. Two small models: bit-identical, slightly faster.
Following the review in TPROD_REPORT.md (gather amplification diagnosis, options A and B).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
The host-side tests run on the mock instead of the HLS simulation: JFJochReceiverIntegrationTest,
JFJochReceiverProcessingTest (the GPU-stills cases too, in raw mode the card writes the generator's
frame), JFJochStateMachineTest, StreamWriterTest. tests/MockCardOutput.h gives the mock what the
card would write for a test's image - its conversion by the host's floating-point reference, pixel
statistics, ROI counts and the strong pixels of the CPU spot finder - so the checks of the receiver's
images and plots stay as they were. Two cases are new: a receiver on two cards at once (not possible
with the HLS simulation) and a cancelled pedestal run.
FPGAIntegrationTest.cpp (60 cases) tested the HLS device itself - conversion, summation, spot finder,
integration, ROI, ADU histogram, pixel counts, in every mode - and is removed, not ported. Its host-side
checks (FPGAAcquisitionDevice and AcquisitionCounters: the internal generator, data streams, missing
modules, delays, lost frames, the last frame, bunch ID, the generator's memory limit) are
FPGAAcquisitionDeviceTest.cpp on the mock, plus a check that the generator's frames are what the mock
writes.
From ce8918062 and 710edd61e (rc186), resolved against rc188: JFJochReceiverTest() without
quick_integrate, the FPGA receiver that indexes and does not integrate, StillsGPUAnalysisFPGA.
JFJochReceiverFPGA_RugnuxOnStaysPerImage is not taken (rc188 keeps live stills on the FPGA receiver).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
~EvaluatorGPU switched to its card to free its buffers and left the thread there. With the
field on every card (f07493c9b) the evaluators are destroyed on the refinement's thread, which
was then left on the last card: the engines the second round makes again (a raised B ceiling)
went with it, so the main engine landed on a card already holding its share of slots.
Measured on a 242k-atom model on 4 x T4, per card, against the footprint OwnSlotPlaces plans
(11290 MiB on every card): with the fix the untwinned arm peaks at 11188 / 11124 / 10836 /
10996 MiB (the field's search adds about 450 MiB per card, freed after it); without it card 3
reached 14295 MiB, the main engine there beside the card's slots. The plan is an upper bound once
the engine stays on its card. The refined model is bit-identical, with the same 16 trials at a
time in both arms (1116 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
The same packets, or the same internal-generator run, on HLSSimulatedDevice and on the mock given what
the HLS wrote; the host must see the same on both: bytes received, packet counters, which buffer each
module went to (the order of completions), delays, complete/incomplete modules, the module headers, and
for the receiver its images, counters and plots. One case per mode: raw and converted JUNGFRAU in every
pixel format, summation on the card and on the host, storage cells, pedestal modes, EIGER 8/16/32 bit,
two data streams, cancel, out-of-order and erroneous packets.
Run once before the HLS simulation is removed: 9 cases, 633 assertions, all pass. From 497073dc9 and
b00e2fbe1 (rc186).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The card as a black box, as the host sees it: which module is written to which host buffer and when,
the order of completions, packet counts and masks, frames missing or late, summation groups, counters,
the end of the data collection, cancel, and the internal frame generator with the refusals of
frame_generator.cpp (return codes 1-3, the data collection cancelled, FPGA_EndAction throwing as
HLSSimulatedDevice does). It computes nothing: what the card writes for a module is given with
SetOutput(); without one, the pixels are the internal generator's frame for it (the frame loaded with
SetInternalGeneratorFrame, 16 bit, not reordered) or zeros, every result zero. The card memory is
32 MiB per HBM interface as in the HLS simulation, so the host's generator memory check (64 module
frames, the last kept free) applies. Any number of these devices can run at the same time.
From 8e3b0f88d (rc186), extended with what the FPGA path gained since: the generator memory check and
the generator's refusal path.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Where the lattice admits a twin law a second, twinned refinement runs beside the untwinned one and is
decided at its end by how far the law is ahead of its null (TwinNull, taken at 3 standard errors, 2
where the data read twinned). That lead is now also measured after the twinned arm's first ladder;
under 0.5 standard errors the arm stops there, its threads go to the untwinned arm, and the law is
decided (not taken) where it stopped. On 19 refinements with a law the lead rose by at most 1.35 from
the ladder's end to the end; the 3 twinned crystals were 8.6-17.7 ahead at the ladder's end.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Where the untwinned and the twinned refinement run side by side on half of the threads each, the
one that ends first left its half idle while the other ran on. Its threads now go to the other arm's
weight trials from its next wave on. Nothing in an arm depends on its thread count.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The normal-matrix solve (20 fixed preconditioned CG iterations per L-BFGS direction) was forward
unstable: run in lockstep on right-hand sides 1e-14 apart, two CPU solves in double drifted apart
5-7x per iteration from about the 5th iteration and ended up to 7% apart (myoglobin: 17% of the
position solves above 1e-5). That is the loss of orthogonality of the recurrence in rounding once
its first Ritz values converge, and it is what set the CPU and the GPU apart: the device solve
matched the host to 1e-14 until the first such call and then differed by exactly that spread. It
is not small curvature: an absolute floor on the X-ray diagonal (1e-3 x median, the review's
remedy) changed nothing, and blocks of up to 8 residues in the preconditioner hardly anything.
Each new residual is now made M^-1-orthogonal to all the earlier ones (modified Gram-Schmidt on r
and z together), on the CPU and on the card. Measured on every solve of a refinement run
(myoglobin, cytochrome C): a 1e-12 change of q now moves the result by 2e-12 (max 2e-11), was
median 2e-7 / max 8e-2; device vs host median 4e-14, max 1e-11, was p90 2e-3 / max 6e-2. The
residual after 20 iterations is the same (mean |r20|/|r0| 0.016 vs 0.017).
Costs 2 x 20 stored vectors per solve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
std::normal_distribution and std::uniform_real_distribution are
implementation-defined, so libstdc++, libc++ and MSVC turned the same
std::mt19937 seed into different numbers: the shaken model behind the
balanced X-ray weights (and with it the whole weight ladder), the B and U
shakes, the trial kicks and the random rotations of MR and the validation
null all differed between Linux, macOS and Windows.
PortableRandom.h draws straight from the engine's 32-bit outputs: a 53-bit
uniform in [0, 1) (Python's random.random() construction), a uniform in
[a, b), and a Box-Muller normal. The six sites keep their seeds. The trial
kick also draws its three components in separate statements, since the
order of a constructor's arguments is unspecified.
RefinementPortableRandom_pinned pins the first draws to values computed
with Python's random module on a MT19937 seeded as std::mt19937 is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
The field's search is iterated conditional modes over the residues, one after another, and
each step evaluates -ln L over every working reflection for each combination of states. On a
large model it ran on the main engine's card alone while the others idled: a 242k-atom model
with 45 NCS copies and 663k reflections spent 1743 s of a 2392 s refinement in it on one of
four T4s.
-ln L is a sum over the reflections, so MakeSideChainFieldEvaluatorGPUs gives each card a
contiguous part of them and adds the parts' sums in card order. The search, its order and its
decisions are unchanged; only the last bits of each sum move, where the parts meet. One card
keeps the evaluator as it was.
On that model, 4 x T4: the field 1743 -> 456 s, the refinement 2392 -> 1119 s, the refined
model bit-identical (same R-free, same md5). Two small models refine bit-identically, in the
same time.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
The calibration of the X-ray curvature in the positions (two evaluations, fixed seed) depends only on
where a trial starts: the model, whether its B factors are refined, the displacement hypothesis, the
kick and the first blur - not on the weights. The trials of one RunTrials() call that agree in all of
these now share it: the first to need it makes it, the others wait for it. The scale and the riders
are bit for bit what each trial would have made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
On the CPU the trial slots share the run's threads, so a trial run ahead of the decision that needs it
takes threads from the trials that are needed. Run them ahead only on the GPU, where a spare slot is a
card of its own. The outcome is unchanged: trials run ahead and not needed were already discarded with
their evaluations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
RunTrial applied a TLS hypothesis's bodies (ApplyTls, which bounds each atom's total B by b_floor)
before its first refit or restore, so the floor was whatever the slot's previous trial had left:
the same trial gave a different result (cytC R-free 0.3492 against 0.3494) depending on which slot
ran it and after what. The floor is now the start's (the fit's), the same in any slot.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdHHZ4cogS1mrGUtfN7
Refinement runs after every decision the battery scores, so leaving it out changes no verdict; the
refinement arm covers it separately. A large model's refinement took most of the validation job on
the runner's card (4nwv about 50 of its minutes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
When the held-out test sends a module back to where it was given, the placement is refitted with that module
held. The held module already fixes the frame the other modules are placed in; the module gauge (their moves
zero in sum) fixed it a second time, and the fit, pulled between the two, crawled: on an 8-panel MPCCD run with
one panel held, 53/51/101 Levenberg-Marquardt iterations a round (the last at the iteration cap) instead of
3-7, 65 s of a 94 s geometry step, and modules pressed against their bounds (a turn of exactly the 1.5 degree
limit, a shift past 12 px). Without the gauge when a module is held: 7/4/3 iterations, the refit 7 s, the
median spot residual 0.58 px instead of 0.66 px, the turns below 0.1 degree.
Still battery against rc188-final-still, the four sets where a module is held (the others never reach the
refit): sacla243 304.5 -> 226.8 s (step 2 92 -> 35 s), d_min 1.45 -> 1.35 A, R_split 3.35 -> 2.85%,
R_free 0.1785 -> 0.1754; euxfel83, euxfel152, alvra103_int16 draws (R_free within 0.0014).
Test: RefineGlobalGeometry_ModulePlacement_OneHeld (moves that do not sum to zero, recovered with one module
held).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The per-residue Cholesky substitutions ran one thread per residue, a chain of m^2 dependent steps
each: on a small model the device solve took 15 ms against the host's few ms (cytc refinement
6.0 -> 9.7 s). Each block's M_g^-1 is now formed on the host in double (from the same LLT,
symmetrised) and applied on the card one thread per element, a row of the inverse times the
block's part of r in float pairs: cytc 5.1 s, 4ybb 497 -> 344 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This reverts commit 6f6e57b36 for the owner's decision. Bisected on 2026-11-17 across the
rc188 builds: the padding alone moves results on knife-edge sets (nacl F 4 3 2 -> P 1,
d_min shifts on two open and one private set, one private space group), where rc187,
pocketfft, the CPU-refinement work and the build just before the padding all agree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
`const auto &fixed_system = experiment.GetIndexingSettings().GetFixedLatticeSystem()`
bound a reference into the IndexingSettings copy GetIndexingSettings() returns, which
dies at the end of that statement. Reading it afterwards is use-after-scope: built
without LTO the stack slot still held the empty optional, but with -flto=auto (the CI
build) it read as engaged, so every rotation run without -C/-S behaved as if the
crystal system were fixed - no triclinic pseudo-symmetry refinement beside a promoted
candidate (has_tri = false), hence no unconstrained alternative for the first pass's
false-promotion demotion. On a triclinic crystal whose metric is pseudo-monoclinic
the first pass then lost the true cell (0/60 validation frames on the promoted cell)
and the run merged an axis-doubled cell; the same build without LTO, on the same
machine, merged the deposited one.
GetFixedLatticeSystem() now returns the optional by value, so no caller can keep a
reference into the temporary settings, and the call site holds a copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The minimiser's initial inverse Hessian (20 preconditioned CG iterations per direction) ran on the host
apart from the J^T J product, which was uploaded and downloaded every iteration. Now the whole solve
runs on the card: the product with the X-ray diagonal, the per-residue Cholesky preconditioner (the
diagonal as 1x1 blocks for B), the vector updates and the dot products, the vectors and the scalars
staying on the device; one transfer in and one out per solve.
Single precision only, Metal-portable: every vector and sum in float pairs (double-float arithmetic),
the dot products in a fixed number of partial sums reduced in a fixed tree, no atomics, so a run is
reproducible. The U and TLS minimisers keep the CPU solve.
Test: RefinementNewton_solve_GPU_vs_CPU (device CG vs ConjugateGradients, and bitwise repeat).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Refinement: L-BFGS on the Gauss-Newton normal matrix with a per-parameter trust floor and a
per-residue preconditioned CG, one protocol on the CPU and the GPU (no population, two
rounds), the GPU Gauss-Newton restraint product, and CPU twin arms splitting threads only
where a twin law exists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
This reverts commit f8eb9ede5. Repeated alternated timings on a quiet machine show it
performance-neutral (refine wall -4..+1%, [refine_bench] unchanged, the kernel's samples
1866 vs 1907), so the simpler walk stays. The grid was bit-identical either way; the
[refine_bench] benchmark test is kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
A TLS trial's first macro-cycles were where a 1e-7 perturbation of the X-ray gradient still grew (cytc
end to end: R-free 0.3410 against 0.3460); with the floor on the U and the TLS parameters as on the
positions and B, 0.3411 against 0.3421.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Without one the twinned arm ends at once and its half sat idle (found by lane t: about 6 of 12 threads
busy through the whole refinement). Identical R-free and evaluations; at 12 threads thau 125 -> 92 s,
9zlo 143 -> 90 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Traced: with the X-ray curvature per atom as the only curvature of a soft mode (a high-B side chain
turning about a single bond, which the restraints leave free), the normal matrix's inverse made steps
of tens to hundreds of A there, and the global step bound then scaled every other atom's step down to
a few percent; a rounding difference of 1e-7 in the gradient moved a trial's R-free by up to 0.004.
- Each atom's diagonal is raised, once per call, to at least |g_i| over the step bound, so that such an
atom's step is held by its own curvature: the first trial's iterations take the full step, and a
1e-7 perturbation stays at 1e-7.
- The CG is preconditioned per residue (the residue's own J^T J and X-ray diagonal, by Cholesky), and
runs a fixed 20 iterations instead of to a tolerance, whose iteration count a rounding difference
can flip.
- The per-atom 3 x 3 blocks (CPU and GPU) are gone with the preconditioner they served.
GPU, 19 sets, against the rc188-merge protocol and minimiser: R-free -0.0014 mean, -0.0008 median,
worse by > 0.002 on 2 and better on 7 (the experiment's build); 158k -> 47k evaluations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A fixed workload (20 evaluations with the gradient, 12 and 48 threads, every atom isotropic
and then half of them anisotropic) that compares CPU refinement kernels across builds in
about a minute, independent of the path an optimiser takes. Hidden, run by hand with
"[refine_bench]".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
The points of a row within an atom's radius are one run (r2 falls and then rises along
the row), so RowRun() finds its ends from the roots of r2 = max_r2 and moves them onto the
points the exact single-precision test takes. The row's points are then filled without a
test in the loop, and AtomDensity adds them to the grid a run of consecutive grid points at
a time instead of by a gathered index per point. The fill and the adds now vectorise with
-march=x86-64-v3.
The points, their arithmetic and the order of the adds are unchanged, so the grid is the
same bit for bit (ModelDensityGPU_BitIdenticalToCPU holds); refinement results do not move.
The same run applied to GatherMapPoints() was slower: its rows are about ten points long,
and finding the run cost more than the well-predicted branch it removed. Vectorising the
gradient reductions by summing in lanes was faster (-16% in AtomGradientsFromMap) but
changes the summation order, which moved the optimiser's path and a final R-free by 0.003;
both were left out.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJz9sA2BQBEgKkuVuL
The L-BFGS's and the conjugate gradients' vector sweeps run on the threads, their dot products folded
in fixed blocks so that the bits do not depend on the thread count; the Jacobian is taken where the
minimiser stands once per direction rather than checked at every product. On a ribosome (4ybb, refine
arm, GPU, quiet) 491 s against 455 s for the former minimiser and protocol (R-free 0.2442 against
0.2472, 1820 against 4790 evaluations): the host side of the normal matrix is what a model this large
still pays for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The minimiser's initial inverse Hessian is the target's sparse normal matrix, solved by conjugate
gradients: the restraints' Gauss-Newton J^T J in full (RestraintNormalMatrix, on the slot's threads or
its GPU), the X-ray target's curvature per atom on the diagonal (calibrated once per trial), the U
restraints' exact Hessian, and per TLS body a dense 20 x 20 block. The L-BFGS pairs learn what the
matrix leaves out (the X-ray coupling of neighbouring atoms). 10 iterations for the positions and 7
for B, U and TLS per macro-cycle, instead of 30 and 20.
The protocol is the same on both paths: no population (kept as a setting, off by default), two
rounds; lean_on_cpu is gone.
Measured, refine-only, 19 sets on the GPU against the former GPU protocol (population, 2 rounds,
plain L-BFGS): R-free -0.0005 mean, -0.0012 median, worse by > 0.002 on 2 and better on 6, -LL free
lower on 11; 158k -> 49k evaluations. The same protocol with the plain L-BFGS on the CPU's former
protocol took 76k. Quiet single runs at 32 threads: cytc CPU 113 s (was 114 s with one round), GPU
6.1 s (10.9 s); myob CPU 41.5 s (53.1 s), GPU 4.1 s (5.1 s).
The damped Newton steps, the joint xyz+B target, the X-ray coupling term, the adaptive stop and the
experiment switches of the prototype are removed: pure Gauss-Newton steps converged each trial further
than the protocol's decisions were made for, R-free +0.0005 to +0.0022 in the mean of the 19 sets. BRestraintCurvature and
the L-BFGS's diagonal and Oren-Shanno scaling are gone with them.
Tests: RefinementMinimiser_quadratic on the new interface, RefinementMinimiser_conjugate_gradients,
RefinementRestraints_gauss_newton (J^T J against the Hessian near the ideal geometry).
Docs: CPU_DATA_ANALYSIS_REFINEMENT 15.6 and the population, references, ACKNOWLEDGEMENT (Hestenes &
Stiefel 1952, the normal matrix after Murshudov, Vagin & Dodson 1997).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
With the engine on a GPU, RestraintNormalMatrix hands At/Multiply/Diagonal/Blocks to
RestraintNormalMatrixGPU (RefinementNewtonGPU.{h,cpp,cu}), on the engine's card on a stream of
its own at the highest priority. J is built on the device, one thread per restraint from the
REFINE_HD terms in float, into a fixed layout worked out once from the set (rows the CPU drops,
E = 0, are kept as zeros). A product is two gathers, J v per row and J^T t per parameter atom,
without atomics, in a fixed order.
Precision: J in float, v/t/hv as double-float pairs (error-free product and 2Sum per term).
Plain float sums are not enough: J v is a small difference of large terms in the soft directions
the X-ray term decides, and with plain float the outcomes moved beyond the CPU's own noise on a
model with stiff restraints (weight choice 2/10 vs 8/10 at the CPU's value over jittered runs).
Float J with exact accumulation matched the CPU's spread, as do the double-float pairs.
On a large rotation set (GPU, Newton joint, relative stop 0.05): product thread-time 1008 s ->
38 s, jacobian 114 s -> 8 s, refine wall 191 s -> 32 s under load; evaluations and R-free within
the spread a 1e-12 relative jitter of the CPU product gives.
Interface: RestraintNormalMatrix now takes the Context (for the device) instead of n_atoms and
nthreads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The dataset column of _plot.txt is now tied to its file in the file itself:
the header has one "# dataset k: <absolute path>" line per input, numbered
as the column. A joint merge lists every sweep given (joined or not), so k
is the report's SWEEP_k_; a run of one input names its one file.
A run of several masters read as one stills run lists each input in report
section 1 as DATASET_nn = <images processed> <indexed> <file>, nn the plot's
dataset number. (INPUT_nn is not reused: its rows carry merged/left-out
counts of --mode scale that a stills run does not have per input.) The
per-input indexing log lines are now made from the same counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The per-image table numbered its rows by the run's processed-image ordinal
only, which is unusable for a run made of several files. It now carries
three numbering columns: ordinal (the run's row, as before), dataset (the
input, from 1 in the order given) and image (the image within that input).
The header describes each column before the legend line.
- Several stills masters read as one: rows are renumbered per master.
- Joint merge of sweeps: the table covered only as many rows as the
reference sweep's file held, so the other sweeps' frames were missing;
it now has one row per frame of every merged sweep, with each sweep's own
angle and per-image analysis (bkg, resolution, spots) from its own run
and the scaling columns from the joint merge.
- The angle is a per-row value now, so each sweep reads on its own goniometer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Still behind RUGNUX_NEWTON=1, off by default.
- The restraints' Hessian is their Gauss-Newton J^T J (RestraintNormalMatrix: the Jacobian's rows and, per
parameter atom, its entries, so that J v and J^T t are both gathers on the slot's threads); the
finite-difference Hessian, the hybrid L-BFGS and the step clipping are gone.
- U: the X-ray diagonal plus the U restraints' exact Hessian (quadratic in U); TLS: per body the dense
20 x 20 A^T C A from TlsBody::Derivatives.
- RUGNUX_NEWTON_JOINT=1: positions and isotropic B in one target, X + R_geometry / w + R_B / adp_w.
- A solve stops where an iteration gains less than RUGNUX_NEWTON_RELSTOP of the call's decrease, at most
RUGNUX_NEWTON_XYZ / _B iterations (30 / 20); per-atom 3 x 3 block preconditioner for the positions.
- RUGNUX_REFINE_PROTOCOL=<population>,<rounds>,<macro-cycles>: one protocol on the CPU and the GPU, the
population not capped by the slots.
- ACKNOWLEDGEMENT: Levenberg (1944), Marquardt (1963), the sparse normal matrix after Murshudov et al. (1997).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
PutMaskOnGrid() takes a `gpu` flag (default on, as PutModelDensityOnGrid's): on macOS the shrink by
the 0.8 A radius (ShrinkMask) runs on the Apple GPU (ShrinkMaskMetal.mm/.metal), one thread per point
applying the CPU's rule - a masked point becomes solvent where a near offset is solvent, or a far
offset is a solvent point with a masked near offset - read from the mask as it was, with the CPU's
offsets; the mask and its copy handed to the GPU in place where page aligned. The atoms, the
symmetrize and gemmi's island removal stay on the CPU. --no-gpu, no device, or a failure (one
warning) keep the CPU's loop.
ShrinkMaskMetal_BitIdenticalToCPU ([portable][Metal]): clustered atoms, 0.4 A mask grids of a
triclinic P1 and a P212121 cell - every point the same bits.
M6 Mac mini, rugnux --mode refine (open-arm sets' merged MTZ and refined model), on top of the two
previous Metal commits: 9s02 184.2 -> 175.6 s, 9zlo 41.3 -> 39.0 s; the refined models
byte-identical to the build before on both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Behind environment switches, off by default; nothing changes without them.
- RefinementNewton (RefinementMinimiser.h): Levenberg-Marquardt steps, (H + lambda D) d = -g by
preconditioned CG, accepted on the target.
- H = the X-ray curvature per atom as a multiple of I (XrayXyzCurvatureShape, the B shape's
counterpart, calibrated once per trial by central differences: 2 counted evaluations) plus the
restraints' Hessian: by differences of their gradient (RUGNUX_NEWTON=1) or Gauss-Newton
(RUGNUX_NEWTON_GN=1, RestraintGaussNewton: J = g / sqrt(2E) per record, planes per atom).
- RUGNUX_NEWTON_XYZ / _B iterations per macro-cycle (10 / 6), _CGMAX, _CGTOL, _CLIP, _RELSTOP,
_TRACE; RUGNUX_NEWTON=2 an L-BFGS with the Newton H as H0 (worse, kept for the record).
- RUGNUX_REFINE_STATS: MINSTAT/TRIALSTAT/TRIALSSET/WEIGHTSTAT lines on stderr (evaluation budget).
- EvaluateRestraints: the per-record terms moved into RecordTerm (same operations, same order).
Measured (refine-only, CPU one round, 6 sets): evaluations -40..-63% at equal R-free on average;
on the GPU protocol R-free worse on 4 of 6 sets. Not for production as is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
RESULT-CHANGING on the CPU (ulp level): the B factor's exp in EvaluatorCPU::Contribution is
BraggExp (bragg_integration/BraggExp.h, at most 0.53 ulp) instead of the libm expf, so that the
Apple GPU, whose exp no host function reproduces, computes the same bits.
MakeSideChainFieldEvaluatorCPU takes a `gpu` flag (the refinement passes settings.gpu); on macOS the
evaluator's Contribution() and the per-state sums of Evaluate() then run on the Apple GPU
(SideChainFieldMetal.mm/.metal): one thread per reflection and state, as SideChainFieldGPU.cu's
contribution_kernel, the evaluator's tables uploaded once, states in batches of 32 so that the
GPU's buffer stays small beside the states' own. Fast math and contraction off - also over the
BraggExp prepended to the kernel source, which a first version left contracted (91 of 60000
values 1-2 ulp off). The likelihood over the combinations stays on the CPU (double). Where the
GPU is hidden (--no-gpu) or fails (one warning) the CPU does the same sums.
SideChainFieldMetal_BitIdenticalToCPU ([portable][Metal]): 20000 reflections, P212121, 70
six-atom states (three batches): every contribution the same bits, every combination's -ln L equal.
M6 Mac mini, rugnux --mode refine of an open-arm set's merged MTZ and refined model, on top of the
Metal gradient (previous commit):
9s02 (20355 atoms, 306298 reflections, 5 NCS copies): side-chain field 149.0 -> 30.4 s, refinement
309 -> 184 s wall; R-work 0.2254 / R-free 0.2390 both.
9zlo: field 7.6 -> 2.4 s, 45.8 -> 41.3 s; R unchanged.
--no-gpu on 9zlo: the refined model byte-identical to the build before (the field moved nothing).
Not run here: SideChainFieldTest.cpp's [Refinement] cases (jfjoch_test is not built on macOS).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A space group given alone no longer fixes the lattice. The first pass asks the
sub-lattice / twin-composite hypotheses and the long-axis hypothesis as in a
run without -S, and a lattice they adopt is then put in the fixed group's own
Bravais class (LatticeSearchForGroup), as the indexer does with the cell it
finds. Only a cell given with -C (with or without -S) fixes the lattice and
its setting; those runs are unchanged (.hkl byte-identical on the checks).
On the private triclinic small-molecule set, -S P1 landed on 2x/4x supercells;
it now gives the primitive cell on 5 of 7 datasets (I/sigma up to 2-5x higher).
-S P12/c1 on dnba, refused in rc187 for a metric violation, now merges.
The doubled-cell hypothesis stays off under -S: without the space-group
search there is nothing to re-express its doubled cell on its centred lattice,
and adopting it doubled the primitive cell.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A lattice is promoted to a centred or higher class on its metric alone, and the geometry and
integration are held to that class. Where the space-group search then finds no symmetry (P 1),
the constraint has nothing behind it. A stored rotation run in that state is now made once more
from the start with the lattice held triclinic (IndexingSettings::FixedLatticeSystem, the hold a
triclinic -C gives), and the merge whose CC1/2, averaged per shell on Fisher's z, is higher is the
answer.
Measured on a triclinic small-molecule set indexed C-centred monoclinic (angles held at 90 deg,
0.05 deg inside tolerance; the C lattice took 45% of the spots against 63% free, and a 27 deg
segment indexed in the other hand of the pseudo-2-fold): bare run CC1/2 0.203, R_meas 41.3%,
I/sigma 11.1 (UNUSABLE) -> CC1/2 0.987, R_meas 10.5%, I/sigma 12.7, identical to -S P1; one extra
run (~3 s). No set of the rc186/rc187 full, private, multisweep or private-multisweep battery runs
ends in P 1 on a lattice other than triclinic P, so no battery set reaches the new path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
- usage: -C without -S fixes the crystal system from the angles as typed, on the typed axes;
RUGNUX_CLI.md says the axes as typed too.
- RUGNUX_ADVANCED.md: a reference -z lifts the group check of several _process.h5 files as
well as the cell check (it fixes the space group too, rugnux_cli.cpp).
- RUGNUX_CLI.md: --mode refine refines against the input reflections; -z is not used there and
a reflection file given only with -z does not select the mode.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
- Plot.jsx: (factory.default ?? factory)(Plotly), so an interop that unwraps the CommonJS
default also works.
- RugnuxPage: the square shell plots took their side from the whole right panel's height, so
with the checks box (up to 40%) above them they overflowed on height-limited windows; the
side now comes from the plots' own division.
- jfjoch_downloads.js: return where the version element is missing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Comparing rotations only passed a model group with a pure 2-fold where the data's group has a
2(1) (P 1 2 1 against P 21 21 2): the model was then refined in its own group, its copies
overlaid where the data have none, and the twin arm switched off. Each operator must now be
one of the larger group's once the subgroup's origin is moved by some p (an operator (R, t)
reads (R, t + (R - 1) p)), so P 1 21 1 in P 21 21 21 (origin z + 1/4) still passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The table has no character of an A/B-centred class, nor of a C/F/R one outside its systems,
so 6 of the 7 searches were empty on a triclinic -C cell and 3-5 elsewhere. Same result: a
randomized probe over P/C/I/F/R lattices of every system (3000 cells, rotated, near-equal
lengths) never found a class outside the new lists.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
CellMappingOperators admits every relabelling within 5% / 3 deg of the given cell, so on a
cell with two lengths that close more than one maps; ops.front() took the first in
enumeration order (the identity where it qualifies) even where another put the cell on the
typed axes better, e.g. a pure 2-fold given on the shorter of two near-equal axes. The
nearest one is now taken (the OnGivenAxes distance), the identity on a tie.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
RESULT-CHANGING (ulp level; owner approved for Metal ports where exactness costs speed)
AtomGradientsFromMap() takes a `gpu` flag; the CPU engine sets it, and on macOS the sums of every
atom's box against the FFT method's map then run on the Apple GPU (AtomGradientMetal.mm/.metal):
one SIMD group of 32 threads per atom, the lanes taking the box's points in turn, as
RefinementGPU.cu's atom_gradient_kernel does under CUDA. Each atom's box, radius and Gaussians are
the CPU's (built on the host as the CPU loops build them), and a point's place and the radius test
are spelled as GatherMapPoints() spells them with fast math and contraction off, so the GPU takes
the same points; the sums are in single precision and in another order, so the gradient is not the
CPU's bit for bit. Where the GPU does not take it (rugnux --no-gpu, no device, a box wider than the
cell) or fails (one warning, then the CPU from there on) the CPU loops run as before.
Parity (AtomGradientMetal_MatchesCPU, [portable][Metal], 2000 atoms, a quarter anisotropic, P1
triclinic and P212121): worst |GPU - CPU| over the largest |CPU| of that derivative 0.6-1.0e-6 for
x, y, z, B and 2.1-7.6e-6 for U; held to 1e-5 and 5e-5.
M6 Mac mini, rugnux --mode refine, an open-arm set (9zlo, its merged MTZ and refined model):
90.6 s -> 45.8 s wall; R-work 0.2697 both, R-free 0.2936 -> 0.2938. With --no-gpu the refined model
is byte-identical to the build before (115 s: --no-gpu also takes the density gather off the GPU).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The histogram length is ceil(2 pi * 2 * max_cell / d_high): 3142 = 2 x 1571
for rugnux's defaults (500 A, 2 A), 1571 (prime) for the broker's (250 A,
2 A), and almost never smooth otherwise - 217 of 222 lengths over the cell
bounds 100-1200 A at 2 and 3.5 A. pocketfft does such a length by Bluestein.
It is padded on to the next length whose factors are 2, 3, 5 and 7 (3142 ->
3150); the bin spacing is unchanged, only the zero padding grows by 0.25%.
The indexer's transform (16384 directions, one thread), on a loaded box
(load 11): 3142 0.75 s, 3150 0.07 s - per FFT-indexed image. Rotation sets,
--no-refine, CPU -N 16: 47.2 vs 47.2 s and 39.9 vs 39.3 s (few images are
indexed); GPU 9.6 vs 9.3 s and 14.1 vs 14.7 s. Results: one lysozyme set
bit-identical on CPU and GPU, the other the same merge to rounding (same SG,
cell, I/sigma, R_meas, CC1/2, ISa; the pre-pass lattice moves by 0.2 A).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
"A near-extinct class is reported as a sub-lattice" built its crystal with the
weak class at 2% of the strong, which reads a modulation of 19.996x against
the 20x gate: FFTW's rounding of the Patterson put it over, pocketfft's under.
Swept on the same synthetic: depth 0.90/0.95/0.96/0.97/0.98 read
12.0/18.0/20.0/22.8/26.2x. The test now uses the weak class at 1% (26x), and
also checks the other side of the gate, which it did not before: a deep but
genuine pseudo-symmetry (weak class at 5%, 12x) is detected and NOT reported
as near-extinct.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
FFTW is no longer fetched, built or linked - not even by jfjoch_test. The
JFJOCH_USE_FFTW define and its #ifdefs are gone: the CPU FFT indexer, the
viewer's FFT chart and the rest are always built. The test that checked
pocketfft against FFTW now checks it against a direct double-precision DFT
(forward) and a round trip (backward), on the same shapes.
The user-facing name stays: -X FFTW, the OpenAPI enum value, the CBOR and
HDF5 strings - the CPU FFT indexer is still called FFTW, only the
implementation changed; the CLI page says so. Under MSVC pocketfft is told
explicitly to run scalar (it already decides so itself: its vector code needs
the GCC/Clang vector extension).
Licences: licenses/fftw.txt, its COLLECT.sh line, the THIRD_PARTY_NOTICES row
and note, and the viewer's embedded copy removed; docs/THIRD_PARTY_NOTICES.md
regenerated with update_version.sh's own sed block. CLAUDE.md and the docs
that named FFTW as the library now name pocketfft.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The twinned arm waited for the untwinned one on the CPU ("a second arm only shares its threads"), so a crystal
with a twin law refined twice in a row, and each arm's phases with few trials left threads idle. Where nothing in
an arm depends on its threads - no population, whose members are as many as the trial slots (the CPU protocol of
RefinementSettings::lean_on_cpu) - the arms now run side by side on half of the threads each. The same refined
models bit for bit (cytC, 7ris, myob and the live cytC placement); --mode refine --no-gpu -N 16: cytC 150 -> 137 s,
the live cytC set 182 -> 146 s, 7ris 154 -> 141 s, myob 72 -> 65 s (the last three also with the commit before).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
What Evaluate() did on one thread between the parallel parts - on a 1.2 A case a quarter of an evaluation on 16
threads (eval 44.6 ms, 105 ms on one):
- the target index of each composed term in the half-l coefficient grid worked out with three integer modulos per
term per evaluation: now once, at the engine's construction (Scatter), the terms added in the same order;
- the coefficient grid and the map allocated (and their pages faulted in) every evaluation, the map copied from
the template grid and scaled on one thread: now kept by the engine, zeroed and scaled on the threads, the inverse
transform written into the map itself;
- the density calculator's grid made anew per Compose(): now the engine's own, kept;
- MapToFPhi()'s conjugate-and-scale pass over the coefficients on one thread: now on the call's threads.
The same values in the same order: the refined models of cytC, thau and 9zlo are bit for bit the ones before.
Engine benchmark (16 threads): 1.22 A thau 44.6 -> 30.5 ms per evaluation, cytC 32.6 -> 29.7 ms. --mode refine
--no-gpu -N 16 (on the CPU protocol): thau 115 -> 84 s, 9zlo 115 -> 108 s, cytC 154 -> 150 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
RefinementSettings::lean_on_cpu (on): where the refinement runs on the CPU - no GPU wanted or none to run on - the
population (2a.) and the second round (4b.) are left out, for about half of the target evaluations. Measured
refine-only on 19 sets (cytC + its live set, 12 AF-start placements of the af-bench, 5 refine-arm depositions), the
same protocol on the GPU where it is fast to compare (the outcome is the CPU's within the path noise): 158204 ->
76424 evaluations; R-free unchanged on 8 sets, median +0.0012, worse by 0.0012-0.0062 on 9 and by 0.0147 on the
live cytC placement (an AF start the population helps), better by 0.0023 on one. Each part alone: population off
-29% of the evaluations, worse on 5 of 19 (max +0.0147); one round -22%, worse on 8 (max +0.0060).
Kept, measured and rejected: the twinned arm left out as well (a third of the evaluations, +0.0012 median) fails
Refinement_twin_decided - a crystal 30% twinned is then refined as untwinned - so both arms still run; the ADP
hypotheses off cost 0.007 in the median (max 0.044); max_steps 2 saves 7% only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The CPU evaluator summed each state's structure factors in double precision with glibc's sincos per atom,
operator and row - 42% of a CPU refinement of a 1.2 A isotropic model, the field 149-157 s of a 263 s refinement
(the GPU's 1.3 s) - recomputed every state's contribution again for a move, and summed the likelihood of the
combinations one after another. Now:
- the contribution is the GPU kernel's arithmetic (contribution_kernel: single precision): per block of rows, the
atom's form factor (the element's part tabulated per row), then per operator the phase's sine and cosine from
SinCos2Pi() - the phase taken to [-1/2, 1/2] turn by its nearest integer and the half angle's Taylor series,
max error 4e-7 (sincosf of the unreduced phase: 1.5e-5) - in loops the compiler vectorises;
- a move computes only the states it takes out and puts in;
- the combinations' likelihoods are summed side by side, each over the rows in order.
thau (AF start, --mode refine --no-gpu -N 16, population 0, one round): the field 148.9 -> 12.2 s, the same
energies and moves (0 of 140), the same refined model (R-free 0.2611); refinement wall 280 -> 125 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
On the CPU the refinement is still two orders of magnitude slower than on a GPU (cytC with its reference
model, --no-gpu -N 16, after the CPU commits before this one: 378 s against 11 s on an RTX 5080; 1610 s before
them), and short of the order of magnitude the owner set as the bar for keeping it on. So a run with no GPU (none,
or --no-gpu) now places and scores the model as before and leaves the refinement out, with a log line and a
Note saying so, and the report's REFINE= NOT_PERFORMED with REFINE_REASON "no GPU (--refine runs it on the CPU)".
--refine (new) runs it on the CPU; --no-refine is unchanged, a GPU run refines as before, and --mode refine (an
explicit request) refines on the CPU as before. RugnuxConfig::refine_without_gpu carries it (the viewer, which
sets refine, follows the same default). Docs: RUGNUX_CLI.md (--refine, --model), RUGNUX_MODEL_AND_REFERENCE.md.
Checked: cytC -e 300 --no-gpu -> REFINE= NOT_PERFORMED with the reason; the same on the GPU -> PERFORMED.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
RunTrials gave each running slot settings.nthreads / slots threads, rounded down: 12 population members on 16
threads ran on 12, a ladder of 14 trials on 14. And a slot that had no trial left kept its threads idle while
the longest trials ran on (a ladder's B-held trials end well before the ones refining B). On the CPU the
remainder is now dealt to the first slots, and a slot that has run out of trials adds its threads to a shared
count that the slots still running take up at their next target evaluation (Context::TakeSpareThreads). A GPU
engine's slots are shared out as before. The outcome does not depend on the threads.
cytC --mode refine --no-gpu -N 16: the same refined model bit for bit; refinement 224.7 + 219.7 s -> 190.7 +
186.3 s, wall 445 -> 378 s, the process busy on 13.5 cores on average (was 10.3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
AtomGradientsFromMap() gave each thread one contiguous range of atoms; an anisotropic atom or one of high B costs
several times another, and the threads of the cheap ranges waited for the rest. The atoms now go in blocks of 16
off a shared counter (ParallelFor), each thread with scratch of its own; every atom's derivatives are the same.
Microbenchmark (2658 atoms, 144^3 grid): 16 threads 20.4 -> 17.2 ms, 8 threads 21.8 -> 17.9 ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The CPU engine and the bulk-solvent mask kept the thread count they were made with: the main slot's all of
them, every other weight-trial slot's one (settings.nthreads / slots). RunTrials and the stages that hand the
threads back set Context::nthreads, which the engine never read - so where fewer trials ran than there were
slots (the trajectory's steps run 4, TLS/U hypotheses, the last trial of a set) the trials in the other slots
ran on one thread each and the step waited for them; top showed the process at 100-500% for most of a run on
16 threads. Context::SetThreads() now passes the count on (RefinementEngine::SetThreads, a no-op on a GPU
engine, and BulkSolventMask::SetThreads), and the engine's and the mask's FFTs are split over the same
threads (MapToFPhi/MapFromFPhiExecute take nthreads; pocketfft splits lines, the result is the same).
The outcome does not depend on the threads: cytC --mode refine --no-gpu -N 16 writes the same refined model and
maps bit for bit, R-free 0.3400 / 0.3366 (two hypotheses) as before.
Refinement time, with the gradient commit before it: 909.6 + 700.7 s -> 224.7 + 219.7 s (wall 1611 -> 445 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
AtomGradientsFromMap() walked each atom's box and summed the Gaussians of every point in one scalar loop
(float -> double conversions, an integer modulo per point, a push of nine accumulators per Gaussian for an
anisotropic atom), which perf showed as 81% of a CPU refinement. The walk now gathers the points within the
radius (map value, place, r^2; the u index wrapped by a counter), the Gaussians' terms are worked out a
Gaussian at a time in loops the compiler vectorises (the exponent in a loop of its own, as the two together do
not vectorise), and the derivatives are then summed over the points in the walk's order, the Gaussians of each
in turn - the same terms in the same order, so the gradient is bit for bit the one before.
Microbenchmark (144^3 grid, 2658 atoms, 2442 of them anisotropic, one thread): 0.415 s -> 0.128 s per gradient.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Every CPU FFT site - the FFT indexer's batched 1D r2c, the beam-centre 2D r2c/c2r,
the model-path 3D transforms (MapToFPhi, MapFromFPhi(Execute), MapToFPhiInParallel),
the tNCS Patterson c2r and the viewer's 1D chart - goes through common/CpuFFT, a
plan-free wrapper over the pocketfft header already vendored with gemmi. The header
is included under a namespace of its own: gemmi includes it with
POCKETFFT_NO_MULTITHREADING, and the linker otherwise merges the two sets of
templates (MapToFPhiInParallel with 4 threads gave garbage until it was renamed).
MapToFPhiInParallel uses pocketfft's own threads (one call per stage) instead of a
ParallelFor over FFTW plans. The programs no longer link fftw3f; FFTW stays fetched
for jfjoch_test only, where [fft] checks pocketfft against it (rel. error < 1e-5).
Measured (Ryzen 5950X, --no-gpu -N 16, one run each, against rc187 aa898ac0f):
cytc_x10sa --model --no-refine 100.7 -> 99.5 s; euxfel83 first 6 files --no-refine
81.7 -> 83.3 s (noise level); full cytc refinement 1514 -> ~1399 s. Merged results
identical (same hkl md5, SG, cell, R-work/R-free/CC of model validation).
Microbench, single thread, FFTW_ESTIMATE as the call sites plan: pocketfft 0.4-0.6x
FFTW's time on the 2D/3D shapes; with FFTW_MEASURE the two are equal on 3D.
Indexer length 3142 = 2 x 1571 is 2.4x SLOWER on pocketfft (Bluestein); a smooth
length (3150) is 0.06 s vs FFTW's 0.25 s at 3142.
BeamCenterFFTGPU_MatchesTheCPUEngine: gate mismatches over 4M centres on the
threshold 7 (FFTW) -> 17 (pocketfft); bound raised to 30. pocketfft's float error
is FFTW's or lower (rms 1.4e-7 vs 1.5e-7 against double).
Known: "A near-extinct class is reported as a sub-lattice" [tncs] fails - its
synthetic reads modulation 20.0x against a 20.0x gate; FFT rounding tips it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
With -C and no -S the typed angles fix the crystal system and the indexer holds the lattice in
it, but the metric-lattice re-ask after the P1 merge could still reindex into a higher class the
metric offers and adopt a group there (a cell typed with one angle at 90.03 fixed monoclinic,
indexed monoclinic, and ended tetragonal). That re-ask only ever commits a strictly higher point
group, so it is skipped when the system is fixed, as the centred-group reduction already is; the
search then stays within the system the cell names, as docs/RUGNUX_CLI.md states. Runs without a
fixed system (no -C, a measured cell, or -S) are unaffected.
Verified: that cell now ends P 1 1 21; -C with 90/90/90 still P 41 21 2 and a monoclinic -C on
a C2 set still C 1 2 1, both with p.hkl byte-identical to the rc187 binary.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The download pages and the landing page were rendered with links for `release`, but the page
script then repointed every button at the newest Gitea release, so a later release candidate
would have moved the 1.0.0 documentation's links. The script now looks up the release tagged
with the page's own version (releases/tags/<version>); if it is not published, the built-in
links stand. The install examples name 1.0.0 instead of an rc.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The notice said the viewer links Qt dynamically, which holds only for the Windows and macOS
packages; the Linux tarball is built against a static Qt 6.11.x (docker/ubuntu2404/Dockerfile).
State both, and for the static build the LGPL-3.0 section 4(d)(0) route: relink with a modified
Qt by rebuilding jfjoch_viewer from the complete source following the Dockerfile, with the Qt
source at download.qt.io. The file is maintained by hand (COLLECT.sh does not write it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The side-chain field's GPU evaluator allocates states x reflections on
the card with nothing bounding it; a failed allocation inside a sweep
took the whole refinement down. Its footprint (SideChainFieldDeviceBytes)
is now compared with the card's total before the evaluator is chosen and
the CPU evaluator runs where it does not fit, as the engines decide.
A round of the weights that is not kept restores the displacement model
it had displaced, so REFINE_ADP_MODEL names what the written coordinates
carry. A decay dropped for the surfaces fitted without it no longer
reports as dose-linear or per-sweep B.
The docs say the rounds decide the weights again once (two rounds in all)
and describe the B ceiling as the code sets it (Wilson-B floor, 1.5x above
a largest B of 200 A^2).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A joint run that processes a failed sweep again hands it the reference
sweep's measured cell as fixed_reference_unit_cell. Its angles are never
the literal 90 or 120, so the rule reading the crystal system off the
typed angles pushed every such rerun to P 1 (or, on a lucky bit pattern,
to a wrong system). The rule now fires only for a cell given with -C.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Nothing includes it (no source, HLS build script or CMake file in the tree
names it), and it carries Xilinx's proprietary "confidential" banner rather
than the Apache-2.0 terms of the other headers in fpga/include, so it should
not be redistributed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
- Files adapted from other projects carry both licences: StrongPixelSet.cpp,
SpotExtractorGPU.cu and the four GEMMI-derived structure_refinement files
"GPL-3.0-only AND MPL-2.0"; FFBIDXIndexerCPU.cpp, LMSolver.{h,cpp} and
xds-plugin/plugin.* "GPL-3.0-only AND BSD-3-Clause"; fpga/hls/parallel_stream.h
"CERN-OHL-S-2.0 AND Apache-2.0"; fpga/hls/icmp.cpp and ip_header_checksum.h
(HPCN UAM, with PSI modifications) "CERN-OHL-S-2.0 AND BSD-3-Clause".
- writer/H5FDpoison_sec2.{c,h} and tests/enospc_shim.c: "OR HDF5" is not an
SPDX id; now "OR LicenseRef-HDF5", with the text named (licenses/hdf5.txt).
The poison driver wraps the stock sec2 driver rather than copying it (only
the H5FD_class_t initialiser layout matches H5FDsec2.c), so the files stay
dual-licensed (OR), not derived (AND).
- 151 first-party files without a tag (CMake, shell, Python, frontend, docs
JS, a few C++) get one: GPL-3.0-only, CERN-OHL-S-2.0 for fpga/, fpga/hls/
and fpga/scripts/, with the copyright line of the neighbouring files and the
year the file was added. Generated and vendored code is not touched;
fpga/pcie_driver/postinstall.sh and preuninstall.sh are left untagged, as
they say they are taken from a blog post.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The embedded CCP4 Monomer Library subset is not an unmodified copy:
links_and_mods.cif and ener_lib.cif are cut down and metals.json becomes
metals.cif. The notices entry now says so and states how the LGPL-3.0
obligations are met (modified files stay LGPL, preferred form and the
generating script in the tree, the user can replace the embedded copy).
Each of the three derived files carries a CIF comment at its top naming its
origin, the modification and the date, and update_monomers.py writes that
comment whenever it regenerates them (the files still read with gemmi).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Ceres is pinned to commit 0c70ed3a1a2d (image_analysis/CMakeLists.txt), the
vendored LZ4 is 1.9.4, nlohmann/json 3.12.0, TinyCBOR 0.6.0 and GEMMI 0.7.5
(from the headers). Zstandard is dual-licensed BSD-3-Clause OR GPL-2.0-only
and is used under BSD-3-Clause, stated as the Mbed TLS row states its choice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
From a licence scan of the tree, code of other authors that had no
THIRD_PARTY_NOTICES.md row:
- fpga/hls/icmp.cpp, ip_header_checksum.h: BSD-3-Clause, HPCN Group UAM
(licenses/hpcn-uam.txt, the files' own licence comment)
- xds-plugin/plugin.{h,cpp}: adapted from Durin (Diamond Light Source /
Global Phasing, BSD-3-Clause); xds-plugin/LICENSE-durin.txt beside it,
collected to licenses/durin.txt, listed in the plugin's own notices and
JFJOCH_XDS_PLUGIN_LICENSES
- fpga/hls/parallel_stream.h: adapted from Vitis HLS hls_stream.h, Apache-2.0
- fpga/include/hls_burst_maxi.h: Vitis HLS simulation header; the file
carries no licence text and its upstream terms could not be established,
which the row says
- the four GEMMI-derived structure_refinement files, as an "adapted" row
- gemmi_gph/stb/stb_sprintf.h (MIT OR Unlicense), gemmi_gph/eig3.cpp
(public domain, after JAMA)
- slsDetectorPackage's bundled OpenSSL md5.c (Apache-2.0, collected from its
LICENSES/Third Party) and RapidJSON (MIT, notice kept by hand)
- the FPGA bitstreams: the Xilinx IP they contain, with the terms
docs/FPGA.md states
update_version.sh rewrites the new docs/FPGA.md link for the docs mirror.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The kernel module declares MODULE_LICENSE("GPL"), which the kernel reads as
GPLv2; the files carried GPL-3.0-only, which cannot be combined with the
GPLv2-only kernel. fpga/pcie_driver is PSI's own code, so it is relicensed to
GPL-2.0-or-later (MODULE_LICENSE("GPL") stays, it is the right tag for it).
- SPDX tags of all driver sources, and tags on its build/DKMS files
- fpga/pcie_driver/LICENSE (GPL-2.0 text), collected to licenses/GPL-2.0.txt
- installed into the DKMS source tree (CMake install and install_dkms.sh)
- LICENSE, docs/LICENSE.md, docs/FPGA_LICENSE.md and THIRD_PARTY_NOTICES.md
name the driver's licence; update_version.sh rewrites the new link for docs
CPack has no per-component RPM licence field, so the driver-dkms RPM still
inherits the package-wide "GPLv3" License: tag.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
molstar 5.11.0 -> 5.13.1, still an exact pin. The breaking changes between them
(createRenderTarget's depthStencil option, the volume-mask extension moved to
volume-tools/mask, the illumination glow/firstStepSize parameters removed) touch
nothing the structure viewer uses; its molstar/lib/... imports resolve as before
and type-check unchanged. The RugnuxStructureViewer chunk grows from 2845 to
3066 kB (gzip 799 -> 859 kB).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
vite 7.3.8 -> 8.3.8. @vitejs/plugin-react-swc 4.3.3 already accepts Vite 8 and
vite-plugin-svgr 4.5.0 / vite-tsconfig-paths 6.1.1 accept any Vite, so no plugin
major is needed; vite.config.ts uses no option Vite 8 removed or renamed.
One source change: Vite 8 resolves a default import from CommonJS the way Node
does when the importing package is "type": "module" - it is module.exports, not
exports.default. react-plotly.js/factory is Babel-compiled CommonJS, so the
factory is now taken as .default of the import; without that the app stopped at
start-up with "(0, x.default) is not a function".
vite build drops from ~24 s to ~1.5 s; dist/ shrinks from 6.11 MB to 5.91 MB
(index 3097 -> 2970 kB, RugnuxStructureViewer 2922 -> 2845 kB).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The residue-by-residue crossover names Nwachukwu et al. (2013) at the
algorithm and the blurred-data schedule names Jack & Levitt (1978) beside
Piela et al.; CPU_DATA_ANALYSIS_REFINEMENT.md's reference list gains the
three side-chain field papers already acknowledged in ACKNOWLEDGEMENT.md.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
tsc is now TypeScript 7.0 (installed as @typescript/native, an npm alias of
typescript@7), and the name "typescript" is an alias of @typescript/typescript6,
the side-by-side setup Microsoft recommends: TypeScript 7.0 ships no compiler
API, and @hey-api/openapi-ts (npm run openapi) emits the client through the
TypeScript API it imports as "typescript" (a peer dependency, so it cannot be
given its own nested copy). npm run openapi is unchanged.
tsconfig: moduleResolution "node" (node10) was removed in TypeScript 7; it is
now "bundler", which is what Vite resolves. tsc --noEmit passes without new
suppressions, in 1.6 s instead of 8.4 s; the bundle is byte-identical.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
npm audit before: 41 (3 critical, 17 high, 18 moderate, 3 low), 14 of them
in the production tree (vite, zod, plotly.js/maplibre-gl, js-yaml, postcss,
...). After: 0.
- In-major bumps: vite 7.3.8, zod 4.6.7, MUI 6.5.0, x-data-grid 7.29.13,
TanStack Query 5.107.1, React 19.3.1, plotly.js-cartesian-dist-min 3.7.0,
plugin-react-swc 4.3.3, @redocly/cli 2.78.0, types; transitive refresh
(plotly.js peer 4.3.0, postcss, browserslist, ...).
- Override js-yaml ^4.3.2: @hey-api/json-schema-ref-parser pins 4.1.1 (and
4.2.0 in the newest openapi-ts), inside the advisory range.
- Remove esbuild-style-plugin: unused (vite builds the frontend) and its only
fix was a downgrade.
No major bumps; the generated client is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
AddSugarLinks paired atoms by distance regardless of altloc, so a pyranose
and its partner both in conformers A and B also got the A-B bonds. A pair
whose altlocs are both set and differ is now skipped; a blank altloc still
pairs with any, as gemmi does. New test: a GlcNAc on an Asn in two
conformers links A-A and B-B only.
rugnux logs one warning when restraint files are given but the run refines
no model (no model, --no-refine, or a mode without refinement).
RUGNUX_CLI.md: the automatic --mode refine also applies when the model comes
from --model and the reflections are among the inputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Each blob of the refined mFo-DFc map now carries the anomalous difference map's peak height
(|F+| - |F-| on the model's phases less 90 degrees, in its rms): its highest point within 1 A of
the blob's peak, not the value at the peak. The two maps are on grids of their own (up to ~0.7 A
apart at 2 A) and the anomalous scatterer need not be the difference density's highest point;
1 A, half the shortest metal-ligand bond, keeps a neighbouring modelled S or metal from being read
as the blob's own. NaN (null in _blobs.json) where the merge kept no Bijvoet pairs, instead of 0.
The ION rule reads the same value.
The report's blob list shows it only where the SUMMARY's Anomalous row says "signal present"
(AnomalousRow, factored out of the summary so both read one decision); without the signal it is
noise and is left out. --mode refine has no merge to judge on and leaves it out.
Live: rugnux_live_blob.anomalous_sigma in /rugnux/status (C++ model, python docs, frontend client
and Redoc regenerated); the Rugnux page's Details get a blob table with an "anom σ" column and the
structure viewer's peak label an "anom" reading, both only where anomalous_signal is set.
Tests: FindDensityBlobs_AnomalousPeakNearTheBlob (a scatterer 0.7 A off the difference peak read
at its own height; no map, no reading), Refinement_blob_anomalous_peak (a model missing a zinc and
a water, data with the zinc's f'': zinc blob 32 sigma, water -0.1), report and live checks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Since the decay is fitted about each frame's mean s^2 it fits the full, unattenuated slope and
cross-validates on sweeps where it did not before. It is fitted first and the time x detector
surface after it takes only what the slope left, though that surface follows a damage trend that
is not linear in dose better than the single slope: on a 1440 deg multi-turn sweep the slope took
34 A^2, the surface's held-out gain fell from +0.45 (no decay) to +0.17 and ISa from 20.7 to 17.4,
R_meas 7.7 -> 8.1%. Applying the decay is a decision the surfaces' outcome should inform, so where
it is adopted the surfaces are fitted a second time without it and the chain with the higher merged
CC1/2 per shell (Fisher's z) is kept. On three such sweeps ISa returns to 20.7-21.3 (17.4-18.9),
R_meas to 7.3-7.9% (7.5-8.2%).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The lattice a cell given with -C fixes is put on that cell's axes
(OnGivenAxes): of the relabellings that keep its centring, the one closest
to the given cell, with a monoclinic cell given a- or c-unique matched b-unique
and written back on the given axes. On a metric close to orthorhombic the
two-fold is the typed unique axis, not whichever fits best.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A C-centred monoclinic lattice with beta 1.75 deg off 90 also passes as a
primitive monoclinic one at the character table's tolerance; taken first,
that cell is not the crystal's and the run lost 0.2 A.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The angles of a cell given with -C (no -S) are read literally and name the
crystal system: three 90s orthorhombic (tetragonal or cubic where the
lengths are equal within tolerance), two 90s and a 120 hexagonal (or
rhombohedral in hexagonal axes), two 90s monoclinic, three equal angles
rhombohedral, anything else triclinic (P 1, as before). The system is
fixed like a fixed space group fixes the lattice: indexing searches the
system's own centrings (LatticeSearchForSystem, one IndexingSettings
field), nothing promotes or demotes it (no sub-lattice change, no doubled-
cell re-expression, no triclinic adoption), and only the space group within
it is searched. Runs without -C, and -S runs, are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
ReadRestraintDictionary looked up the block <code> / comp_<code> of a CCD-style entry and used it
without checking it was found; a user file naming its block otherwise crashed. It is now refused
with the file name, like the other unreadable dictionaries.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
- LatticeSearchTest: the metrically monoclinic-C triclinic cell is now a synthetic one
(24.0 7.2 11.4 90 103 90 reduced), chosen for the test only.
- Refinement: the Wilson B and the B ceiling are computed after the "no atoms" early return,
so an empty model no longer computes a Wilson B from zero model intensity.
- ACKNOWLEDGEMENT / CPU_DATA_ANALYSIS_REFINEMENT: the combinatorial-refinement paper
(Structure 21, 1923-1930, 2013) is by Nwachukwu, Southern, Kiefer, Afonine, Adams,
Terwilliger and Nettles; it was credited to "Terwilliger et al.". DOI checked against Crossref.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A cell given with -C and no -S is taken at its word: an angle is 90 or 120
deg only where it was typed so. A cell with no such angle, and not three
equal angles (rhombohedral), names a triclinic lattice, so the run fixes
P 1 and keeps the lattice and setting exactly as -S P1 does. Cells with a
typed 90 or 120 deg, and runs without -C, are unchanged.
docs/RUGNUX_CLI.md: the -C and -S rules (fixed group = lattice and setting
kept; literal -C angles).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
With -S the first pass no longer asks the sub-lattice / twin-composite
hypothesis or the long-axis hypothesis: the lattice, its volume included,
is the one the group's class gave at indexing. With -C as well, a group
whose axes are not all alike is put on the given cell's axes rather than on
the axes the absences prefer, so the written cell is the user's setting
(P2/c forced on a C2/c crystal keeps a and c as given instead of
swapping them). Runs without -S are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
P 1 has no centred setting in the group table, so when the space-group
search finds no symmetry on a lattice indexed centred it won on the
centred cell itself and kept the positions the centring adds as
observations. It now goes through the existing doubled-cell reduction with
the lattice's centring. Only runs without -S that end triclinic on a
centred lattice change; none in the reference battery runs does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
With -S the lattice is put in the Bravais class of the user's group at
indexing (LatticeSearchForGroup) instead of the most symmetric class the
metric supports, and nothing promotes or demotes it afterwards: no
centring the group does not have, no metrically-hexagonal re-expression,
no length-equality demotion pass, no triclinic pseudo-symmetry adoption.
A primitive group on a lattice with no primitive cell of its system (P2 on
a C-centred monoclinic lattice, P3 on a rhombohedral one) is put on the
centred cell of that system taken as primitive, so the positions the
centring extinguishes are predicted and merged as asked. Stills with a
group but no cell follow the same rule. Without -S every path is unchanged.
On a triclinic small-molecule set whose metric is monoclinic C (-C <P1
cell> -S P1) the lattice stays triclinic through indexing, refinement,
integration and merging, P 1 on the given setting, where forcing the
monoclinic-C metric gave an unusable merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
LeaveOutOtherCellForms named the other form only where a population's bounds fell outside the kept
range. Those bounds come from CellPopulations on this function's volumes, the kept range from
GivenCellPopulation on volumes computed its own way, so the valley the two share can land a rounding
inside the kept range. On a two-form sample run with the smaller form's cell, the larger form (3702
frames, 14%, population mean +5% in volume) was left out of the merge as it should be, but reported as
CELL_FORMS= 1 with no other cell. Tell the population by its mean instead. Which frames are merged does
not change; only the report (CELL_FORMS, OTHER_CELL_FORM_*) does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The fulls loop re-chooses the smoothness (lambda) of the per-frame scale curve by
cross-validation every round until its step falls below the acceleration onset. The
held-out error is often bimodal - a curve following a few frames and one straight over
tens of them, minima a few percent apart - and which minimum is lower depends on the
scales the round started from, i.e. on the last choice. On the flip-flopping loops the
choice walked down a decade a round and then jumped back up four or five decades; every
jump moved the scales by rms |dlogG| 0.2-0.8, so the step never reached the onset and the
loop was stopped unsettled (thaumatin 3.8 keV: 1.5e4 <-> 1.5e9 every ~4 rounds).
Now the first choice that turns back the way the previous ones went is kept, and the loop
is one map from there. A loop whose choice only moves one way is unchanged.
CPU battery (--no-gpu --no-refine), 33 open/in-house sets incl. the guard list:
unsettled fulls loops 65 -> 54, final merge unsettled 9 -> 6; thaumatin 3.8 keV ISa
27.8 -> 31.5 (XDS 30.8), 4.6 keV 38.1 -> 38.6, 6jgj 13.5 -> 13.7, all three now settled;
every other set's ISa, R_meas, low-res R_meas and REFMAC R-free unchanged. Private arm (5):
unsettled loops 9 -> 5, both unsettled final merges settle, values within 0.4 ISa.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Where the sweep budget leaves the copies of a residue together only, each position is now a single state
holding every copy's atoms, so a variable of k copies takes a k-th of the states, of the contribution work
and of the card memory. With five copies, 305k reflections and up to 325 positions for a side chain with a
planar end, the states of one copy each took ~4 GB beside the refinement's pool and the refinement failed
on the GPU (out of memory). The energies are the same sums as before. The log says when the copies were
searched together only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The prediction window's three-term width (constant + angular + bandwidth) was
fitted, unweighted in s^2, to its own equal-count shells, crowded at high
resolution. The finest shells decided it and the low-resolution window was an
extrapolation of them. Over 13-2 A the three terms are nearly collinear, so a
tiny input change could flip which one carries the width. Masking two hot
pixels took one run's constant term from 7.4e-5 to 0, so the window at 13 A
fell from 8.0e-5 to 2.7e-5 1/A, below the 4.5e-5 measured there. Its
lowest-shell balance factor then moved 0.97 -> 0.68 and R-free 0.1855 -> 0.192.
The formula is now fitted to the same equal-width-in-d* table the partiality
uses, each shell weighted by 1/s^4 so that every shell counts the same in
proportion. On the logged tables of six still-arm runs (rc184 and rc186
inputs) the fitted curve stays within ~10% of the table at every shell. The
13 A window no longer depends on which terms the fit keeps: 4.5e-5 vs 4.7e-5
on that run, 1.08e-4 on both inputs of another (it was 1.26e-4 vs 1.72e-4).
The terms on their own are still undetermined, which the code and docs now say.
The merge-stage balance also gives a shell it cannot balance its neighbour's
factor instead of leaving it as measured, which had left the outermost shell
at half the balanced shell inside it. The balanced table is then made
non-decreasing again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
`rugnux --mode scale -o combined run1_process.h5 run2_process.h5 ...` reads
every file's reflections and per-image metadata, keeps each frame's own
geometry, numbers the frames on from file to file, and runs the existing
stills merge (MergeStills) on the union - so the cell-form selection, the
relabel onto the run's cell (RelabelOnSublattice, which also takes crystals
of a file indexed on permuted axes) and the data-only indexing-ambiguity
resolution act over all crystals of all files at once. The cell, group,
given cell and rocking curve are the first file's; the ice gate pools all.
Checks (ScaleInputsRefusal): every file stills; each file's cell the
first's lattice in some setting (a change of basis of determinant one),
unless -C/-z gives the cell; one space-group number unless -S. A mix of
stills and rotation files is refused, and so are several rotation files
(pointing to --mode mx, which merges sweeps from their masters); the
multi-sweep machinery is not touched. -s/-e are refused with several
files; --live-checkpoints replays the files one after another.
Per file the log and the report (INPUT_nn= images indexed merged
left-out file) give the counts; MergeOnTheFly::Mask is made public so
"merged" is the merge's own definition.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The reference of a joint multi-sweep merge was the best sweep on its own (resolved ISa, finest d_min,
then ISa). On a set of three non-merohedrally twinned crystals the first look made the reference a
sweep that had integrated the crystal in a wrong C2 setting: it reached 1.65 A and resolved an ISa,
but its CC1/2 was 0.02 and no other sweep correlated with it (CC -0.02..0.04). Every other sweep then
had "no axes" and was processed again given its cell, and the final merge was one sweep at 37.5%
completeness - while two of the others correlated with each other at 0.92 and 0.58.
Every sweep is now put on the axes of every other (ChooseReferenceSetting, all ordered pairs, which
also serves the star relation that follows), and among the sweeps with a resolved ISa those on whose
axes the most others reach MIN_REFERENCE_CC are the candidates; the existing rule (finest d_min
within 1%, then ISa) picks among them. Where every sweep agrees with every other - the passing
multi-sweep sets measured - the reference is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A sweep whose cell is several percent short (header distance kept) can share too few reflections with the
reference to be put on its axes at all, and the distance re-run skipped every such sweep: the cell error
that kept it off the axes was never corrected. Every sweep with a merge whose cell differs from the target's
by more than SHARED_CELL_AGREEMENT is now processed again from the distance that makes the two cells agree,
unless the two volumes are a whole multiple apart (another lattice). For a sweep on no axes there is no
held-out residual to compare; its second run is taken where it has no fewer fulls, and the relation made
again decides its join on its intensities.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The distance re-run loop read each sweep's decision from the relation made before the -C re-runs, so every
sweep those runs rescued ("no merge"/"no axes" on the first look) was skipped and kept its header distance:
on a four-sweep set two rescued sweeps stayed at the header distance with cells ~3.5% short while the other
two had found a distance ~3% longer. The sweeps are now related again (and the target sweep chosen again)
after the -C re-runs, before any cell is compared.
A re-run replaced the first run whenever its held-out residual was not higher, however small the
difference: runs were taken at differences of 0.01-0.03% (inside the bootstrap error), one with 40% fewer
fulls and a cell overshooting the target, one of a second crystal whose cell legitimately differs. A re-run
is now taken only where its held-out residual is lower by more than twice the two residuals' errors combined,
and where it has no fewer fulls. The log gives both errors and both fulls counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Where a crystal is a twin composite, the first pass of a sweep adopts the stronger domain's cell and
leaves the reflections the other domain overlaps out of integration (RotationIndexerResult::
TwinOverlap). A sweep of the same crystal that indexed the domain's cell directly, without the
composite, integrates those reflections with both domains' intensity in them. They are reflections
the reference does not have, so neither the isomorphism ratio nor the held-out residual sees them -
they count as "unique added" - and joining such a sweep put the overlapped class into the merge: on a
twinned small-molecule multi-sweep set weak reflections came out several times too strong and SHELXL
R1 rose from 0.05 to 0.13, whichever sweep was the reference. The sweep's run now reports its
twin_overlap_index to the joint relation, and against a reference that left the overlap out such a
sweep is left out ("twin overlap").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A restraint mmCIF (monomer library, AceDRG, grade, eLBOW: data_comp_<CODE>, data_link_*, data_mod_*)
is read by the caller (ReadRestraintDictionary, no file I/O in the library) into a RestraintDictionary
that RefinementSettings carries; BuildRefinementTopology takes its entries over the embedded library's
of the same name. A PDB CCD entry (bonds without lengths) is restrained to its ideal coordinates'
bonds, angles and sp2 planes (no torsions, no chiral volumes); without ideal coordinates it is refused.
Positional inputs: an mmCIF with _chem_comp_atom/_chem_link/_chem_mod and no atoms or reflections is a
restraint file; a model and reflections with no images and no --mode run --mode refine.
Embedded: RET (LYS-RET), HC4, CRO, GYS, CR2, NRQ (their pept links), BLA, LBV, CYC (CYS-CYC).
CMO stays out (library C-O 1.25 A).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
NAG and NDG join the embedded subset of the CCP4 Monomer Library, and the
pyranose group joins the groups whose links and modifications are kept, which
brings the glycosidic links (ALPHA/BETA 1-2..1-7), the sugar-Asn/Ser/Thr links
and the DEL-O1/DEL-HOn modifications. A lysozyme-chitotriose model (6qnb) went
from 3 NAG held fixed to all restrained, with its two BETA1-4 LINK records
applied.
A sugar bonded to a neighbour without a LINK record is now linked too
(AddSugarLinks: a pyranose atom within 2 A of an atom a library link joins it
to); without it the glycosidic bond was restrained by nothing and repelled as
a contact. The same model with its struct_conn records stripped makes the two
links and refines to the same geometry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The warning also fired when only the P1 merge the space-group search read had an unsettled
scaling loop - 17 of the 26 flagged reports of one battery - although nothing the report
quotes is read off that merge. The search merge's loop is reported in the log; the warning,
the flag and SCALING_ITERATIONS now describe the final merge. ProcessResult loses the
search_scaling_converged member that carried it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Molecular replacement moves every copy of the model (centred, turned,
translated, symmetry image and lattice shift, rigid-body fit) and renames
its chains, but carried the model's TLS metadata as read: origins in the
model's own frame, T/L/S unturned, selections on the old chain names. The
refinement then read U from origins tens of angstroms off, and the model's
own groups lost to isotropic B by a wide margin (R-free 0.55 against 0.36
on a three-chain MR case).
PlaceTlsGroups rebuilds the groups on the placed structure from the
model's: each placed chain is matched to the model chain it is a rigid copy
of (best superposition), and each group's origin is moved and T, L, S are
turned (R X R^T) by the superposition of its atoms; whole where each of
its chains was placed once, one group per copy where a chain was placed
several times. Called on the MR candidates after compaction, on the carried
placement, and on the validated placed model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The population (now step 2a) runs before the solvent build (2b) and the side-chain field (2c), and the
documentation lists them in that order. Two caps keep the stages affordable on large models:
- The side-chain field searches the copies of a residue each alone only where a sweep stays within 4e8
likelihood evaluations (states times reflections, about 3 s on one card); beyond that the copies are
searched together only, which is where a model of many copies spent its time.
- The population has at most as many members as there are trial slots, so it runs in one wave; with fewer
than two slots it is left out.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The reference was the finest per-sweep d_min, ties broken by ISa - but sweeps of one crystal mostly
stop at the same detector edge, with d_min apart in the third decimal by which reflections happen
to be measured there, so the tie-break almost never fired and the reference was picked on noise.
On a twinned small-molecule multi-sweep set this flipped when unrelated upstream changes moved the
long, best sweep's edge d_min by a few thousandths of an A: a short sweep with half its ISa and a
different beta became the reference, was joined, and SHELXL R1 went from 0.050 to 0.135. Within 1%
of the finest d_min the sweeps now count as reaching the same resolution and the ISa decides.
The per-sweep log line now also prints each sweep's d_min and ISa.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A single descent from a placed prediction sees the basin it starts in. After the weight trajectory, twelve members
refine side by side in one wave, each a trial a macro-cycle longer than a step at the trajectory's weights but on a
path of its own: the positions as they are or kicked at random by up to 0.2-0.3 A per coordinate, on the data as
they are or blurred along a schedule (60 then 20 A^2 - the landscape of 4 A, then 2.2 A -, 100/30, 30/10, then the
resolution as it is). The blur is a continuation on the resolution with no engine change: the likelihood's variance
of each reflection raised by exp(B_h s^2/2), which the Rice likelihood reads as Fo and F_model both blurred, with the
X-ray weight balanced again for the gradient left (TrialWeights::blur, kick, seed; RunTrial takes the TrialWeights).
The crossover is made on one 2mFo-DFc map averaged over the members with the free reflections left out
(PopulationMap): each residue is taken from the member whose atoms sit in the most density, the junctions are mended
by a macro-cycle at the ladder's tightest rung (at the state's weight the chimera ends out of the geometry bounds),
and the child is refined by one trial at the state's weights. The child displaces the state by the rule every trial
is held to - within bounds, R-free lower beyond the tie; the members are never chosen among on the free set, whose
best of twelve would be the best of twelve (RefinementSettings::population, 0 for none).
Measured: on 13 predictions placed by MR (af-bench) the child was kept on 4 - R-free 0.011, 0.010, 0.007 and 0.005
lower - and the other 9 come out as before; on 22 refined models (the battery's refine arm) one came out 0.009 lower
and none worse.
On the live cytochrome C set the gain, where the child is kept, comes in the rounds that follow it (0.42 -> 0.387 on
two of four jittered starts, nothing on the other two, and not repeated with other kick seeds): the population alone,
with the rounds off, takes 0.003 off. Six members, or members a step long, did not reach that basin. Blurred rungs
tried as members of the first ladder were never chosen on any of the 13 sets (the blur's own R-free at the rung is
worse; its gain comes later in the path) and are not kept; forced on every rung they took 0.004-0.006 off four sets
and put 0.02 on one. Cost: the members' evaluations are GPU-bound, so the stage costs 1.2 s per refinement pass on a
5000-atom model and 4-5 s on 13k atoms on one card (a quarter of that on four); refinement time on af-bench +15-45%.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A new stage after the weight trajectory and the solvent build (SideChainField.h): every searchable side chain is a
variable whose states are its staggered rotamers (planar ends every 30 degrees), the conformation it has, and
absent (pruned to CB). The energy of a state is the refinement's own target with the rest held: -ln L over the
working reflections at the fit as it stands, plus the repulsion term over the X-ray weight, plus, for Asn and Gln,
the hydrogen bonds the amide can and cannot make (the two orientations fit the density alike). The structure
factors are linear in the atoms, so a state's contribution is computed by direct summation (a dozen atoms over
the group's operators, a CUDA kernel; a CPU evaluator for the other builds and the test) and the target is
re-evaluated at every reflection for every state in one pass, with no map and no gradient. The copies of a
residue the NCS relates are one variable, searched together in each rotamer (the NCS restraints of the step after
pull a copy moved alone back) and each alone. The field is minimised by iterated conditional modes, small side
chains first and arginine last; a move must beat the state as given by the log of the number of alternatives.
One step at the trajectory's weights from the field is judged against the same step from the model as it was and
kept where both its R-free and its -ln L of the free reflections are lower; the field never sees the free set.
On the live cytochrome C placement (three copies, no haem) the field moves 3-12 side chains per run, 15 of 18
of them toward the crystal structure (Asn104, Glu22, Ile82 in all copies; the amide of Asn104 the right way round
only with the hydrogen-bond term), the step from it lower on the free set in every run; the final R-free moves
with the path (+0.001, -0.008, -0.022 on three jittered starts). The field costs ~0.8 s on the GPU plus the two
steps; on a model whose side chains the data hold it moves nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
MakeSideChain, SideChain::Build, the staggered rotamer list and the overlap against the rest of the model move from
ClashRelief.cpp to SideChainRotamers.{h,cpp}, unchanged, so that the side-chain field can turn side chains and
score their contacts the same way. RelieveSideChainClashes is as it was.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Two start-conditioned defects of the first round, found on a flat-B start of the
crystal model and on a placed prediction (pLDDT-derived B of ~6 A^2), both taken from
the B column as given:
- The ceiling of B was 2 x b_max of the model, i.e. 100 A^2 for a flat or pLDDT B
column, and it bit through the TLS tensors (BoundedB scales them down to it) long
before atoms reached it: on the flat start one round gave R-free 0.4040 at ceiling
100, 0.3989 at 140, 0.3870 at 200, 0.3848 at 300, while the refined max B was 123.
The ceiling is now set from at least 2.5 x the data's Wilson B as well (WilsonB,
fitted against the model's own atoms from 4 A), which leaves a deposited model's
ceiling where it was.
- The ADP weight was balanced before the overall B of the scale was moved into the
atoms, which every trial does: the B similarity's gradient goes as 1/B, so a column
at 6 A^2 balanced a weight ~100 x too stiff (2350 vs ~20 once refined). The move is
now made once after the placement, before the weights are balanced.
Together these give in round 1 most of what the second and third rounds were adding:
flat start 0.4040 -> 0.3822 (base with rounds 0.3798 at 3 x the time), placed
prediction 0.4162 -> 0.4038 (0.3982 with rounds), crystal model with its own B
unchanged (0.3790). 13 AlphaFold starts from images with the rounds as they are: six
lower by 0.005-0.021, none higher beyond 0.0025, median -0.002, refinement time
unchanged.
Measured and not adopted: elastic-network and packing-density B starts (within noise
once the ceiling is right), and a coarse-to-fine B solve along chain / network-mode /
residue shifts before the per-atom one (same optimum to 0.0002 at +25 % evaluations).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A predicted model's rotamer, placed by molecular replacement, can run through the main chain of another
copy or of a symmetry mate - as where the copies' side chains meet on a non-crystallographic axis, atoms
within an angstrom of each other. The repulsion only pushes the two atoms apart along the line that joins
them, so such a side chain left the clash in whatever direction that line pointed and settled in a rotamer
the density did not hold, with B factors to match.
Before the topology is built, every side chain with an atom more than 2 A inside its contact distance to
another residue (symmetry mates included; LINK partners, disulfides and metals excepted) is turned about its
chi angles to the staggered rotamer (the terminal planar group every 30 degrees) with the least squared
overlap against the rest of the model; main chain, CB and B factors stay as given, hydrogens are placed
anew by the topology. Later passes let the side chains turned so far choose again in the others' new
places. Logged per residue with chi before and after; REFINE_SIDE_CHAINS_TURNED in the report.
On the live three-copy MR placement of a prediction the three amides and one carboxylate are turned; at
the same ladder rung R-free 0.4444 -> 0.4431 (no NCS) and 0.4402 -> 0.4375 (NCS), final 0.4183 -> 0.4146.
The rotamer the reference crystal structure holds would give 0.4379 / 0.4100: the remaining gap is the
rotamer choice, which only the density can make.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
On a model refined before, R-work falls step after step while R-free does not, and the trajectory ends at its best
anyway; each such step is a wave of trials. A model of 490k atoms spent six of them (240 s of a 760 s refinement) on
a round it then gave back; with the rule it refines to the same R-free in 620 s. On 28 refinements of 22 deposited
models and 6 predictions the rule changed nothing.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The map preparation (refine.maps, ~0.38 s per refinement) was the CPU bulk-
solvent mask of the missing reflections: 0.33 s of it, on the main thread
while the two FFT maps (0.07 s each) and Fcalc ran beside it. Inside the mask,
gemmi's serial shrink (set_margin_around) took 0.135 s, the FFTW transform
0.12 s and gemmi's island removal 0.06 s.
The shrink is now a gather over the w planes (ParallelChunks): every mark gemmi
makes is the same store on a masked point and a marked point still reads as
masked, so each masked point can read from the unchanged mask whether gemmi
would have marked it - the same mask bit for bit. The gridding test now also
runs a 0.5 A grid, where both rings of the shrink are active, against gemmi's
mask (5 groups, 1 and 4 threads). Sub-stage timers (maps.*, missing.*,
fmask.*) stay.
cytochrome C live refine: refine.maps 0.37-0.44 -> 0.28-0.35 s, fmask.put 0.22
-> 0.12 s. R-free identical on the 22-merge guard. The FFT is left serial: a
parallel decomposition of the r2c transform is not bit-identical to FFTW's 3D
plan (4e-10 of the largest coefficient, measured) and the island removal is a
serial flood fill.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
StreamPoller spun on cudaStreamQuery for every wait, which was a quarter of the
CPU samples of a refinement: with a dozen weight-trial slots on one card, a
dozen threads burning a core each for the length of every evaluation. Each
card now counts its engines; an engine alone on its card still polls (the
main engine before and after the trials - an evaluation is a millisecond or a
few, and a sleeping thread's wake-up would sit on the critical path of each),
and where other engines share the card the thread sleeps on a blocking-sync
event instead.
cytochrome C live refine (two refinements per run, 12 slots), interleaved A/B
x3 against the previous commit, R-free identical in every run:
32 threads wall 7.95 -> 8.00 s CPU 38.9 -> 20.8 s
-N 8 wall 7.95 -> 8.11 s CPU 37.4 -> 18.5 s
8 cores wall 8.23 -> 8.36 s CPU 35.9 -> 17.6 s
Every sleeping variant tried (blocking event, 50 us sleep-poll, 100 us spin
then block, event-spin while alone) halved the CPU time and cost the same
0.1-0.15 s of wall, so the simplest is kept. The freed CPU is for the host
the refinement shares with the integration and the other slots.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A model whose B factors arrive far from where they refine - a prediction's pLDDT - balances a weight that is not the
one its B wants; the lower rungs took 0.002-0.010 off R-free on five such starts (molecular-replacement placements
of predictions), nothing off eight deposited models, at 5-30% more refinement time. They run as the balanced weight
did: at the middle X-ray rung first, at the others where refining B comes within two ties of holding it.
Also: the rounds of the weight and displacement-model choice are a setting (RefinementSettings::rounds); every trial
reports the -ln L of its free reflections; and the displacement model named in the result stays the one an earlier
round chose where a later round changes nothing (it was reported as isotropic).
Measured against: deciding every trial on -ln L free instead of R-free (0.003-0.012 better on the prediction starts,
but 0.004-0.02 worse on six of 22 deposited models, so not taken); a hypothesis restarting the isotropic B from flat
(never displaced any); 60 L-BFGS iterations per macro-cycle (another 0.004 on the predictions, +60-100% time, mixed
on the deposited models - not taken).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The ceiling of B was clamp(2 Bmax, 100, 300), and PrepareModel cut every B to 300 before it was set: a structure
refined at 4-5 A (mean B 380, most atoms above 300) was flattened, its as-given R-free 0.05 above the validation's
on the same model (0.359 against 0.306), and the refinement ended worse than the validation (0.339). The 300 bound
stays for every model under it (it sizes the gather); a model above it gets a ceiling of 1.5 times its largest B
(BCeiling), PrepareModel keeps only the floor, and an anisotropic U is scaled to the model's ceiling, not the
constant. The case above: as given 0.325, refined 0.242.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
A deposited P2(1) model whose eight copies are pseudo-symmetric, validated against a merge in P4(1)2(1)2 (the data
cannot tell the two apart): the validation puts the model through a change of basis into its own group and scores
it there, but the refinement was handed the model under the data's group. The operators the model does not have
overlaid its copies four times over, the solvent and the overall B ran to their bounds, and the "refined" R-free was
the null's (0.33 unrefined -> 0.60 refined; p_model.pdb said P 21 1 1, p_refined_model.pdb P 41 21 2).
RefineModel now refines in the placed model's group where that is a proper subgroup of the data's (every rotation
and centring of the model's group among the data's, the data's group the larger): the merge is expanded to the
subgroup - each reflection standing for every mate the data's group made equivalent, at the merged amplitude, with
its free flag, the Bijvoet pair swapped for a Friedel mate - and no twin law is tested, the data carrying the larger
symmetry exactly. RefinementResult::space_group names the group refined in; the run's refined model and
`--mode refine`'s output are written in it. On the case above the refinement now starts where the validation ended
(as given 0.322 against 75965 reflections in P 21 1 1) and ends at 0.320.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Three things wrong with the PDB/mmCIF a refinement wrote:
- A LINK record read with the model was written back whatever the atoms
do in the refined model. The records are never restraints here (a record
the dictionary has no link for is ignored by prepare_topology, and the
bond count with and without them is the same), but the next program
takes each as a bond: phenix.refine made four "custom bonds" of a placed
model's clashes between NCS copies and wrote them as LINKs at 0.7 A, and
rugnux carried them on to its output at 4-5 A. A record whose two atoms
are farther than 3.5 A apart in the model written (longer than any bond,
a metal's included) is now left out, with a log line; hydrogen-bond
records stay. This also drops a metal's outer-sphere records that
AddMetalLinks refused as bonds, which otherwise reach the next program
as bonds.
- A HELIX record's blank length (columns 72-76) was read as 0 by the
vendored gemmi PDB reader and written back as 0, which
phenix.model_vs_data refuses ("Bad helix length"). Blank now stays
"not given" (-1), as Helix::length means it, and is written blank.
- The REMARK 3 TLS block was inserted before CRYST1, i.e. after HELIX and
LINK; it now follows the title section, where the format has it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The positional-input classifier knew a PDB file by its first record, from
a short list (HEADER, TITLE, REMARK, CRYST1, ATOM, ...). A model written
without a title section starts with whatever comes first in the format -
rugnux's own refined model starts with its HELIX or LINK records - and was
taken for an image file, so `rugnux --mode refine <mtz> <refined.pdb>`
ended in "Several inputs are read together in --mode mx only" while the
same file worked under --model. The list is now every record type the
format puts before the coordinates.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The refinement makes its engines by the dozen (one per weight-trial slot, remade
in every round where the B ceiling rises), a few dozen device buffers each, with
cudaMalloc - which synchronises the whole device, stalling the engines already
evaluating. All of them (lane, card, engine, restraint sets, ModelDensityGPU)
now come from the stream-ordered pool CudaDevicePtr defaults to; the free is
safe because every engine waits for its stream before its buffers go, and a
lane and the card go after the last engine on them. ModelDensityGPU takes the
allocator as a parameter so that the rigid body keeps its synchronous
"wait for the work beside" buffers.
cytC live merge, 3 rounds, 12 slots on one card: cudaMalloc/cudaFree 3514 -> 0
(cudaMallocAsync 3610, 39 ms instead of 1.2 s of device-synchronising calls),
cudaMemcpyAsync API time 2.0 -> 1.3 s, graph launches 0.96 -> 0.52 s; wall
8.27/8.28/8.31 -> 7.98/7.94/7.92 s, R-free identical (0.4028). [Refinement],
ModelDensityGPU_BitIdenticalToCPU and RefinementXray_gradient_GPU pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The GPU's density gather reserves its pairs for every atom at the B ceiling at most. Two paths could
leave an atom wider than that: the TLS bodies applied to the atoms bounded the ceiling first and then
raised the smallest eigenvalue to the floor, which could lift the largest above the ceiling again; and
the model's own TLS groups were read (as total B and as residual B) without any bound on the anisotropy.
Either failed the refinement with "model density: gather pairs over the reserve" - seen on a model
brought in by molecular replacement with its TLS groups, and whenever riding hydrogens were off (with
them, the hydrogens' share of the reserve hid the overflow). One helper, BoundedB, now bounds all of
these as the per-atom U already was: floor first, ceiling last.
Battery refine arm identical to rc186; [Refinement] and [refine_only] pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The rounds (fabcb0e4d, a64d3e3c4) ran the whole refinement again from its written result - setup, placement,
ladder, trajectory, final ladder, displacement hypotheses - and a round not kept had cost a full refinement. What a
second round changed, read off the logs and by instrumenting the balanced weights and the TLS trials: the ceiling of
B (twice the model's largest B: 100 on a placed prediction given with B factors of 6 A^2 under an overall B of 45,
which refines at B near 50-70 - the TLS bodies and the U per atom were held at the ceiling, and the TLS hypothesis
lost to a U per atom; 200 from the written model), the B factors' weight (balanced on the B factors as given, 2500
against 23 on the refined model: the B similarity's gradient goes as 1/B), the rung (the middle one with NCS on flat
B factors, the tightest without NCS once the B factors were refined) and the TLS bodies fitted from refined B factors
instead of flat ones. None of this can be decided in the first round - it is the alternation of positions and
displacements - so the rounds stay, reduced to what changes.
A round is now inside RefineOnce, after the displacement model: the ceiling and the balanced weights from the model
as it is (the engines made again where the ceiling rises), the ladder from it with its displacements as they are, the
trajectory, and the hypotheses from there; kept where it lowers R-free beyond the tie and undone otherwise, three
rounds in all. Where the ladder and the trajectory leave the model where it was, the hypotheses would run identical
trials, so the round ends there - an exact rule: a model refined before pays one ladder and one step. No round where
the refinement does not displace the model as given. The first round is the one refinement as before, so a model
the rounds do not improve comes out as it did. RefineRounds and the placement and keep_below settings go.
On the cytochrome C live merge and MR model, R-free 0.4183 (one round) -> 0.4028 (0.3999 with the full rounds,
which also re-placed the chains and re-ran the solvent build each round). Tried and rejected: the trajectory
continued from the chosen displacement model with positions and U together (no gain: the U per atom overfits), B
refined ahead of B held in the ladder's trust (worse), the B weight balanced with the overall B moved in in the first
round (fixes the prediction, halves the weight on models with an overall B of 15 and costs them 0.002), and the
ceiling raised with the overall B at setup (changes the first round: one AF-start run 0.006 worse).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The shape of the first layout - rail, image, model, a right panel, the collection plots along the bottom - with
the right panel's two tall shell plots made small and square, one above the other under the checks. Below the
stage the right panel stands beside the image and the model from 1200 px; narrower than that the shell plots
follow the collection plots as in the layout before. For comparison with it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
Below the wide stage the page now flows for the common case of a browser beside other windows: the
merge's numbers as a row of tiles under the header, the image and the model side by side as large as
the width allows (one above the other under 1000 px), the collection plots wide, then the shell plots
square with the checks beside them (under them where the row is too narrow).
Mol*'s camera reset frames the structure's bounding sphere within the panel's shorter side, which left
a compact molecule small in a wide panel: the model is now fitted by the atoms' extent across and up the
view (fitCamera.ts), on the first load, on "whole model" and whenever the panel changes shape.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The stage's right column goes: the diffraction image and the model (the run's step until there is one)
take the whole middle, side by side, the image square with the row's height. The bottom row holds
four plots - the collection image by image on the left, wide, a point an image; the merge by resolution
on the right, completeness and CC1/2 side by side and square, a point a shell - and the checks sit under
the merge numbers in the rail. For comparison with the layout before it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
The title line is the phase and the dataset: what the run found wrong was there as well as on the side,
and the checks on the side are its place (a failed run's message goes there too). The resolution the
shell plots mark is now the one the report quotes - where the CC1/2 fall-off is fitted - published
with the merge as `resolution_fit`; the written range, a shell beyond it on purpose, is the end of the
axis, so the line sat on the plot's edge and said nothing. The Resolution tile shows the same number,
with the written limit in its tooltip. The progress bars' axis spans the collection and the deadline
after it from the start and keeps that scale (it extends only once the run is past it), instead of
jumping when the last image came and again at the end. The two shell plots name the axis once and
the limit once; the run console drops its "now" caption and shows the current step's notes instead,
which the centre had room for.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
- The rail gets a Unit cell tile as soon as there is a cell: the lattice probe's on the first degrees,
then the lattice the run indexed, then the merge's; a serial run shows the cell it was started with.
- The right-hand plots are two: completeness and CC1/2 by resolution, one under the other, lines without
markers in the colours, grid and font of the data-collection plots, the merge's resolution limit as a
dashed line, the axis labelled at round d values - as many as the panel's width holds, so the labels
no longer run into each other on a narrow window.
- The header names the dataset (the last part of the file prefix), and a run with warnings says what they
are about ("Processing result: warning - Twinning", the conditions of the report's checklist) instead
of "Done · warnings"; on a narrow window that goes on a line of its own, reserved from the start.
- The phase line keeps its height when the Clear button comes, and the button goes to the right: the
bars under it no longer move at the end of the run. Below lg the progress details go under the bars.
- The spots / predictions / indexed pills of the image only show where the images carry them: a serial
run's images after the pre-pass. A rotation run's images are analysed inside rugnux's own passes, so
the image buffer's copies never had spots, predictions or the indexed mark - the pills did nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
A rotation run published its space group, resolution, ISa and shell table only when the whole run was
over, so the live page showed dashes through molecular replacement and refinement although the merge
had long been made. RugnuxObserver gets OnMerge, called on the pass that makes the result once its
merge is made (before the "Analyses" step); the rotation receiver publishes the headline and the shell
table there, and again at the end as before (with the model's R-free by shell, and a model can still
change the hand). The API descriptions say so; the generated clients carry the new words only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
A further round starts from the model the round before refined, so the
rigid-body and per-residue-B placement is skipped there; and a round whose
R-free does not beat the one before is returned before its maps, twin null
and blobs (it is discarded anyway).
The placement skip changes the path of the later rounds, not the rule: on
22 battery merges with their placed models R-free equal to the rounds
without it except lyso_x06da_ref 0.2413 -> 0.2430 and cytc_x06da_1
0.3143 -> 0.3128; from the timing benchmark's AlphaFold models (MR, mx):
cytc_x10sa 0.410 -> 0.399, 9zlo 0.283 -> 0.284, myob_x10sa 0.312 -> 0.312,
9s02/9gjx equal; cytochrome C live merge, MR model 0.405 -> 0.400. Saves
about 1-2 s of the 7-8 s the rounds add on cytochrome C (wall 15.5 -> 13.6 s).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
One refinement chooses its X-ray weight and NCS on the B factors as given
and its displacement model (TLS / U) only after them. Where the
displacements change much - a placed prediction, whose B factors are its
pLDDT - the weights stand on B factors the model no longer has. Running
the refinement again from its own result (as REFMAC users do by hand)
lets the weight ladder and the NCS choice see the refined displacements.
Cytochrome C, three copies placed by MR from the AlphaFold model, 1.73 A,
live merge: round 2 chose an eight times lower X-ray weight without NCS,
R-work 0.367 -> 0.361 at bonds 0.002 -> 0.001 A, R-free 0.418 -> 0.405;
with --refine-solvent 0.427 -> 0.409 -> 0.401. REFMAC5 from the same model
reaches 0.41-0.43. On 22 battery merges with their placed models no
R-free rose; lyso_x06da_ref -0.003, cytc_x10sa -0.005, 9khr -0.003, 6hv2
-0.002, the rest unchanged. End to end from the AlphaFold DB models of
the timing benchmark (MR, rugnux mx): myob_x10sa 0.341 -> 0.312, cytc_x10sa
0.419 -> 0.410, 9zlo 0.289 -> 0.283, 9s02 and 9gjx unchanged. Cost: refinement time x1.3-4 (cytochrome C
6 -> 12 s on the GPU; 9s02 22 -> 30 s), the second round being run to be judged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The anisotropy caution for a run with more than 10 A^2 of relative B said the scale model removes the
average monotone damage and advised --relative-b. The decay is applied only where it cross-validates,
and on the rotation battery it never did (every damaged set skipped it), so the sentence was false
wherever it was printed; --relative-b on such a run (a 360 deg trigonal sweep, +11 A^2) is itself
refused by its cross-validation, and forced it lowers ISa 20.4 -> 18.6 and CC_anom 0.45 -> 0.33.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The broker's live mode (and --fast-live) held r1 at the default 4 px because the starvation
near-tie is re-checked on the whole sweep, which fast_live never does. Nothing else needs the whole
sweep: the spot width is measured on the pre-scan frames and the crowding of the widened background
rings is counted on the pre-pass frames, both available to the live run before its canonical pass. A
near tie now simply stays where the pre-pass decided it, like every other fast-live decision.
Simulated live (--simulate-live 10 --fast-live), before -> after, now equal to the offline run:
360 deg trigonal sweep (r1 4 -> 6): ISa 18.97 -> 20.44, d 1.730 -> 1.717 A, Wilson B 58.3 -> 54.6,
reported relative-B over the sweep 15.6 -> 10.9 A^2
monoclinic sweep (r1 widened): d 1.652 -> 1.556 A, ISa 9.8 -> 11.3, R_meas 17.5 -> 16.8%
two sets the width leaves at r1 4: unchanged
wall time: no increase (30 vs 33 s, 19 vs 24 s, 22 vs 22 s on a shared GPU)
Not a free win on every axis: on the trigonal sweep the r1 4 merge reads better against a model
(R-free 0.3617 vs 0.3628, shell-scaled R 0.318 vs 0.323, Fe anomalous peaks +1.2 sigma each), and a
10-set battery with --integration-radius 4 forced offline splits the same way (R-free better on 5 of
8, anomalous peak height worse on 5 of 7, one set loses its space group). The radius rule itself is
the open question; this commit only makes live and offline give the same answer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The modulation band's sort of every acentric reflection, and the class ladder's two sorts, compared
fields read from whole MergedReflections - a cache miss per comparison on a large merge, serial, and
the largest single cost left in the pre-promotion analysis on a P1 merge of a large cell. The keys
are now copied into arrays first; the comparisons and so the orders are unchanged.
4nwv (I222, offline, --no-refine): pre-promotion analyses wait 1.9-3.4 s -> 0.0-0.1 s, report
identical. [tncs] tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The per-frame scale G already holds each frame's average scale, so a radiation-damage B only tilts a
frame about the frame's (I/sigma)^2-weighted mean s^2. The single-sweep decay and the per-batch
relative B were fitted and applied about s^2 = 0 instead: the fit was attenuated (it had to explain
frame offsets G had already removed) and the applied factor moved every frame's average a second
time, which the even/odd-frame cross-validation then scored as a loss.
Measured: a synthetic single sweep whose B rises 0 -> 10 A^2 now fits 8.95 A^2 (before 2.74), and its
high-resolution fulls come out on one scale from first to last frame (new test). The two-sweep
synthetic with a 3 A^2 step now fits 3.43 (before 0.97) and applies it (GPU hash re-pinned; the CPU
pin is not reproducible on this machine at the base either). On the rotation battery the decay was
never applied: the six damaged in-house sets read held-out gains of -17%, -146%, -8%, -3%, -3% - the
signature of the double-applied average. With the fix they read -1.1, -0.2, -6.2, -3.1 and a negligible
dB; still skipped, and no merged result moves beyond noise (guard: 10-set battery + those 5 sets
against the rc186 baseline, only time differs). On a 360 deg trigonal sweep with relative B +11 A^2
the decay fit goes 2.91 -> 6.45 A^2, held-out gain -3.8% -> +0.4% (still under the 2% gate).
The second-crystal stretch test passed only because the spurious decay (-19.9 A^2 on data with no
B) gave the partials arm a resolved error model; on error-free synthetic data the run takes the
fulls-alone merge, which the test is not about, so the test data get a 3% proportional error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
In a live run the molecular replacement of a prediction starts on the geometry pre-pass's merge
and its placement is carried to the final merge where it holds there. The two passes index the
sweep each on its own, so on a lattice with an indexing ambiguity the final merge can come out in
the other hand, the carried placement then fits at R-free ~0.6 and the whole search is run again
after the space-group step, on the critical path of the tail.
CarryMolecularReplacement now picks the ambiguity operator (identity included) that takes the
searched merge's reflections onto this merge's by the correlation of their intensities - a pass
over the reflections, milliseconds - and turns the copies by it before the rigid body.
Measured on a P3_221 rotation set with three copies and tNCS (--simulate-live 10 --fast-live,
1800 images): the pre-pass placement, turned by -h,-k,l, holds at R-free 0.417 (the re-search
reached 0.413); MR step 0.91-1.40 s -> 0.18-0.26 s. Refined R-free 0.4175 against 0.4255 from the
re-searched placement (same merge, same refinement). New test
MolecularReplacement_CarryAcrossIndexingAmbiguity (fails on the old code at R-free 0.51).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The twin null replaces a law's mates by unrelated reflections of the same
resolution and asks what averaging alone buys. TwinnedModel puts a mate on
the scale k_total of the reflection it is the mate of - right for a twin
law, whose domain reflection is measured at the same spot - but an
unrelated mate lies in another direction, where the anisotropic scale can
differ many times over. On a trigonal crystal with a pseudo-translation,
refined from a model whose atomic U carried the anisotropy opposite to
k_total, a weak reflection's random mate was put on a scale 22 times its
own (amplitude, at 2 A): the null R-free read 0.466 against an untwinned
0.406, and a law at fraction 0.04 was adopted 7.9 standard errors ahead of
it. With each mate on its own scale the null reads 0.400 (z 0.8,
untwinned); the same crystal's experimental model, adopted twinned at
z 3.1 in rc186, reads z 0.6.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK
The pseudo-translation searches (the gate's vector search and its 20 permutation nulls, the
lattice-averaged search with up to 12 rotation images, and the full-resolution ladder) evaluate
cos(2 pi h.u) thousands of times over up to a few hundred thousand reflections. On a rotation
run with a hexagonal lattice and a pseudo-translation, the libm cosine under CosineCorrelation
was ~40% of all CPU samples of the whole run, and the pre-promotion analysis on the P1
all-observation merge sat on the critical path of the space-group step for 4-7 s.
cos(2 pi (h u_x + k u_y + l u_z)) is now the real part of the product of three table entries
exp(2 pi i m u_axis), built once per evaluation (ClassTable); contrast_over shares it.
Measured on a rotation set (1800 images, P3_221, three copies with tNCS), offline and
--simulate-live --fast-live: pre-promotion wait 5.5-5.7 s -> 0.2-0.3 s, final tNCS analysis
1.3-1.9 s -> 0.3-0.4 s. Report, merged MTZ reflections, placed and refined models identical
to the previous binary. [tncs] and twinning tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQsU5a8AJUdJPgWevAK