v1.0.0-rc.182 #93

Merged

leonarski_f merged 47 commits from rc182 into main 2026-11-04 11:13:34 +01:00

Conversation 0 Commits 47 Files Changed 288

+6160 -1319

leonarski_f commented 2026-11-04 11:13:24 +01:00

Owner

Copy link

Copy source

• Rugnux: Fixed indexing of stills of a centred lattice with a known cell.
• jfjoch_broker (experimental): Live Rugnux processing of serial collections.
• Rugnux: Stills detector modules are placed from the data and written to <prefix>_module_placement.json; --panel-refine replaces --refine-modules.
• Rugnux: More reliable rotation processing - indexing, monoclinic setting, space-group choice and refinement of multi-domain twins.

* Rugnux: Fixed indexing of stills of a centred lattice with a known cell. * jfjoch_broker (experimental): Live Rugnux processing of serial collections. * Rugnux: Stills detector modules are placed from the data and written to `<prefix>_module_placement.json`; `--panel-refine` replaces `--refine-modules`. * Rugnux: More reliable rotation processing - indexing, monoclinic setting, space-group choice and refinement of multi-domain twins.

leonarski_f added 47 commits 2026-11-04 11:13:24 +01:00

GPU analysis of converted JUNGFRAU frames with a per-run pre-pass ... 585cc777d5

With rugnux enabled, a serial (non-rotation) collection in the FPGA workflow
is analysed as the DECTRIS frames are: the frame transformation threads
compress each converted frame and hand it to StillsGPUAnalysis, whose threads
run MXAnalysisWithoutFPGA (GPU spot finding, indexing, integration) and pass
the result back to the receiver to be written.

Before any frame is analysed, the first 100 are run through rugnux as
`rugnux --no-merge` on them (RunStillsPrePass, in memory): its beam centre
(the direct beam, from the scattered background and, where frames index, the
stills geometry trial or the bundle refinement with a given cell) and its mask
(plus the beam-stop shadow and the defective pixels) are what every frame of
the collection is then analysed with. Frames arriving meanwhile are held. The
start message waits for the pre-pass, so it states that centre and mask; the
END message carries the centre as refined_beam_center_x/y.

The start message dataset now carries the detector modules, unit cell and
space group, which the pre-pass needs.

tools/jfjoch_stills_gpu_replay replays a stored serial collection through the
same analysis (pre-pass, indexed fraction, sustained rate).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

jfjoch_stills_gpu_replay: -c indexes de novo (Auto indexer) ... 10c5e0d1e2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Stills pre-pass: apply the refined detector distance, report it in END ... 6195e71210

The GPU analysis of converted frames now takes the pre-pass's detector
distance (rugnux refines it with the geometry bundle where a cell is given)
along with the beam centre. END carries it as refined_detector_distance_mm;
that field and refined_beam_center_x/y are now in the CBOR END message, so
they reach a writer over the wire, not only in process.

The receiver's plots are set up again on the pre-pass geometry: a moved
distance changes the azimuthal binning, and combining profiles of the two
binnings threw and cancelled the collection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

GPU analysis of converted frames: rugnux's stills spot finder ... 72eccd6b71

The live analysis indexed 56.3% of the frames of a serial lysozyme run where
offline rugnux indexes 59.3% of the same frames. The whole gap is the spot
finder: the broker's default fixed threshold with two-pixel minimum spots
against rugnux's adaptive threshold that keeps single-pixel spots. With
those two settings (on top of the broker's others) the live analysis
indexes 59.3% of all 24960 frames, as offline; the sustained rate is
unchanged (~1080 Hz on the workstation). The pre-pass decided the centre and
the mask with the same spot finder.

jfjoch_stills_gpu_replay analyses as the receiver does; -F keeps the
broker's fixed-threshold finder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Stills pre-pass: no whole-range distance search ... 4c38fe6718

The broker's distance is the beamline's, refined by the geometry bundle; the
30-3000 mm search (for files whose distance can be off by a factor) took
3.4 of the 9.8 s pre-pass on 100 JUNGFRAU 8M frames. Without it 6.0 s, the
same beam centre (within 0.01 px from the file's centre and from 8-19 px off),
the same distance and a bit-identical mask. Offline rugnux keeps the search
(ProcessConfig::stills_distance_search, default on).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

/rugnux/status: the serial pre-pass and GPU analysis ... 4a54e0c6f5

A serial collection analysed on the GPU after the pre-pass now reports in
the rugnux live status: the run as steps (Pre-pass, GPU analysis) with
images received and progress, as rotation live does, and a `stills` block -
the pre-pass state (holding N of 100, running, done, failed) and duration,
the stated and decided beam centre and distance, the beam-stop shadow and
defective pixels it masked, and the images analysed and indexed and the
analysis rate since. Schema rugnux_live_stills; C++ model, frontend client
and Redoc regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Rugnux live page: serial pre-pass and GPU analysis ... c78e6b99be

Where the run is a serial one analysed on the GPU, the page shows its
pre-pass (state, images held, duration), the beam centre and distance
stated and decided with the shift, the masked shadow and bad pixels, and
the indexed fraction and analysis rate, in place of the rotation headline
and result. Rotation runs render as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: panel refinement opt-in (--panel-refine), module placement JSON (--panels, <prefix>_panels.json) ... fd8704c00c

Module placement in the stills geometry refinement now runs only with --panel-refine (replacing
--refine-modules), for every multi-module input - a JUNGFRAU NXmx master, SACLA/CXI panels and a
CrystFEL .geom alike - and, when it runs, the placement is used from there on. Without the flag the
modules stay where they were given; the frame sample of the geometry bundle keeps the size it had when
the modules were always measured, so a JUNGFRAU run without the flag is unchanged.

A run that places the modules also writes <prefix>_panels.json: one entry per module, its box in the
image (x0, y0, width, height) and where that box lies on the detector plane (ox, oy, fx, fy, sx, sy).
Modules are named by their box, not by their order, which differs between broker configs, written
masters, the gap-mask reader and CrystFEL/SACLA panel lists; --panels <file> reads it back and refuses
a file whose boxes do not match the image's modules one to one. ModulesPlaced does the matching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

broker: /start takes a module placement (dataset_settings.module_geometry) ... 3b60dace7b

The module_geometry array of rugnux's <prefix>_panels.json can be given with /start. Each entry names
its module by its box in the assembled image; the list has to name exactly the detector's modules
(any order), else /start is refused. The image is still assembled at whole pixels as custom_geometry
lays it out; the placement becomes the experiment's modules for the dataset, so it reaches the start
message (detector_modules), the NXmx module_NN groups and the broker's own analysis. A /start without
it goes back to the modules as assembled. Generated server model and frontend client regenerated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: <prefix>_panels.json keeps its keys in the documented order ... c706dfbac1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Test: NXmx module chain of turned, staggered, mirrored JUNGFRAU modules matches DiffractionGeometry ... a82037a41b

Every corner pixel of every module, placed in the image the way the receiver
assembles it (modules turned a quarter beside unturned ones, staggered, Y
mirrored, tilted detector, with and without an image orientation), lies in the
file's module of the same index, and following the master's own depends_on
chain takes it to DiffractionGeometry::LabCoord of that image pixel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: panel study log of held-out residual per panel and shell (RUGNUX_PANEL_STUDY) ... ffc990a330

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: stills module placement on by default, decided per module on held-out spots ... 0976d5836c

The placement was opt-in; without it a multi-panel MPCCD loses most of its
merge quality (R_split about 3x), and an all-or-none rule (refine only when
every panel is populated) gives that up wherever a few outer panels are sparse.

Now, by default:
- a module is placeable from 100 spots on its frames' lattices (was 500). The
  500 gate held outer panels that split-half fits place consistently (0.3-1.3 px
  agreement on 4-5 px shifts with 120-200 spots per half) and whose held-out
  spots then fit much better; held, they were the ones the others' move hurt;
- the placement is used if it predicts the held-out spots better as a whole
  (as before), and then each moved module keeps its move only if the held-out
  spots on it are predicted better; the others go back to where they were given
  and the placement is refitted with them held. On a 32-module detector with
  weak, low-resolution data most moved modules predicted their own held-out
  spots worse while the global residual still improved.

--panel-refine keeps every move without the tests; --panel-refine=off keeps the
modules as given; =diagnose is the default plus the placement diagnostics.
GeometryRefinerSettings::modules_held holds chosen modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: drop the panel study log (RUGNUX_PANEL_STUDY); the held-out tally keeps only the per-module sums the placement rule uses ... 26d751f89c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Regenerate the Python client and Redoc for dataset_settings.module_geometry (update_version.sh) ... df14ce9c9f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

broker: dataset_settings.module_placement with box_*/plane_*/step keys; one conversion in JFJochAPIConvert ... 96633cea6b

The unreleased dataset_settings.module_geometry becomes module_placement, so one name no longer means
both rugnux's --module-geometry file (.h5/.geom) and the placement JSON. Its entries name a module by its
box in the stored image (box_x, box_y, box_width, box_height) and give where the box lies on the plane
(plane_x, plane_y, x_step_x/y, y_step_x/y): no key is called x0/y0 any more, which in custom_geometry
means the readout corner before mirror_y. custom_geometry's x0/y0 now say so in the yaml and the broker
docs; their meaning is unchanged.

The conversion between the generated model and DetectorModule, and the parse/write of the
{"module_placement": [...]} dataset_settings fragment, live in a small library, JFJochAPIConvert
(broker/ModulePlacementConvert.*), which depends only on JFJochAPI (the nlohmann-only generated models,
already built in every configuration, the portable ones included) and JFJochCommon - no pistache, no
broker stack. OpenAPIConvert uses it; so can rugnux. Generated model, Python client docs, frontend client
and Redoc regenerated with update_version.sh (VERSION unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

broker: settings that parse but fail validation answer 400, not 500 ... 7a9c73999e

handleOperationException mapped every JFJochException but WrongDAQState to 500, so a /start whose
module placement does not name exactly the detector's modules (one left out, named twice, or not the
detector's) came back as an internal error although the API documents 400 for invalid input. The input
categories (InputParameterInvalid/AboveMax/BelowMin, WrongNumber, WrongUnit) now answer 400 with the
message as text, as a parsing failure does; everything else stays 500. Test:
JFJochBrokerHttp_StartModulePlacementMismatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: module placement file through the OpenAPI model (--module-placement, <prefix>_module_placement.json) ... bcee3fb5b7

rugnux's own JSON parser and writer for the placement file are gone: the file is the dataset_settings
fragment {"module_placement": [...]} and is read and written through the generated model and the
broker's conversion (JFJochAPIConvert), so /start and rugnux accept exactly the same thing. The unreleased
--panels becomes --module-placement and <prefix>_panels.json becomes <prefix>_module_placement.json; the
released --module-geometry (.h5/.geom) keeps its meaning. Keys are now written in the model's
(alphabetical) order.

The file is written for every stills run on a detector of several modules unless --panel-refine=off,
also where every module was held: the placement as given is a placement too. <prefix>_modules.h5 is still
written only where modules moved. The log names a module by its box corner, as the file does, not by its
centre. Results unchanged (a stills set on an 8-module detector gives the identical report).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

module_placement in custom_geometry's words: x0/y0/fast_axis/slow_axis after mirror_y ... 532f795348

dataset_settings.module_placement (detector_module_placement) now names a module as custom_geometry's
detector_module does - x0, y0 (image position of the readout pixel 0) and fast_axis, slow_axis
(detector_module_direction) - but in the stored image, after mirror_y; x0_plane, y0_plane and
fast_step, slow_step give where pixel 0 lies on the plane and the plane step per readout pixel. Both
schema descriptions state the frame and the mirror (y0 = H - 1 - y0, Yp <-> Yn).

DetectorModule carries the readout axes in the image (pixel 0 follows from the box): from the
detector geometry after mirror_y (ModulesAsImaged), from how the XFEL reader turns a panel, and
Xp/Yp for a panel read along its own array. They travel in the CBOR detector_modules
(fast_axis/slow_axis) and in the NXmx module_NN groups (readout_fast_axis/readout_slow_axis, beside
the NXmx fields), so a stored file names its modules as the broker does. ModulesPlaced matches the
four keys exactly; a module not of the detector, placed twice or left out is an error (400 at /start).
The rugnux log names modules the same way.

New test ModulePlacement_TurnedMirroredModules (Yp/Xn module, mirror_y on) and PanelLayout_Placement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Merge stills-live-simple: live stills pre-pass (beam centre, distance, masks) and GPU analysis in the broker f6234cd3a3

Regenerate the Python client docs, frontend client and Redoc after the stills-live merge (update_version.sh) ... d2faa88781

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

rugnux: decide a cell against its integer multiple on integrated cosets ... 7907d2980e

A cell and its axis multiple were decided by spot counts in several places,
which cannot work: the larger lattice contains the smaller, so it fits every
spot at least as well. The decision now sits in one place, on the integrated
intensity of the reflections the larger cell adds.

- First pass: a pair of scheme answers whose primitive volumes are 2-4x apart
  and that tie on validation frames keeps the SMALLER cell. The spot-count
  arbiter (larger vs smaller and vs own sub-lattice, with its late re-check)
  is gone; the index-2 larger cell is recorded as an offered hypothesis.
- First-pass ladder: the spot-count harmonic refusal of a winning rung is gone.
- Pass-1 supercell guard and the pass-2 supercell-collapse guard lose their
  "the arbiter settled the larger cell" exemption (larger_settled,
  supercell_settled_), which no longer exists.
- Supercell probe: every half-integer class is read against its own sibling
  null, all classes that stand clear are listed, and group closure of the two
  strongest (their sum must stand clear too) is reported.
- Doubled-cell hypothesis: adopts the doubled cell also where the class is part
  of the lattice (a quarter of the lattice's intensity or more in every
  measured shell, coarsest included), and, for a doubled cell an indexing
  scheme offered, where the class grows to a quarter of the lattice's
  intensity, without the one-pseudo-translation test. A superstructure only the
  probe has seen is decided exactly as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

RESULT-CHANGING: FFT indexing finds rows shorter than its floor; the short-axis pass is retired ... 9b4acd6ab4

The per-direction peak search starts at the 10 A floor, so a lattice row shorter than that was
reported at a multiple of itself past it and the cell built from such rows was an exact integer
supercell. A second first-pass hypothesis at a 5 A floor (the short-axis pass) won the true cell
back on frame counts, and could not reach an axis below 5 A at all.

On rotation data (a scanning axis) every shortlisted row reported within three times the floor is
now asked, by a direct sum over the accumulated spots, at each whole fraction of its length down
to 3 A; the shortest fraction that keeps half the row's amplitude replaces it. The cells of those
rows are reduced (widened: a small cell's in-plane combinations fill the standard window) and
refined beside the standard cells, and ranked with them, so where a cell and its supercell index
alike the smaller one is taken. Protein shortlists have no rows that short, so nothing changes for
them; stills and grid scans keep the search they had. The short-axis pass, its speculative pool and
its live warm-up are removed; the refined-length bound of the cell fit goes from 5 to 2.5 A.

The 40 % overlap test of the candidate refinement no longer deletes a cell for its overlap with an
integer multiple or sub-cell of it - one lattice contains the other, so that overlap is total by
construction. Such candidates are kept after every cell the test accepts, which keep their order,
so the first lattice and the extra crystals are what they were. The rotation indexer carries the
refined integer supercells of the cell it chooses, and the twin-composite test asks them as well as
the standing cell: once the search offers a twin domain's own cell, the coincidence lattice it used
to stand on is one of those.

Measured (battery subsets vs rc181-b2): a small-molecule crystal with an axis under 5 A, indexed as an
index-5 supercell now gives its true cell and group; the 13 sets the short-axis pass rescued keep
their verdicts; the sparse reticular twin keeps its domain and overlap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Scaling surfaces: pool resolution shells for the cross-validation gate on small cells ... 3453d0975d

The half-set gate of the detector/absorption surfaces reads a correlation only in shells of at
least 50 reflection groups. With 20 shells, a cell of a few hundred unique reflections has no
such shell, the gain came out exactly 0.0000 and every surface was refused without being
evaluated (a 254-unique cubic garnet, a 189-unique KDP set); a set of ~1100 unique lost its
last shell. Adjacent shells are now pooled until a bin holds 50 groups, and a short remainder
joins the bin before it. Where every shell already holds 50 the gate is unchanged.

Measured (rc181 vs this, SHELXL recipe of tools/battery/shelx_check.py, R1 / wR2):
  garnet 254 unique: surfaces now evaluated and refused (-0.03..-0.59 on z): unchanged 0.1014 / 0.239
  KDP 189 unique: 3 surfaces adopted: 0.0475 / 0.121 -> 0.0479 / 0.122 (neutral)
  L-cystine 1104 unique: surfaces now refused: 0.1237 / 0.335 -> 0.1186 / 0.323
  aspirin, citric acid, HEPES, lysozyme and thaumatin reference sets: identical results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Battery: in-house L-ascorbic acid row (orthorhombic P 21 21 21 small molecule, 0.56 A) ... 87912e1ca8

A short-axis small-molecule case: its shortest axis is below the 10 A FFT floor and a second axis
is not an integer multiple of it, which de-novo indexing used to resolve to a wrong cell. The
reference is our own XDS processing of the same sweep; no COD entry is linked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Battery: per-set rugnux options (args), for files that misdescribe their experiment ... 6c5fbc8dae

A row may carry "args": [...] and an "args_why". The options go into that set's command
ahead of any the run passes to every set, and show in the row's cmd. They are meant for files from
a retired acquisition system that misdescribe the experiment in a way rugnux is not expected to
detect, e.g. a placeholder detector distance, or a run padded with frames recorded after the
shutter closed. They should stay limited to what the reference itself was given (its frame range,
its refined distance), so the comparison remains like for like.

Also names L-ascorbic acid in the Arms table and in doc_check's specimen map; the row was added
without them and doc_check failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Live stills: a failed START or analysis thread no longer hangs the broker ... 41696976f4

With rugnux on, a serial FPGA collection sends its start message only after the pre-pass. If that send
failed (no writer listening), or the azimuthal mapping or IndexAndRefine could not be built, the latch the
analysis threads wait on was never released: no END, and the next /start hung in the receiver's
destructor.

Now the setup after the pre-pass runs in a try. On failure the latch is still released, the analysis
threads drop the held frames (nothing can be written without START), the receiver is cancelled at once
(as on an error in a frame thread), and StillsGPUAnalysis::Finish rethrows the error. FinalizeMeasurement
then skips END, as the old path never had one either, still stops and joins the acquisition devices, and
the error becomes the collection's result: /wait_till_done and /status report it, the state is Idle and
the next /start runs. The non-stills path reports the same error from /start itself, as the start message
is sent in the receiver constructor there; the stills path can only report it once the pre-pass is done.

An analysis thread no longer dies on a non-JFJoch exception: a thread whose MXAnalysisWithoutFPGA cannot
be built passes its frames on unanalysed, a frame whose analysis throws std::exception is written
unanalysed, and a failing sink does not stop the thread, so the queue is always drained and END goes out.
Such errors finish the live status as Failed with the first message. GPUSink also catches std::exception,
and ~StillsGPUAnalysis cannot throw.

Tests: StillsGPUAnalysis_StartFailureReleasesThreads, StillsGPUAnalysis_FailingFrameDoesNotStopAnalysis,
JFJochIntegrationTest_ZMQ_GPUStills_writer_down (HLS-simulated JF4M, no writer, then a writer).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Live stills: held frames wait in the image buffer, bounded to half of it ... 10f83da9e8

The FPGA receiver held every frame for the GPU analysis in a heap vector sized to a full buffer slot and
then shrunk, which keeps the capacity: ~25 MB per JUNGFRAU 8M frame instead of its compressed size, in a
queue with no bound. During the pre-pass (~1000 frames at 100 Hz), or whenever the analysis runs slower
than the frames arrive, that grew the broker's heap without limit on top of the configured image buffer.

A held frame is now compressed straight into a slot of the receiver's ImageBuffer, the configured buffer
the frames are written from. Two buckets per frame: the slot the compressed image waits in (in
preparation, so never previewed or sent), and the slot GPUSink writes the CBOR message, analysis and
image to, which is sent as before. The waiting slot goes back to the buffer, unsent, when the last copy
of the HeldImage is gone. Nothing is copied to the heap and no frame memory is allocated per frame.

The two buckets are also two pools: at most half the buffer's slots wait for analysis (the hold queue is
bounded to total_slots / 2), so the analysed frames always find slots to be sent from. A frame that finds
no free slot, or the hold queue full, is dropped and writer_queue_full is set, as the receiver does for
any frame that finds the image buffer full; the frame thread never blocks. A full hold queue also starts
the pre-pass on what it has. Analysis slower than the arrival rate therefore costs frames, not memory.

jfjoch_stills_gpu_replay holds its frames in an ImageBuffer too (-b MiB, default 8192), and waits for a
slot rather than drop a frame.

Measured on the HLS-simulated CrMX JUNGFRAU 8M broker (40 frames, image buffer 8 GiB): RSS grew by
~26 MB per held frame before, linearly until the pre-pass; now it is flat once the receiver threads have
warmed up (+1.05 GB less at 40 frames, ~25 GB less for a 1000-frame pre-pass). The stills replay of a stored
serial collection gives the same beam centre, distance, mask and indexed count as before the change.

Test: StillsGPUAnalysis_HoldIsBounded; the other StillsGPUAnalysis tests check every slot is returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Output setting: test beta >= 90 on the unique angle only, not all three ... 27fb60f514

With no reference, ChooseOutputSetting writes a monoclinic group with the
unique-axis angle >= 90 deg, as ITA does. It applied that test to the minimum
of alpha, beta and gamma with a 1e-3 deg tolerance. The cell it is handed is
refined without the monoclinic constraint, so the two angles the symmetry holds
at 90 come out a few thousandths of a degree either side of it (on laspartic
alpha 89.9988, gamma 90.0070). No change of axis signs then puts both at or
above 90 while keeping beta obtuse, so every near-standard description was
refused, and the least oblique survivor was a non-reduced one: the reduced
P 1 21 1 cell 5.13 6.92 7.59, beta 99.73 was written as 7.59 6.92 8.41,
beta 143.05 (reindexed -l,k,h+l) in the .mtz/.cif/.hkl/.ins.

The test now reads only the angle the group's unique axis leaves free (the
axis its two-fold keeps, in the landed setting). The same defect wrote 9b22
(beta 123.39 instead of 91.79) and 9ea5 (141.10 instead of 108.73) in a
non-reduced setting without a model; the open battery's --model step undid it
there, which is why only laspartic showed it after rc182 perturbed its cell.

The new [setting] test uses a synthetic P 1 21 1 cell with alpha and gamma a
hair off 90: it must be kept as is, and the same lattice on a non-reduced
basis must come back to it. It fails before the change and passes after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Tests: link JFJochAPIConvert into jfjoch_portable_test ... 022d3af51a

JFJochReaderTest.cpp, part of the portable test binary, uses ParseModulePlacement/ModulePlacementJson,
so the portable (viewer-/rugnux-only) build failed to link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

FFBIDX indexes a centred lattice in its reduced primitive cell ... 42719adbf1

Known-cell (FFBIDX) indexing was given the conventional cell of a centred
lattice. Its own candidate scoring and the indexed-spot count of Refine() then
worked on the primitive lattice of that cell, which is 2x (A, B, C, I), 3x (R)
or 4x (F) denser than the crystal's, so a wrong orientation that fits only
those extra nodes indexed as many spots as the true one:

- R in the hexagonal setting: the obverse orientation and the reverse one (60
  deg about c) share every node of the primitive hexagonal lattice. About half
  of the frames came back reverse; only the l = 3n layers agreed between them,
  and the merge showed a false translational pseudo-symmetry.
- C/I/F: false orientations that break the centring were accepted.

The choice is made inside the indexer's top-candidate scoring, so a centring
condition on the count in Refine() alone does not always settle it: on
synthetic R3 stills the obverse orientation was sometimes not among the
candidates at all.

FFBIDX (GPU and CPU) is now given the Niggli-reduced primitive cell of the
given lattice (Indexer::IndexingCell, with the centring of the given space
group), and every lattice it finds is put back into the given conventional
cell by the integer matrix between the two (Indexer::ToConventional), so
everything downstream sees the conventional basis as before. Every node of
the primitive cell is a possible reflection, so no centring count is needed in
Refine(). The reduction matters: the plain primitive cell of an elongated C
cell has an angle far outside the window Refine() accepts, and such a set then
indexed nothing. The seed-escalation fraction in IndexAndRefine counts only
spots the centring allows on the FFBIDX path. A primitive space group, a cell
without a space group, and the de-novo (FFT) indexers are unchanged.

All known-cell stills flows reach this through IndexAndRefine and the indexer
thread pool: the broker's FPGA and SIMPLON receivers, live rugnux stills and
offline rugnux.

Rotation sweeps of R3/R32 crystals processed as stills with the known cell:
reverse frames 47-50% before, 0% after; indexed spots obeying -h+k+l = 3n
0.67-0.68 before, >= 0.997 after; merged <I>(l=3n)/<I>(l!=3n) 1.2-11.9
before, 0.95-1.01 after. Still battery: primitive control unchanged; on an
I222 set fewer frames index (67% -> 58%) because the extra ones were on a
wrong lattice (indexed spots obeying the centring 0.54 -> 0.99), with R_split
0.088 -> 0.078, CC1/2 0.962 -> 0.981, R-free 0.162 -> 0.156; I23 indexes 3%
more frames; C2221 unchanged.

Tests: synthetic R3 obverse vs reverse and C-centred false orientation in
Refine() on primitive candidates; FFBIDX CPU end-to-end on R, C (including an
elongated one), I and F cells; IndexAndRefine on predicted R3 stills with
FFBIDX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Merge branch 'ffbidx-centring' into rc182 ... cfed2b2dd1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

RESULT-CHANGING: Twinning: index-2 twin subgroups looked up at every origin shift GEMMI can express ... 8492052c87

IndexTwoTwinSubgroups found a subgroup in GEMMI's table only after shifting its origin on a 1/12
grid. The orthorhombic subgroup of P4_12_12 / P4_32_12 keeps 2_1 axes at z = 1/8 of GEMMI's
P2_12_12_1 origin (I4_122 over I2_12_12_1 likewise), so P2_12_12_1 was never offered as a twin
subgroup there and a pseudo-tetragonal P2_12_12_1 twin merged in P4_x2_12 got no _twin_ merge.

The lookup now takes the table's groups with the subgroup's rotations and centring and tries every
origin shift in GEMMI's own translation unit (1/Op::DEN = 1/24, which holds 1/8 and 1/12 alike).
Across the whole table this adds P2_12_12_1 under P4_x2_12, I2_12_12_1 under I4_122, and subgroups
of a few non-Sohncke groups (Fdd2, Fddd, I4_1/amd, I4_1/acd, Fd-3m, Fd-3c); no subgroup listed
before is lost (under Fd-3c:1, Fd-3 is now named in its origin choice 2). Faster than before where nothing matches: the shifts are only tried against
candidates with the right rotations.

Test: "Index-2 subgroups with a two-fold twin law" (P 41 21 2, P 43 21 2, I 41 2 2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

RESULT-CHANGING: Refinement: twin refinement with every domain of the lattice as a second rung; free set closed under the twin domains ... 48d895d715

From the twin-multi prototype (af742b889, d197507c7), rebased onto twin-gate rule B and cleaned up.

The twin arm takes a list of laws: RefinementReflections::mates per law and mates_from (the converse,
which the GPU gathers the gradient by, as a three-fold is not an involution), one fraction per law,
the twinned amplitude k sqrt(a0 |U(h)|^2 + sum_d a_d |U(T_d h)|^2) on the CPU and the GPU, and the
fraction fit as the same intercept regression with one column per law (a law whose coefficient comes
out negative taken out). RefinementTwinDomains: every coset of find_twin_laws, of any order.

DecideTwin keeps the single-law rung as it was (null at 3 SE, or 2 with the L-test). Where that law
is taken and the lattice has more domains, all of them are refined at once and taken where the added
domains beat their own null (their mates unrelated, the adopted law's kept) by 3 SE, summed per twin
group, each at a fraction of 0.05 or more (domains under it dropped and the rest refined again).
TwinNull generalised from pairs to twin groups: kept adjacent, partners (Laws+1)(k+1) apart; for one
law it is the null it was.

CloseFreeSet, in RefineModel before both arms: every orbit under the domain operators in one set.
An unsplit orbit stays; a split one goes free with the probability of its own free share (a salted
hash of the orbit), so the free fraction is kept - the prototype's majority took a C-centred
pseudo-hexagonal merge's free set from 4020 to 891, an unsalted hash inflated free sets by 60%.

Open arm, against twin-grid on 23 twin/pseudo-symmetric sets (runs twinfix-grid2 / twinfix-multi /
twinfix-multi2): the second rung ran on 9 sets and was taken on one, 9hnc (P2 on a tetragonal metric,
deposited four domains at 0.06 each): domains 0.058/0.058/0.055, z 13.1, R-free 0.2174 -> 0.2063. On
the other 8 the added domains refine to 0.000-0.016 and the single law stands. The closure moves
reflections on 5ebi and the two C222 pseudo-hexagonal sets 6fvz/6fwc (their law, FREE_SET_SPLIT
before, is now tested and refused); 5ebi's R-free 0.347 -> 0.377 on the closed free set. Single-law
sets move by up to 0.003 (the regression and GPU gather round differently), all others <= 0.0002.
Wall time +10-50% on the sets that reach the second rung. 5ebi deposited data (--mode refine): the
closure moves 1 reflection and its strongest law is tested: R-free 0.390 -> 0.364.

Report: REFINE_TWIN_MULTI*, REFINE_TWIN_FREE_CLOSED. Tests: RefinementTwin_domains_gradient,
RefinementTwin_free_set_closed. Docs: §15.12, RUGNUX_REPORT.md, ACKNOWLEDGEMENT.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

RESULT-CHANGING: the centred-lattice test asks only about a class the lattice metric hosts ... 486c0b06d1

The centred-lattice test (run after a triclinic indexing whose search arms
could test no operator) takes its candidate from the Niggli character table,
which names a class on 3 deg relative tolerances. Le Page, which measures each
two-fold's obliquity on the lattice itself at the 1 deg bound every other
metric question of the run uses, can find fewer rotations than that class
carries. The test then adopted a group the lattice does not host: on a
triclinic crystal whose C-monoclinic two-fold is 1.7 deg oblique (XDS refines
the same angle), C2 was adopted at operator CC 0.46 and written on a cell the
group cannot describe, and the Le Page re-ask that follows cannot demote.

The test now runs only where Le Page carries the candidate's holohedry;
otherwise it logs why and leaves the question to the Le Page re-ask. Over the
rc181 open, in-house and private battery rows this was the one final point
group with more rotations than the metric hosts; every other centred-test
adoption is hosted and unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Beam stop: a background summed below zero is no deficit of photons ... 8d7cf18277

On frames taken with the shutter closed the pixels scatter about zero, and
poisson_deficit_sigma read a pooled sum below zero as a deficit that grows with
how far below it is. The scattered dark pixels then passed the 6-sigma test,
were bridged into one piece and filled: on rounded dark noise of 0.35 photon
the live stills pre-pass masked 70 % of a JUNGFRAU 8M as beam-stop shadow, and
the real frames that followed the closed shutter were analysed under that mask
(6 % indexed instead of 69 %). The observed count is now taken as at least zero,
on the host and on the device alike; a shadowed pixel still reaches
sqrt(2 * expected), so a real stop on a measurable background is found as before.

Replay of a serial run: mask, beam centre, distance and indexed images are
unchanged on the beam-on frames; dark frames with the detector's own negative
tail mirrored find no shadow (was 0.6 % of the detector), 0.35-photon dark
noise 1.8 % (was 70 %), and the frames after 200 such dark frames index as
after 200 empty ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Stills GPU replay: report the per-image analysis time and spot count ... 9e6eb605b3

The rate after the pre-pass is the rate the frames are read at whenever the
input is slower than the analysis; the mean processing_time_s of the frames
(one thread) and their mean spot count say what the analysis itself costs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

Merge branch 'live-empty' into rc182 ... 544565e8c4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJKPUFLoeAchjzv3x4

docs: rc182 - live serial collections in the broker, FFBIDX centred lattices, smaller fixes ... a4f3003329

- JFJOCH_BROKER.md: Rugnux live mode covers serial FPGA collections; new "Serial collections
  (experimental)" section (pre-pass on the first 100 frames, GPU analysis, held frames bounded
  to half the image buffer, failure handling, /rugnux/status stills block, replay tool).
- CPU_DATA_ANALYSIS_INDEXING.md §4.3: FFBIDX is given the reduced primitive cell of a centred
  lattice and its result put back into the conventional cell.
- CPU_DATA_ANALYSIS_IMAGE.md: beam-stop deficit takes the pooled count as at least zero.
- CPU_DATA_ANALYSIS_INTEGRATION.md: the surface cross-validation pools shells to 50 groups.
- RUGNUX.md: a stills run writes <prefix>_module_placement.json (and _modules.h5 only where
  modules moved).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

CHANGELOG: 1.0.0-rc.182 entry ... e67ad671c5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

docs: the centred-lattice test runs only where Le Page hosts the candidate's holohedry ... b272223822

Describes 486c0b06d in the space-group decisions section, beside the Le Page re-ask it defers to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

CHANGELOG: rc182 centred-lattice stills entry worded as a fix ... 0208f6347b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

1.0.0-rc.182: VERSION and the version strings update_version.sh writes ... 5c25f13b6e

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Tests: the FFBIDX stills test builds only with CUDA, as its helpers do ... 2668550eb3

IndexAndRefine_StillRhombohedralFFBIDX uses MakeRotation/RotateLattice, which are defined inside the
file's JFJOCH_USE_CUDA block, and FFBIDX itself exists only with CUDA, so a non-CUDA build failed to
compile jfjoch_test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Indexing: the supercell step-down enumerates an eighth of the triples and tests the volume as one dot product ... c631204be9

StepDownToReference (rc181, -C supercell step-down) also runs on every rotation re-index with an
internal given cell - the long-axis hypothesis and the floor-lowered re-index - and there it was the
largest single CPU cost of the first pass: 39% of all cycles on a no-crystal control, where the
first pass then took ~4x what it took at rc176. Nearly all of it was the triple loop over the
candidate basis vectors, almost every triple rejected by the volume test.

Two exact cuts:
- of v and -v only the one enumerated first is kept. Both span the same lattice, every test after
  it reads them alike, and the first triple with the most spots in the full enumeration is made
  of first-met vectors, so it is the one returned either way - from an eighth of the triples.
- the volume is taken as |(v0 x v1) . v2| with the cross product per pair, asked first against a
  bound wider than the determinant test, which then runs as before on what survives.

Measured on a loaded box, rc182 vs this: no-crystal control first passes 79 + 122 s -> 27 + 34 s
(wall 214 -> 70 s); the merged MTZ, HKL and the report are byte-identical on four sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

tNCS: the modulation search on the data runs beside its nulls; no zero step in the search ... af1b67398d

The lattice-averaged tNCS test (rc181) runs on the P1 merge before the space-group decision
wherever the Patterson peak is significant, and on large cells that merge has millions of
reflections and 8-12 lattice rotations: the search on the data took as long on one thread as
its 20 nulls on all of them, and the space-group step waited for it (33 s on 4nwv,
16 s on 4mtk). The search on the data now goes into the same parallel pass as the nulls -
none reads another's answer - for the vector test and the averaged one alike.

The greedy search also evaluated a step of 0 on every axis, which is u itself and can never pass
the improvement test; it is dropped. Both changes leave every number as it was.

Measured on a loaded box: pre-promotion wait 46 -> 18 s and 20 -> 10 s on the two large-cell sets;
reports, MTZ and HKL byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

Integration: the profile-vs-box guard keeps a strong reflection the profile reads above the box sum ... 12538dcdb9

The guard replaces the profile fit with the box sum when the two differ by more than 10 box sigmas.
It exists for weak reflections where the fit runs away. On a strong spot the Gaussian profile reads a
few per cent above the box (the box misses the tails), and at I/sigma above ~100 that is more than 10
sigmas, so most of the strongest reflections of bright data were being replaced with the box sum.
A profile intensity above a box sum of more than 20 sigmas is now kept; the low side and weak
reflections are guarded as before. On data where the guard never fired nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdL4qBGYRiSXSGXfCLu

leonarski_f merged commit 3f8c72cd1e into main 2026-11-04 11:13:34 +01:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k) wang_m (wang_m)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#93