410 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	c0bcf1a7a9	rugnux report: XDS geometry keywords as one report section, not p_XDS.INP ... The refined geometry is no longer written as a whole p_XDS.INP (and the report's XDS_INPUT= key is gone). Instead the report carries one section, XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated, as verified with XDS IDXREF). No cell or space group: XDS determines those. The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste. The same keywords were logged by LogXDSGeometry at the start of every pass, with the axis in rugnux's sense; that printing is removed, so the report section is the one place they are stated. Docs updated (RUGNUX_REPORT, RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL). Output only; p.mtz md5-identical on 3 battery sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:09:27 +02:00
leonarski_fandClaude Opus 5.5	7ed1544136	rugnux report: BEAM_CENTRE_OFF only beyond three times the tolerance ... At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery sets - most headers are a few pixels off. It now fires beyond three times the tolerance; between one and three times the summary's Beam centre line states the offset and recommends correcting the beamline value, without a warning. The background-only branch (runs that refined no centre) uses the same bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:57:06 +02:00
leonarski_fandClaude Opus 5.5	244f3c0f9d	rugnux: write the refined geometry as <prefix>_XDS.INP ... A rotation run now writes the geometry it integrated at as an XDS input beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the detector axes carrying the tilt, wavelength, the rotation axis the run integrated at (ProcessResult::used_rotation_axis, after any sign rescue), oscillation (with an adopted rotation scale), starting angle/frame, data range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left commented where the group is not in its number's reference setting), FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the master file as an x_??????.h5 template and a LIB= line to point at the reader plugin; a CBF series gets its ???? template and its own frame numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch (2010) Acta Cryst. D66, 125-132. The rotation axis goes out negated: rugnux turns the crystal by -phi about its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged (apart from the job list): 99.8% complete, ISa 23.3, direct beam within 0.15 px and cell within 0.07 A of rugnux's. Output only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	2632b77eb5	rugnux report: count overloads and overlapping spots, warn with advice ... Overloads: the integrated reflections whose signal disk held a saturated pixel are counted on the integration outcome before the merge, over the whole range and over the reflections coarser than 4 A, where they concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution reflections, and recommends attenuation / a shorter exposure. Measured: a strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%. Overlaps: both integration engines count the signal disks a nearer neighbour's prediction takes pixels from (the overlap ownership the default --overlap exclude already builds), and those left under --overlap-minpk of their own - BraggIntegrationCounts::overlapped / overlapped_major (log line "Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning fires from 1% in the second and recommends a longer distance, finer slicing or a smaller beam; on 10 battery sets it fires on the two long-axis ones (8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both; documented in RUGNUX_REPORT.md. Counters only; nothing reads them. p.mtz md5-identical to the rc175 battery on 10 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	f462047038	rugnux report: BEAM_CENTRE_OFF warning when the file's beam centre is wrong ... The pre-scan's beam-centre check and the post-refinement both measure where the direct beam lands, but the report only carried the centre the run used. Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the log's check prints, here on the run's own cell). A shift above the tolerance is a BEAM_CENTRE_OFF warning naming the file's, refined and background centres, the difference in px and mm and the value to correct the beamline to; a run that refined no centre is warned on the background alone where its fit resolves the difference (3 sigma). The summary gains a Beam centre line. The refined centre, not the background, is the trigger wherever one exists: on a battery set where XDS's refined direct beam and rugnux's agree to 1 px, the background fit is 12 px away from both. Report-only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:07:22 +02:00
leonarski_fandClaude Opus 5.5	fcb9c94134	Merge branch 'gpusf' into rc175: model validation on the GPU, in parallel, on every card ... Structure factors, maps and the null on the GPU (with gemmi's mask shrink step), the second validation started on a forecast beside the first, change-of-basis candidates scored at the same time, validation work spread over the visible cards, the merge's GPU engine freed before the validation, a CUDA error failing the validation instead of restarting it on the CPU, and no first-validation maps where the forecast one is kept. Output files md5-identical to the CPU path's reflection files and placed model on 8sa8, 8xtg, 9ea5, 8tyy, 7qis and myob; maps within float tolerance of the CPU path and bit-identical run to run and at -N 8. Validation time 8sa8 24.0 -> 7.4 s, 8xtg 18.4 -> 5.0 s, 9ea5 19.1 -> 3.0 s, 8tyy 47.7 -> 3.0 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 14:39:09 +02:00
leonarski_fandClaude Opus 5.5	b43844ae06	rugnux: the merging engines sit on their card's NUMA node while they drive it ... The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two threads that drive the merging tail were not: the quality guard took its card with set_gpu(1), and the main engine runs on the pipeline's own thread, which goes on to everything else. On a multi-socket server their pageable copies to the card went through a remote socket. RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost, IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask themselves; the new test checks both that and that a pinned caller keeps its own mask. No flag, no environment variable. On one NUMA node (this workstation) only the device setting happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base = rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa, cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs), -N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise, e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:51:38 +02:00
leonarski_fandClaude Opus 5.5	d5fcf2f05b	rugnux: model validation's structure factors and maps on the GPU ... ModelStructureFactorsGPU computes what compute_model_factors() and map_from_coefficients() compute on the CPU - F_calc from the model's density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data() does) and F_mask from the Refmac bulk-solvent mask, both on the reflections prepare_asu_data(d_min) lists, in its order; and a map from ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a device. Made once per cell, group, resolution and model, then evaluated as often as the coordinates change, so refinement or MR can call it in a loop. The device is an explicit parameter; every call leaves the calling thread's current device as it found it. Pieces: - ModelDensityGPU: the rigid body's deterministic brick gather, moved out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 -> 16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1 cell) are no longer refused; existing zones are gridded identically. - One copy of the content is gridded and the symmetry composed in reciprocal space (SymmetryComposition), operators applied on the fly; the mask is ModelMaskGPU (every image of every atom, islands, shrink). - Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the coefficients written are the same), in-place cuFFT c2r, transposed back to XYZ on the device. One map at a time, in the engine's buffers. Decided once, up front, per card, from its TOTAL memory: the engine's bytes (16 N + cuFFT work + reflections, N the larger of the structure- factor and map grids) must be at most half the card - the rigid body's engines take at most a quarter beside it. Otherwise, or where the gather cannot grid the cell, the CPU path runs, logged with needed vs total. Anything to a resolution other than d_min (the null's 3.5 A fits) stays on the CPU, so all replicates and the real model's side of the null are computed the same way. A CUDA failure takes the existing path: the validation restarts on the CPU. Measured, model validation total per run (CPU path -> GPU), 16 GB card: F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations; 14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s; P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s; P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s; small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s. p.mtz md5-identical on all 13 sets. Against the CPU path: FC within 1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within 1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC, FOM, k_sol and anomalous site list identical at the printed precision, except where a rigid-body commit sat on an exact R-free tie (0.2155 -> 0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result is bit-identical run to run and with -N 8 (maps, map MTZ, placed model). Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked at 14.4 GB with what the merge still holds). Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A: same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:38:37 +02:00
leonarski_fandClaude Opus 5.5	6025d51a48	ModelMaskGPU: gemmi's shrink step, so fine rigid-body zones stay on the GPU ... The GPU bulk-solvent mask had no shrink step (SolventMasker::shrink(), set_margin_around() with Refmac's r_shrink = 0.8 A), so every rigid-body zone whose grid has an offset within 0.8 A was sent to the CPU. That is not only fine grids: in an oblique setting (P2_1, beta ~141 deg) the 3.5 A zone's lattice-plane spacing is 0.69 A, and on such a set the real fit and all nine null replicates ran their rigid bodies on the CPU. The shrink is now two passes over the grid: mark the solvent points with a macromolecule point among gemmi's near offsets, then turn into solvent every macromolecule point with such an edge point at any stencil offset (near or far, gemmi's split at the coarsest grid step). That is gemmi's rule in both of its branches, and the stencil is built on the host in double exactly as gemmi builds it. ModelMaskGPU::ShrinkIsNoOp() and RigidBodyGPUEngine::MaskSupports() are gone; the rigid body no longer refuses such zones. Verified (ModelMaskGPU_ShrinkMatchesGemmi): the shrink alone on gemmi's post-island mask is bit-identical, and the whole GPU mask equals gemmi's put_mask_on_grid() bit for bit (0 differing points) on the five test groups at 1.5 A and an oblique P2_1 cell at 3.5 A; repeats are identical. Measured on the oblique-setting set of the open arm (P2_1, 14k atoms, 1.66 A): model validation 17.2 s -> 5.2 s with the GPU structure factors of the next commit in place (the first validation's ten rigid bodies, ~12 s on the CPU, now take well under a second). p.mtz md5-identical; the null moves from +73.1 to +86.1 sigma (the replicates are refined on the GPU, which agrees with the CPU to rounding), verdict unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:38:37 +02:00
leonarski_fandClaude Opus 5.5	f956eb25e8	rugnux: model validation does the same work in less time ... --model validation (battery-only for users) was 18% of the battery's time. Every number it produces is unchanged to the bit (p.mtz, maps, placed model and every model-validation line of the report md5/diff-identical on 11 open sets); only when and where the work runs changes: - The bulk-solvent grid fit (FitModelScale, most of the CPU time) fits each solvent pair on a copy of gemmi::Scaling's target that takes |Fcalc + k_sol exp(-b_sol s^2) Fmask| once per pair instead of at every solver evaluation; same expressions, same types (new test checks a grid point against gemmi's own Scaling fit with ==). - Fcalc density and the solvent mask are made on two threads; the model's structure factors beside the GPU engine reservation. - The indexing probe fits the relabellings concurrently. - The null's replicates run beside the real model's placement (they start from a snapshot of the model as read); one GPU engine per replicate plus one for the real fit instead of a cap of 4 (engines are interchangeable and deterministic). - The 2mFo-DFc, mFo-DFc and anomalous maps are made and written concurrently; the placed model is written beside the reflection files. - A rigid-body zone whose solvent-mask grid needs gemmi's shrink is sent to the CPU when the engines are reserved (ModelMaskGPU::ShrinkIsNoOp), instead of failing on the GPU and validating everything again on the CPU - the same CPU result, without the wasted first attempt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 00:01:14 +02:00
leonarski_fandClaude Opus 5.5	41411b51b8	DectrisSimplonClient: separate connections for commands and status reads ... Commands and configuration writes (initialize, disarm, threshold change) can take minutes. With one shared connection a status poll waited behind them; now commands/config writes go through one kept-alive client and status/config reads through another, each with its own mutex and fixed timeouts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:53:01 +02:00
leonarski_fandClaude Opus 5.5	9807bc40df	Broker: log why a DECTRIS run ends in a detector error, reuse one SIMPLON connection ... "Error in detector operation" was raised whenever the wait-for-idle loop in JFJochServices::Stop() ended in any state other than idle. For DECTRIS that includes a single failed status request (connect/read timeout, non-200), which GetState() silently turned into NOT_CONNECTED, and a cancel whose disarm failed - none of them distinguishable in the log. - DectrisDetectorWrapper::GetState() logs the failure reason before returning NOT_CONNECTED. - Stop() logs the state that ended the loop, whether the stop command failed, and what the detector reports (DetectorWrapper::GetErrorDescription(); for DECTRIS the SIMPLON state and the detector status "error" list). - Cancel() logs why the stop command failed. - A non-200 GET now reports the HTTP status and body (it printed "Success"). - DectrisSimplonClient keeps one keep-alive httplib::Client, serialised by a mutex, instead of opening a new TCP connection per request. - The wait-for-idle loop polls every 100 ms instead of every 5 ms. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 22:45:42 +02:00
leonarski_fandClaude Opus 5.5	a29edb69cf	rugnux: decide whether there is a lattice on distinct reflections, against every wrong angle ... The first pass's last gate - is the lattice this crystal's at all - compared the validation spots on the lattice with the median of five displaced-spindle-angle nulls, against the null's binomial noise floored at one spot. On the two in-house no-crystal controls it is fed a handful of genuine stray reflections from something rotating with the sample (the same detector positions, a pair symmetric about the beam 198 frames apart, in both sweeps, offset by the 15-frame start difference), plus one reflection near the spindle axis that stays in diffraction for ~46 frames. With the decisions taken from the first part of the sweep the 60 validation frames are 3 deg apart instead of 6, so that one reflection is counted on three validation frames, and the gate passed on 4 spots of 151 (null median 0) and 7 of 72, where the whole-sweep flow had seen 3 and 3. Now the evidence unit is the distinct Miller index - a reflection on several validation frames is one piece of evidence - and the test uses every displaced angle's count: with no crystal the recorded angles are one of six exchangeable arrangements, so the gate is the binomial tail of the recorded arrangement's share at p = 1/6, at the same 0.001 significance (SPOT_BUDGET_SIGNIFICANCE_Z). No threshold is added; the median and the one-spot floor go from the gate. The comparisons between lattices (ValidationEvidencePrefers) are unchanged. no-crystal control 1: 4 distinct vs 2 over 5 wrong angles, tail 0.009 -> no lattice (was P 1) no-crystal control 2: 5 distinct vs 2, tail 0.002 -> no lattice (was P 1) p.mtz md5 identical to rc175 7c9a6c7d1 (GPU) on 5nw5, 5reo, 7atg, 8v4o, 9qw8, 9rci, 5ky6, insu_I_x06da_low_isa, myob_x06da_powder_2; cytidine and myob_x06da_split pass. The weakest real lattice in the open/in-house/private battery logs has 117 spots against a null of 0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 21:00:40 +02:00
leonarski_fandClaude Opus 5.5	2981aedd3e	Broker: a collection can be followed from /start, its start message comes in the event stream ... With a DECTRIS detector the broker starts the writers - and so learned the collection's start message - only when the detector's own start message arrived, after /start had returned. A reader started right after /start found no collection (404) and had to wait for, and race with, the detector. - /start registers the collection in LiveCollection (Expect) as it is accepted, so GET /live/events is accepted from then on. The image pusher fills in the start message when it starts the writers (Begin) - at once on the FPGA path, when the detector's start message arrives on the DECTRIS path. - The event stream's first event is now `start`, carrying that start message (CBOR, base64) with run number, file prefix, image count and images per file; it replaces `collection`. Until it is known the stream only keeps alive. file and end follow as before. - However the measurement thread ends, it ends a collection the pusher did not end (EndIfOpen), so a cancel or a failure before the detector streamed reaches the reader as `end` with an error. The wait is event-driven: it ends with the start message or with the measurement. - GET /live/start.cbor is removed: the stream carries the start message, one route less. - rugnux takes the start message from the stream (BrokerFeed::WaitForStart), so it can be started right after /start; files reported before it attached the reader are replayed to it. Tests: LiveCollection_ExpectedAtStart; BrokerFeed_StartFromEventStream (an event stream whose start message comes late, then a file and the end). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 17:03:09 +02:00
leonarski_fandClaude Opus 5.5	21f30549a7	rugnux: follow a collection from the broker while it is written ... rugnux <path>/<prefix>_master.h5 http://<broker>:<port>/ - when the master file is not on disk yet - processes the broker's current collection as it is written: - the start message comes from GET /live/start.cbor; the given path has to end in the collection's master file name, and the data files are read next to it; - JFJochStartMessageReader builds the dataset from the start message (detector, geometry, goniometer, mask) and reads the images straight from the data files; a read of an image whose file the broker has not reported waits for it; - BrokerFeed reads GET /live/events in a thread of its own and hands the reader each closed file. On end, the run finishes with the images the collection delivered (a read past them returns no image). A broken stream, another collection, or a refused follower makes every waiting read throw LiveCollectionLost, which ends the run like a fatal resource error; - the token is taken from JUNGFRAUJOCH_HTTP_TOKEN (the variable the viewer uses) and sent only in the Authorization header; without it, and with no master file on disk, rugnux stops at once; - rotation data and --mode mx only; plain http only (TLS is not linked into rugnux). With the master file on disk, the URL is ignored and the run is offline as before. The streamed run does not read the master file and is not compared with it: its result can differ from an offline run of the same files in the last bits of the metadata. Tests ([Live]): LiveCollection, the server-sent-event parser, the FileWriter callback, the TCP pusher's closed-file reports with two real writers, and JFJochStartMessageReader against the master the writer produced. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:22 +02:00
leonarski_fandClaude Opus 5.5	509a54ef92	Broker: tell a reader which data files of the current collection are closed ... So that rugnux can process a collection while it is written, the broker reports each data file once a writer has closed it and renamed it into place: - Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the images the file holds. TCP protocol version 5. Files closed at END are reported before the END acknowledgement. - LiveCollection (image_pusher): the start message the writer of the master file received, the closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher has no back channel and offers none. - Two routes, protected by the dataset's bearer tokens like the other dataset routes: - GET /live/start.cbor: that start message; - GET /live/events: server-sent events (collection, file, end, superseded) with the history first and a keepalive every 10 s. One follower at a time; a second gets 409. Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged. - The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for the whole collection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:21 +02:00
leonarski_fandClaude Opus 5.5	f970972163	Writer and broker: the master file appears only after every data file it links to ... The final master marks a finished collection for anyone watching the directory (rsync, autoprocessing, a reader following the collection), so it must not appear before a data file it links to. - FileWriter: at END every data file is closed (renamed into place) before the master is finalized and renamed. A data file that fails to close does not cost the master; the failure is reported after it. For NXmxIntegrated, file 0 is the master's own image dataset, so there a failure still stops the master. - With several writers, the master was written by the writer of the first socket as soon as it got END, while the others could still be closing their data files. END now goes to every other writer first: - TCP: the connections are taken in reverse order, and each END waits for its acknowledgement, which a writer sends only after closing its files; - ZeroMQ, with a writer notification socket: END to every other socket, then their notifications (the same ones Finalize collected before, now collected here), then END to the first socket. Without a notification socket there is nothing to wait on and the order is all it changes. Tests: TCPImageCommTest_MasterWriterGetsEndLast (three mock writers, the master's END after the other two acknowledged), StreamWriterTest_ZMQ_MasterAfterAllDataFiles (two real writers, the second started only after the collection was sent; the master's ctime is after every data file's). Both fail without this change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 14:55:56 +02:00
leonarski_f	d9ddc1613d	Merge branch 'pplot-det' into rc175	2026-10-08 13:46:57 +02:00
leonarski_fandClaude Opus 5.5	49abb4a079	XDSPluginTest: expect a signed image's INTx_MAX to pass through as saturated ... Plugin 1.1.0 (2629f3821) passes a saturated pixel of a signed image (INTx_MAX) through with its value so XDS flags it against OVERLOAD; only the error marker (INTx_MIN signed, UINTx_MAX unsigned) becomes -1. The legacy test still expected INT16_MAX -> -1. It now checks the saturated value explicitly and also that module-edge (bit 30) and chip-edge (bit 31) mask pixels read as -2. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 12:36:40 +02:00
leonarski_fandClaude Opus 5.5	f290e40d20	GPU azimuthal profile: sum the corrected values in fixed point ... The fused adaptive spot finder's azimuthal profile (and the standalone AzIntEngineGPU) summed the corrected pixel values - pixel times a float correction - with float atomicAdd, in shared memory and then across blocks. Float addition is not associative, so the per-ring sums depended on the order the threads and blocks happened to arrive in, and on the grid size chosen for the GPU. The profile, and the per-image background estimate read from it, moved in their last bits between two runs of the same command: on a thaumatin JUNGFRAU 4M sweep one bkg value in _plot.txt differed in the 4th digit between runs. The raw ring sums that drive detection were already exact integers; only the reported profile was affected. Each corrected value (and its square) is now quantised to 2^-12 on its own and summed as a signed 64-bit integer, the convention the raw ring sums and the Bragg-integration profile already use. Integer addition is associative, so the sums are the same whatever the order or grid. The repeat test of the fused finder now also requires the profile and its std to repeat bit for bit (it failed on the old kernel at the first repeat); the AzIntEngineGPU test re-runs 20 times. Fused finder benchmark: 0.313 -> 0.324 ms/frame. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 12:33:34 +02:00
leonarski_f	889418e00e	Merge branch 'xds-plugin-license' into rc175	2026-10-08 11:13:48 +02:00
leonarski_fandClaude Opus 5.5	2629f3821c	XDS plugin 1.1.0: mask module/chip edges, pass saturation through; recommend Durin ... Plugin: - Module-edge (bit 30) and chip-edge (bit 31) pixels are marked -2 (untrusted). Chip-edge pixels are split from a larger pixel with integer division and read low at low counts; feeding them to XDS as real pixels was what made the three plugins disagree in CI. - A saturated pixel in a signed image (INTx_MAX) is passed through with its value, so XDS counts it as an overload; only the error marker (INTx_MIN / UINTx_MAX) becomes -1. - Release version 1.1.0 set in CMake (library name and start message). VERSION_* in plugin.h stay 0: XDS refuses a plugin whose info-array major version is not 0. - Built and packaged on Linux only; the untested macOS archive is dropped. - The XDS linking permission is removed from LICENSE; the plugin is GPL-3.0. CI: - XDS.INP beam centre corrected to 1091/1137 (ORGY was 7.7 px off, which put IDXREF on a knife edge where one plugin's run fell into a false minimum). - Each XDS case must reach CORRECT, IDXREF spot SD must be < 2 px, and all file layouts read by the same plugin must give identical SPOT.XDS. Docs: recommend Durin (PSI or Global Phasing build); mask-bit table rewritten from measurement. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 10:56:34 +02:00
leonarski_f	af3ea9ccad	Merge branch 'gpu-tail' into rc175	2026-10-08 10:22:03 +02:00
leonarski_fandClaude Opus 5.5	87f0bdc870	RotationScaleMerge: fit the error model on the device in the GPU build (tier F) ... FitErrorModelGPU puts the samples in the host's rank order with four stable radix passes, so the equal-count bins hold the same samples; computes every normalised deviation with the host's rounding, so every bin median is the host's to the bit; and hands the per-bin terms to the host's own update step, now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing that differs is the order each bin's two sums are added in: a fixed tree on the device, the order nth_element happened to leave the bin in on the host. a and b^2 can therefore differ from the host fit in their last bits - measured on two synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The device fit is deterministic (same pool, same bits). p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the difference does not survive the float rounding of the merged sigmas there. A knife-edge decision elsewhere could still see it, which is why this is kept as its own commit. The host fit took 0.4-0.6 s per fit on the large merges, about half of it the initial equal-count split, with only 16-way parallelism in the iterations. Device peak about 72 bytes per sample (the rank sort; each stage's buffers are freed when it is done); the free memory is checked first and a device without it stops with a message naming the CPU build. Kept last on the branch so that it can be dropped on its own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 08:52:04 +02:00
leonarski_f	ebc008ad94	Merge branch 'pp50-reg' into rc175	2026-10-08 07:59:33 +02:00
leonarski_fandClaude Opus 5.5	8ed898d5a1	RotationScaleMerge: sort the outlier band's log-spread pairs on the device ... The multiplicative spread of the outlier band is the weighted median of (|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of that step. SortPairsGPU sorts them on the GPU (by the second key, then stably by the first) and leaves the sequence std::sort leaves: two pairs that compare equal are the same two numbers. The values themselves (a logarithm) and the running sum stay on the host. p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case checks the device order against std::sort. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:27:25 +02:00
leonarski_fandClaude Opus 5.5	b6f2096015	RotationScaleMerge: the Wilson outlier test on the device in the GPU build ... WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the device and returns the same result to the bit: the two orders the host sorts into are total orders, so a stable radix sort of the indices reaches the same sequence; every sum is taken by one thread over its shell or reflection in the host's order; each product the host rounds on its own (-ffp-contract=off) is rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z, u, the tail scale) are taken on the host from downloaded counts and medians. MergeAndStats calls it where the merge itself is on the device. 8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the host one on a population that reaches every branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:07:32 +02:00
leonarski_fandClaude Opus 5.5	697ffc9a18	ResolutionCutoff: a bin's CC1/2 is the median over groups of its reflections ... The automatic cut fitted its logistic to one Pearson CC1/2 per bin, which has no bound on what one pair can do to it. In a weak shell a single reflection with a huge intensity in both halves - an artefact the halves reproduce, or one observation with a sigma fifty times the shell's - carries nearly all of the covariance. Each bin's reflections are now dealt into up to seven groups by their indices (groups of at least ten reflections), and the bin reads the median of the groups' CC1/2. On a clean shell every group estimates the same CC1/2, so the median is the Pearson value; an outlier takes one group with it and the median does not follow. Only the cut decision changes; the reported shell table is unchanged. Found on a weak split myoglobin crystal: one reflection carried 50-99 % of the covariance in most fit bins past 2.6 A (e.g. 43647 in a bin with median I 1.3), reading them at CC1/2 0.93-0.98 where the median of groups reads 0.2. The cut swung between 1.21 and 2.03 A with --prepass-fraction; it is now 1.85 A at 0.5 and 1.89 A at 1 (XDS 1.51 A, rc.174 1.96 A). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 06:53:40 +02:00
leonarski_fandClaude Opus 5.5	1bb6d59bf7	HotPixelFinder: mask dead pixels, and run on CCD sweeps too ... A pixel that reads at most half of its own sector's level on average, by more than the bound a hot pixel's excess has to clear, is now masked as dead - the mirror of the hot rule. Nothing but a defect keeps a pixel dark frame after frame, and every reflection whose disk covers it is integrated short. The finder now also runs on CCD sweeps (no sensor depth): a dead CCD pixel reads its offset, far below any ring with background, and the hot rule's ten-fold ratio does not depend on Poisson counts either. Found on 5jk4 (ADSC Q4, 2304x2304): the last row and the first two columns read the offset (~20 ADU against ~60-100 around them) and were unmasked. Reflections touching them merged to large negative intensities, and one observation at y=2298 with sigma 10334 swamped the outer shell's half-set CC1/2 (3.9 %) and the overall CC1/2 (0.786; 0.998 at rc.174). With the 6630 dead pixels masked the fall-off is clean: CC1/2 0.998, cut 0.96 A (fit 1.01 A), at --prepass-fraction 0.5 and 1 alike. Counting-detector sets with low background are untouched (myoglobin X06DA: 0 dead). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 06:53:40 +02:00
leonarski_fandClaude Opus 5.5	35f42e7334	BraggPredictionRot: walk only the l of each column that can pass the phi test ... Rotation prediction enumerated the whole (2h+1)(2k+1)(2l+1) box every frame and kept the ~1% whose rotation solution lands near the frame. A solution within phi_limit of the frame needs |f| = ||p0|^2 + 2 S0.p0| <= 2 |S0_perp| |p0| phi_limit, and along an (h, k) column f is a quadratic in l, so the l that can pass are at most two intervals. Those are walked - widened by 1% in f and by a whole index at each end - in the same order and with the same arithmetic; the rest the phi test rejected anyway. Without a phi limit (min_zeta 0) the whole column is walked as before. The GPU kernel is unchanged. A new test predicts 1000 random lattices, geometries, axes and rocking widths both ways and requires the same reflections in the same order, every value bit for bit (30000 cases passed once locally). CPU build, two interleaved warm runs: 8a1a output loop 241.5 -> 216.3 s, pre-pass loop 90.7 -> 73.8 s, whole run 427.0 -> 384.2 s; 8qaw (with the gzip change before this) loops 116.6 / 251.3 -> 94.4 / 206.8 s, user CPU 14700 -> 12718 s. GPU build within noise. p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 06:46:28 +02:00
leonarski_fandClaude Opus 5.5	0c1c963379	Move SpotWidth to bragg_integration/, SpindleCuspLoss and LeftoverLattices to indexing/ ... A pure move of three method files out of rugnux/: SpotWidth (spot width and bandwidth, which set the integration radius) joins JFJochBraggIntegration, which now links JFJochImagePreprocessing for the ImagePreprocessorBuffer it reads; SpindleCuspLoss and LeftoverLattices join JFJochIndexing. Only include paths and CMake lists change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:05:37 +02:00
leonarski_fandClaude Opus 5.5	9ad92b6bfe	Move the atomic-model code to image_analysis/structure_refinement/ and WriteModel to writer/ ... A pure move. ModelValidation, RigidBodyRefine, RigidBodyGPU, ModelFFT, ModelGrid, ModelScaling, ModelMaskGPU, ModelScaleGPU and SigmaA - everything that works on an atomic model - become the JFJochStructureRefinement library, linked by JFJochImageAnalysis. WriteModel (the placed-model mmCIF/PDB writer) goes to writer/ as its own small JFJochModelWriter target, so JFJochWriter, which a writer-only build compiles, does not gain a gemmi dependency. Only include paths and CMake lists change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:05:37 +02:00
leonarski_fandClaude Opus 5.5	08ca1d5764	Move the hot-pixel finder from rugnux/ to image_analysis/hot_pixels/ ... A pure move: HotPixels.{h,cpp} and HotPixelsGPU.{h,cu} are compiled into JFJochImageAnalysis (like beam_stop/) instead of JFJochRugnux. Only include paths and the CMake source lists change; the objects are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:05:37 +02:00
leonarski_fandClaude Opus 5.5	e6bbfea4c4	Indexing: a coplanar cell leaves the frame unindexed instead of throwing ... The online broker cancelled a collection with "Crystal lattice is coplanar and has no reciprocal cell". The candidate filter in PostIndexingRefinement (used by FFT, FFTW and ffbidx) tests lengths, angles and spot count only; three rows 120 deg apart in one plane pass all of them, and spots along the plane normal index as (0,0,0). Such a candidate - from ffbidx, or one the least-squares refinement walked flat - went on to LatticeSearch and XtalOptimizer, which refuses it and leaves it in place, and then reached Astar() in AnalyzeIndexing/prediction, which throws; the receiver turns that into a cancelled acquisition. - Refine() rejects a candidate below MIN_BASIS_VOLUME_FRACTION, like any other bad candidate. - XtalOptimizerInternal builds the refined lattice first and counts a coplanar result as a failed refinement, writing nothing back (the residual's volume clamp lets a solve end there and report success). Test: PostIndexingRefinement_CoplanarCandidateIsRejected threw the exact message before the fix. rugnux p.mtz unchanged on the three in-house reference sets at --prepass-fraction 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 13:55:44 +02:00
leonarski_fandClaude Opus 5.5	280ff0a4a0	rugnux: remove --estimate-beam-center and the spot-symmetry beam-centre estimator ... --estimate-beam-center was opt-in, off by default and used by no battery run. Removed with it: the half-turn spot-symmetry estimator (image_analysis/geom_refinement/BeamCenterFromSpots and its tests, used nowhere else), its pre-scan branch and paired-frame sample, the sparse-sweep extra reads, the BEAM_CENTER_* constants used only there, beam_center_placed_, the spindle fit that only ran inside it (--fit-spindle / --no-fit-spindle, ProcessConfig::fit_spindle; its estimate was only logged), the viewer's "Measure beam centre before indexing" job option and the RugnuxCommandLine emission. The background centre (FindBeamCenter on the pre-scan projection), the asynchronous beam-centre capture, the beam-centre check that indexes at both centres and the measured centre's bound on the post-refinement are unchanged, and so is the pre-scan sample for the shadow, the defective pixels and the spot width. Warnings that pointed at --estimate-beam-center now point at --beam-x/--beam-y. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:35:40 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00