408 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	04655bd8bf	MolecularReplacement: the rotation function on the Apple GPU (Metal), equal to rounding ... On macOS, where a Metal device may be used (not under --no-gpu), each merge's rotation stage - the sphere and the peaks' refinement - runs on the GPU: the same trilinear interpolation of the same transform in float, one threadgroup per orientation, every bin's sums of |F|^2, |F|^4 and |F|^2 (E^2 - 1) in a fixed order, and the correlation made of them in double on the host. The GPU reads its own copy of the transform in 2 x 2 x 2 bricks (1.3x faster than the grid's rows, as the eight corners then lie on fewer cache lines), made for the rotation stage and given back before the translation function; the seeds' orientations later on stay on the CPU. Where the GPU fails, the stage goes on on the CPU with a warning. The kernel is MolecularReplacementMetal.metal, carried in the binary as the Bragg integration's are. Elsewhere nothing changes. Measured on an Apple M6 (16 GB), alternating with rc176 + the CPU prefetch, two runs each: - 4nwv (the timing benchmark's moved model): rotation 78.9 -> 30.7 s, wall 344.5 -> 298.8 s; peak footprint 16.1-16.8 -> 18.5 GB. - 9s02: rotation 2.99 -> 0.28 s, wall 79.2 -> 76.4 s. p.hkl, the placed model and the maps are identical to the CPU's on both, and so is every line of the molecular-replacement log (RF Z, TF Z, R values). The RF itself agrees with the CPU's to 1e-4, with the same best orientation (MolecularReplacementMetal_MatchesCPU, [portable]). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018My7tqt8a4LyKKjP3FhScT	2026-10-18 12:51:49 +02:00
leonarski_fandClaude Opus 5.5	78c2a69267	Merge broker-rugnux-live into rc176: rugnux in jfjoch_broker as a live thread during a DECTRIS rotation collection (off by default), the rugnux library without file I/O (results in ProcessResult::outputs, the CLI writes them through JFJochRugnuxOutput), END carrying the merged table, map coefficients and placed model ... Branch reviewed at e16db07ff (review: owner's decisions accepted - _maps.mtz instead of .ccp4 in the viewer, -o "" computes everything incl. MR/validation, Ctrl+C/error paths may leave fewer files, the writer links gemmi/zlib) and merged at 55184eb09 with its review fixes (END size guard, frame drop as in Lite, outputs released before re-runs, pool priority, docs, frontend gating). Conflicts: rugnux/Rugnux.h (kept rc176's host_memory_budget_bytes, output_prefix removed as on the branch), rugnux/RugnuxPipeline.h (MtzTable future + rc176's GPUMerge pointer), MolecularReplacement.h/.cpp (rc176's GPU cost model, packed transform and split functions with the branch's RugnuxModel signature; search_model with rc176's other_budgets, result.model_name). tests/RugnuxLargeTest.cpp: rc176's host-budget test writes its files through WriteRugnuxOutputs now that the library writes none. Generated code (broker/gen, frontend client, Python client docs, Redoc) regenerated from the merged jfjoch_api.yaml with update_version.sh (version unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 21:01:38 +02:00
leonarski_fandClaude Opus 5.5	55184eb096	tests: the parallel unmerged MTZ table is compared bitwise again ... UnmergedMtzTable's columns hold NaN for missing values, and vector<float> == compares NaN unequal to itself, so the serial-vs-parallel check could fail spuriously. Each column is compared with memcmp, as the test did before the MtzTable change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 20:51:08 +02:00
leonarski_fandClaude Opus 5.5	e16db07ffd	broker rugnux live: no files, END built from the result in memory, and review fixes ... No files. The receiver no longer makes a /tmp scratch directory or reads rugnux's files back: AddToEndMessage moves the merged MtzTable and the map coefficients out of ProcessResult::outputs and converts the placed model with PlacedModelFromStructure (now given the data cell, as WritePlacedModel is). FillRugnuxResultBlocks and MtzTableFromFile are gone; their test now builds the blocks from the same in-memory outputs and still checks the writer reproduces rugnux's own files byte for byte. The live status report is built from the result with the writer's file prefix as its label. AddToEndMessage never throws: a failure is logged and the END goes without the blocks. Review fixes: - MakeRugnuxLiveSetup sets nthreads_auto, as a bare `rugnux` does, so the per-image loop keeps to GPUWorkersPerCard per card instead of starting one GPU worker per CPU thread; the 60-degree probe gets a quarter of the threads (at least 2). - A failure setting rugnux up (start message, reader, thread start) marks the live status Failed and nothing else; the collection goes on as in Lite. MeasurementThread and ForwardThread also catch std::exception, so no exception skips the latch or the reader's Ended(). - The ParallelFor pool is lowered to nice 10 only where it can be raised back (CAP_SYS_NICE, or RLIMIT_NICE reaching 0); otherwise it is left alone and that is logged once. - The preview and metadata sockets get the END without the result blocks. - Receiver plots and scan result are fed only from the canonical pass (RugnuxObserver::OnImagePass): the pre-pass reports [0, E) again at the header geometry and the plots accumulate, so it would count those images twice. The canonical pass reads every image once. - A collection the image buffer cannot hold is still not analysed, and now says so in the broker status (Warning) as well as the rugnux status. With rugnux live off nothing changes: the extra END resets are of empty blocks and analysis_warning is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 18:13:23 +02:00
leonarski_fandClaude Opus 5.5	d356cc5da5	ImageBuffer reader: one waiting thread at a time looks at the buffer ... Every rugnux worker waiting for an image called ImageBuffer::GetImage every millisecond, and GetStatus every 100 polls. Each call takes the buffer's mutex and scans every slot, so sixteen waiting workers contended with the forward threads for the lock they put the images in under (GetImageSlot, ReadyToSend). Now one waiting thread at a time asks the buffer, at most every 2 ms, with a single GetStatus, and notes the images it holds in a per-reader bitmap; the other waiters sleep on a condition variable and are woken when it is refreshed or the collection ends. GetImage is called once, for an image known to be there (or given up on). MAX_IMAGE_LAG and Ended() mean what they did. ImageBuffer itself is unchanged. The test gains eight readers waiting at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 18:13:03 +02:00
leonarski_fandClaude Opus 5.5	9c9b987928	Merge host-mem-2 into rc176: a host-memory budget from the machine's total RAM decides what runs side by side, the CPU ingest reads the reflections in place (24 B per partial), merge scratch is really freed, and CPU MR/validation workers are bounded by their bytes; output unchanged ... Peak live heap, CPU path -N10, before -> after: 4nwv scaling 14.8 -> 11.9 GiB, analyses 13.5 -> 7.3; 8qaw scaling 34.0 -> 27.1; 9v16 17.2 -> 13.5; 9zlo/9gjx/9s02 7.7/7.4/8.9 -> 6.3/6.3/7.5. With the budget of a 16 GB machine, 4nwv validation 18.4 -> 10.8-12.3 and MR ~17 -> 12.5. GPU 4nwv 16.2 -> 14.8. Byte-identical to its base on GPU and --no-gpu (timing standard + inhouse, 4nwv/8qaw/9v16/ lyso_x06da_ref/cytc_x10sa), at the machine's budget and with a 16 GB one. Conflict in RugnuxScaleMerge.cpp: perf-8gb-cpu's card-byte declaration of the merge and this branch's host-budget decisions on the post-refinement and the all-observation arm, kept both, in that order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 17:59:30 +02:00
leonarski_fandClaude Opus 5.5	5f731db81d	rugnux: the library returns its outputs in memory; the callers write the files ... The rugnux library wrote its results itself: _process.h5 through a FileWriter, the merged/twin/_mr reflection files, the unmerged MTZs, the maps and their MTZ, the placed models, _detector.jpg and _plot.txt, and on a doubled cell it renamed files on disk. It now writes none of them. - MtzTable (common/MtzTable.h): an MTZ file's header, columns and batch headers with no file behind it. It replaces MergedReflectionsMessage and MapCoefficientsMessage in the END message, with one CBOR encoding. WriteMtz (writer/WriteRugnuxFiles.h) is the one serialiser, used by rugnux and jfjoch_writer alike; it refuses a table whose columns differ in length, name a missing dataset, carry a malformed batch header or name a space group gemmi does not know. MergedMtzTable and UnmergedMtzTable build the tables the old writers built. - ProcessResult::outputs (RugnuxOutputs) holds, for the output pass only: the answer's merge (MTZ table plus what the .cif/.hkl are written from), the twin hypothesis' and the model's choice's merges, the unmerged and unmerged-partials tables, the maps (coefficients always; the grids only with ProcessConfig::keep_map_grids), the placed models, the detector JPEG and the per-image plot table, and the start message of the no-lattice _process.h5. Map RMS is taken from the grid in memory, as the CCP4 header stated it, whether or not the map is kept. - The per-image analysis goes to a caller-supplied RugnuxImageSink; rugnux and the viewer give a FileWriter-based ProcessFileSink. The geometry pre-pass no longer makes a _01_process.h5. - The doubled cell keeps both output sets: the answer's in outputs, the other's in DoubledCell::other_outputs with its suffix; the report names files by suffix after the caller's prefix. - The output prefix, write_process_h5 and the observation dump path are gone from ProcessConfig; --dump-observations is a callback. - What used to be gated on having files to write (model validation and MR, the twin-hypothesis merges, the finalist ledger, the scaled fulls, the unmerged tables) is gated on the output pass instead, so a caller with no prefix gets the same results. - RugnuxOutputFiles (new library JFJochRugnuxOutput, linked by rugnux, the viewer and the tests) writes every file from the result with the names and formats rugnux has always used. The viewer writes its job's files the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 16:51:20 +02:00
leonarski_fandClaude Opus 5.5	4bf41983ed	rugnux: the model is parsed by the caller and handed in as RugnuxModel ... The analysis read the --model file itself, four to five times per run (the model kind, every validation, molecular replacement, the stills reference intensities) and located the PAE sidecar from the file name. The library is to do no file I/O, so the model now arrives in memory: - RugnuxModel (image_analysis/structure_refinement/RugnuxModel.h): the gemmi structure as read, whether it is ModelCIF, the PAE matrix where a prediction came with one, and the name the report gives it. - ParseRugnuxModel / ReadRugnuxModelFile (writer/RugnuxModelFile.h, in JFJochRugnuxFiles): bytes in, gzip by magic, PDB/mmCIF by content, _ma_qa_metric detection as before; the file reader takes the PAE JSON from the same place PaePathFor named. - ProcessConfig::model replaces model_path; PrepareSearchModel, RunMolecularReplacement, ValidateAgainstModel and ModelReferenceIntensities take the model; result fields name it by model_name. - rugnux reads the model before the run starts and stops with an error where the file holds no model (it used to run and report the model as not validated). The viewer reads it when the job is set up. - The broker parses an uploaded model at once and answers 400 where it is not one; it keeps the parsed model in memory, with no temporary directory or file. It takes no PAE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 16:11:58 +02:00
leonarski_fandClaude Opus 5.5	b6f3e3b928	Merge sf-ceil into rc176: the GPU spot finder's local-box test, count threshold and sigma-clip bounds are the CPU's to the bit ... RESULT-CHANGING in principle, unchanged in practice: diagnostic builds counted 0 disagreements of the SNR and count tests over ~1.9e10 first-pass and 2.9e7 candidate tests on lyso_x06da_ref, cytc_x10sa, myob_x10sa, yag_x10sa_20keV, 4nwv and thau_bl1a_3p8keV, and 2 clip-bound flips each on lyso, yag and 4nwv; p.hkl bit-identical before and after on all six. New knife-edge parity test fails on the old GPU code in both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 16:07:48 +02:00
leonarski_fandClaude Opus 5.5	0903e35f73	Merge perf-8gb-cpu into rc176: the geometry probe beside the merge is decided on bytes, the GPU rotation prediction walks only the l that can cross the sphere, and the mmCIF/SHELX numbers use std::to_chars; output unchanged ... Byte-identical to an rc176 base on GPU and --no-gpu: standard and inhouse timing benchmarks and a run of lyso_x06da_ref, cytc_x10sa, myob_x10sa, yag_x10sa_20keV, 4nwv. With the card faked to 8.55 GB the geometry check drops 0.81 -> 0.17 s (myob) and 0.44 -> 0.09 s (lyso); the GPU Bragg prediction kernel on 4nwv 1968 -> 415 ms on an RTX 5080 (loaded box, kernel time). A merge above ~9 GB (8qaw class) now makes the probe wait for it on a 16 GB card too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 16:07:17 +02:00
leonarski_fandClaude Opus 5.5	c8c33b79e0	Merge twin-isa into rc176: the space-group search asks the twin-immune zone of a lattice-holohedral candidate and no longer reads centring absences as screws; reference_mismatch is a report note ... RESULT-CHANGING. 3r6o I4122 -> I41 (fail -> pass; ISa 4.15 -> 7.77, R-free 0.303 -> 0.270); 8c3e P622 -> P321 (fail -> unscored: the c screw is undeterminable from its sweep, the point group is right). Unchanged on the other 18 sets of a GPU battery (run 20261017-1241_520254_twin-isa-t2b vs 20261016-0057) and on a CPU screen of 53 controls. The reference_mismatch key on 3mc4 and 3r6o is a note in the report only: nothing is skipped or re-scored. The report.py conflict was the note column, which now carries both the SHELXL R_free note and the reference note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 14:45:17 +02:00
leonarski_fandClaude Opus 5.5	f215300a07	ImageBuffer reader: give up on an image MAX_IMAGE_LAG (100) behind the newest, and a unit test ... Images reach the buffer in order give or take a few, so one still missing when the buffer holds an image 100 past it is lost, not late: the read returns false instead of waiting for the end of the collection. The newest image number comes from ImageBuffer::GetStatus, which scans every slot under the buffer's lock, so it is asked only every 100 polls (~10 Hz). The test puts CBOR images into an ImageBuffer the way the receiver does and checks read-back of the pixels, waiting for an image not yet there, the lag limit, the end of the collection and the bounds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 14:27:10 +02:00
leonarski_fandClaude Opus 5.5	9c4e22465a	ImageSpotFinderGPU: the local-box test is the CPU's to the bit (RESULT-CHANGING) ... RESULT-CHANGING for the GPU spot finder (classic and adaptive, which shares this kernel for its local-box passes) - only on pixels where the two used to disagree. Two differences between pixel_result and the CPU's StrongInWindow: - SNR test. Both form T = float(var) * strong2 in float. The CPU compares the int64 square (in_minus_mean)^2 with (int64) ceil(T); the GPU compared it with T directly, which converts the square to float. Below 2^24 the two differ only when the square equals ceil(T) with T not an integer (an exact-integer strong2 such as 9 or 16 never gives that); past 2^24, where T is an integer, the GPU's rounded square can land on T and fail a pixel the CPU accepts. The GPU now uses the CPU's integer-vs-ceil comparison. - Count threshold. At photon_count_threshold = 0 the CPU still requires val > 0; the GPU took 0 as "no threshold". It matters only for a pixel <= 0 above a negative local mean (a pedestal-subtracted detector); the adaptive finder always runs this pass with threshold 0. The GPU now applies val > threshold unconditionally, with the threshold carried as int64 like the setting. Measured: a diagnostic build evaluating both semantics in every pixel_result found zero disagreements in ~2e10 first-pass and ~2.9e7 candidate tests over lyso_x06da_ref, cytc_x10sa, myob_x10sa, yag_x10sa_20keV, 4nwv and thau_bl1a_3p8keV (JUNGFRAU); the battery GPU runs before and after give bit-identical p.hkl on all six sets. The new knife-edge test constructs both cases (a checkerboard window whose square is var * 25 exactly and rounds onto the float threshold; a 0 above a negative checkerboard) and failed on the GPU before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 13:18:19 +02:00
leonarski_fandClaude Opus 5.5	7c1befd484	SearchSpaceGroup: ask the twin-immune zone of a lattice-holohedral candidate too, and do not read centring absences as screws ... RESULT-CHANGING. A tetragonal crystal in 4 twinned ~0.3 by k,h,-l (3r6o, I4_1) was merged in I4_122. Since the beam-stop fix both search arms passed 422 (H ratio 1.51 < 1.85); before it the Lorentz-filtered arm refused at 2.13x and was overridden by the all-observation arm. Either way the twin-immune zone - the one reading a twin cannot fake - was never asked: the Stage A zone gate skipped any candidate that holds every rotation of its lattice, on the argument that no twin law exists for such a group. But the subgroup's twin law is one of the candidate's own operators, so the holohedral promotion is exactly the commonest merohedral-twin case (4 -> 422, 3 -> 32 on a rhombohedral lattice, 321 -> 622), and the L-test-under-merge gate leaves such a twin in the higher group by design. The gate now runs whenever the P1 merge reads twinned (<|L|> in [0.375, 0.44)), holohedral or not. 3r6o reads 422 over 4 at -58 nats (acentric). With 422 refused, the all-observation arm took 222 over 4: the same-order tie-break credits a group's axial two-folds with the screw evidence of their rows, and on a body-centred crystal the I centring kills the odd class of every principal row (30 nats each), so a 222 of the real 2-fold plus two twin laws beat the crystal's own 4. Rows whose odd class a centring extinguishes are no longer counted where the merge reads that centring on its general reflections (the centring class at < 0.25 of the rest's mean E^2; measured 0.11-0.13 for I on 3r6o, 0.97-1.00 for the absent centrings). Changed (CPU screen of 53 sets against the kept rc175 run 20261016-0057, plus 3r6o): - 3r6o: I4_122 -> I4_1 (deposited I4_1), TWINNING INDICATED, twin law h,-k,-l ~0.32; ISa 4.15 -> 7.77, R_meas 0.218 -> 0.146, CC1/2 0.989 -> 0.993. - 8c3e: P622 -> P321 with P3_121 | P3_221 as alternatives, screw on c undetermined (deposited P3_121, twinned): 622 refused at -70.8 nats; ISa 5.69 -> 6.91. Unchanged: 3mc4, 5j23, 6toc, 6oww, 6iu5/6/9, 2wnn, 2xfw, 4bwl, 5ebi, 2wnq, 6vww, 7k1l, 6pxb, 8xte, 8xtg, 9i80, 5uth, 6h5t, 8k1g, 6moj, 7kcn, 5vml, 3p85, 7rji, 8xtf, 9h0q, 7l84, 6v2r, 7t5t, 5m17, 6p8p, 3meb, 5cc8, 9lxl, 8v4j, 6zqr, 9fcf, 5src, 11if, cuhf2, lyso_x06da_ref, lyso_x10sa_90deg_1, thau_x10sa_16keV, thau_bl1a_6p5keV, insu_H_x06da_twin/notwin, insu_I_x06da_ref, cytc_x06da_1, myob_x06da_split, kdp_x10sa_20keV (space group identical, ISa within run noise; 7l84/5src/11if differ only by the model-adopted enantiomorph label). Test: a body-centred 4 twinned at 0.45, given its cell, stays in 4 (222 without the centring rule). [SearchSpaceGroup], [twinning], [twin] pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 13:01:11 +02:00
leonarski_fandClaude Opus 5.5	f117484592	GPU rotation prediction walks only the l of each column that can cross the Ewald sphere ... bragg_rot_kernel_3d ran one thread for every (h, k, l) of the box on every frame, while the CPU predictor already skips the l of an (h, k) column whose phi solution cannot lie within the rocking-curve test's reach (BraggPredictionRot: f(l) = |p0|^2 + 2 S0.p0 bounded by f_max, at most two intervals of l). The GPU predictor now walks the same l: - RotationColumnRange.h: the CPU predictor's phi limit and column bound, moved into a host/device header shared by both, unchanged in arithmetic except that the four divisions by 2 C*.C* are a product with its reciprocal (each end is widened by a whole index, which covers the rounding). - column_range_kernel: the l range of every column, one thread a column, in double; the column's b and c are evaluated as polynomials in h and k from coefficients set up on the host, and the blocks are 32 threads, because a consumer card runs double at a small fraction of the float rate on a unit the warps of a multiprocessor share (a warp per column with the range on lane 0 was slower than the whole box). - bragg_rot_kernel_columns: a warp per column, its l spread over the lanes; per (h, k, l) the same compute_reflections_rot. The output is sorted afterwards (OrderAndSelect), so the predictions and their order are those of the whole box. Kernel time on the RTX 5080 (nsys, whole run): 4nwv 1968 -> 415 ms (bragg 374 + ranges 41), myob 45.5 -> 49.9 ms (small box: the ranges cost more than they save there, ~4 ms a run). On the Windows laptop bragg_rot_kernel_3d was 9.4 s of 4nwv. BraggPredictionRotGPU_ColumnRangeKeepsEveryReflection: on 300 random lattices, geometries and rocking widths the pruned walk predicts bit for bit what the whole box does (the GPU twin of the CPU test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 12:43:20 +02:00
leonarski_fandClaude Opus 5.5	0d7e176b3c	Speculative geometry probe beside the merge decided on bytes, not on a 12 GB card size ... The speculative geometry probe waited for the main line's merge on every card below 12 GB (gpu_takes_turns), so on an 8 GB laptop card it always ran after the merge, as a serial "Geometry check" at the end of the run (Windows RTX PRO 2000: myob 1.4-1.7 s, lyso 0.6 s, 4nwv 3.2-4.4 s, against ~0.15 s where it overlaps). Now it is decided from bytes, up front and never from free memory: - GPUMerge::Holds(bytes): the main line says what its merge holds of the card once the scaling engine is ingested - the engine's DeviceBytes (an upper bound), twice where the all-observation arm gets an engine of its own - before the post-refinement that may start the probe is let go. A merge that has not said yet counts as leaving no room. Before the model stage it says the whole card: model validation and MR budget 80% of the card for themselves. - GPUWorkBeside(after_merges, bytes): the probe waits for the merges only where its own bytes and theirs exceed 80% of the first card's total (the same 80% the merge's and the model's GPU budgets use; on WDDM the rest is the paging margin). The image-pass / probe-pass turn-taking below 12 GB is unchanged. - Rugnux::IndexingProbeGPUBytes: the probe's first-pass spot workers at 24 bytes a pixel (GPUWorkersPerCard's figure) plus 0.65 GB per FFT indexer (1.24 GB measured for two). The spot-worker count is now one function (FirstPassSpotWorkers) for both. - The model stage waits for GPU work beside to end before it plans its GPU engines. Estimates on an 8.55 GB card: myob probe 4.1 GB + merge 0.6 GB -> beside (was: after); 4nwv probe 3.9 GB + merge 4.2 GB -> still after. On the RTX 5080 (16.3 GB): myob 4.8 + 0.6, 4nwv 4.5 + 8.4 = 13.0 of 13.04 GB -> beside, as before. Concurrency only: what each pass computes is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 12:41:45 +02:00
leonarski_fandClaude Opus 5.5	2f3885259f	Merge mr-gpu-r2 into rc176: MR and model validation spread their rigid-body engines over the cards, and MR packs the transform where its cost model expects it faster; output unchanged ... Validated on the 4x T4 server (round 2): p.hkl, p_model.pdb, MR_* and the candidate table identical on 9zlo/9s02/9gjx/4nwv against round 1 on 1 and 4 cards; 4nwv on 4 cards 140.8 -> 127.9 s wall (MR 34.0 -> 21.2 s; rotation 18.4 -> 4.8 s), 9zlo/9s02/9gjx 1-3% faster. Conflicts were the crash fix both branches carry: the branch's side taken (the per-card Reserve of the device test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 12:35:42 +02:00
leonarski_fandClaude Opus 5.5	aca27be862	jfjoch_writer writes rugnux's .mtz, _maps.mtz and _model.cif from the END blocks ... New gemmi-only library JFJochRugnuxFiles (writer/), linked by JFJochWriter, so the writer needs no analysis library and builds with JFJOCH_WRITER_ONLY: - WriteMergedMtz / WriteMapCoefficientsMtz / PlacedModelStructure make the files from the structured blocks; WriteRugnuxResultFiles writes whichever blocks END carries beside the master, each under a temporary name and renamed into place, an existing file replaced only when the run's overwrite is set. A file that fails costs only itself (logged), not the run. - FillRugnuxResultBlocks reads the blocks back from the files rugnux wrote at a prefix (p.mtz, p_maps.mtz, p_model.cif), for the broker to fill END. WriteMtzReflections now goes through WriteMergedMtz (MergedReflectionsForWriter builds the block); its output is byte-identical to before, checked by md5 on a synthetic set in both the Friedel-merged and the anomalous layout. StreamWriter writes the files after the master and before the END ACK, so the ACK means they are in place; END receipt and each rename are logged at INFO with the wall-clock time, for last-frame-to-files latency. The broker's direct HDF5 pusher writes them too. Test: rugnux's three files -> blocks -> CBOR -> writer reproduces all of them (and _model.pdb) byte for byte; a ~25k-reflection END is 2.7 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-17 12:19:51 +02:00
leonarski_fandClaude Opus 5.5	bc5577d32c	Merge mr-multigpu-crash-fix into rc176: rigid-body engine calls restore the caller's card and the MR GPU functions run on their own card, fixing an illegal memory access in MR with engines on other cards; output unchanged on one card ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 11:00:42 +02:00
leonarski_fandClaude Opus 5.5	a47f030074	Merge singular-fix into rc176: the model-scale fit no longer stops on a singular normal matrix (a reflection whose |Fcalc| the solvent cancels exactly; an underflowed Levenberg-Marquardt step), output unchanged where it ran before ... Fixes the GPU-path crash after MR on 8sqo, 8v4o and 9zmu (all three now place in the deposited group) and the CPU-only ModelValidation_CCModelUnrelatedModelConfirmsNothing failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 10:51:05 +02:00
leonarski_fandClaude Opus 5.5	210664f454	Model scale fit: no log ratio at |Fcalc| = 0, and no LM step once the model has underflowed ... Two ways the scale fit's normal matrix went singular and gemmi's jordan_solve threw "Trying to reverse singular matrix. Column 0 is zeroed.", ending the run: 1. The isotropic start (fit_isotropic_b_approximately) takes log(Fobs/|Fcalc|) over the strong reflections. At a phase-restricted reflection whose Fcalc and Fmask share a phase, some solvent pair of the k_sol/b_sol grid can cancel the model exactly in float, |Fcalc + k_sol exp(-b_sol s^2) Fmask| = 0. log(Fobs/0) = inf made that grid point's k_overall and B NaN, every sum of the Levenberg-Marquardt NaN, and the solve failed. This is what stopped three open-arm sets in the output pass after MR on the GPU (the scale fit after model validation's rigid body): one strong reflection, one grid point. The CPU path has the same exposure; which grid point hits an exact zero depends on the last bits of Fcalc, so the --no-gpu run of the same set went through. Such a reflection has no log ratio and is now left out of the start - in the GPU kernel, in FixedSolventFit::Start and, as a marked local change, in gemmi's own fit_isotropic_b_approximately, so the three stay equal and the held-solvent fits are covered too. 2. In FitModelScale's CPU Levenberg-Marquardt (ModelValidation_CCModelUnrelatedModelConfirmsNothing in a CUDA=OFF build): on 66 reflections of an unrelated model the fit walked to k_overall -424 and a B where k exp(-B s^2) |F| underflows at every reflection; the squares in J^T J were all 0 while J^T r kept a residue near 1e-210, which jordan_solve rejects. The fit now ends there, at its best so far, as it would at the lambda limit; that grid point then loses on R. The GPU fit takes the derivatives in float and their squares in double, so it cannot reach this state. Both changes only touch fits that threw before. Byte identity checked against an rc176 build on the standard and inhouse timing benchmarks and on 4nwv, 8tyy, 6oel (p.hkl, p_model.pdb, MTZ data, MR_* report keys). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 10:48:36 +02:00
leonarski_fandClaude Opus 5.5	80f634b641	MR GPU: engines spread over the cards; the transform packed where it is expected faster ... Engine placement: the rigid-body engines past this card's first go one card after the other in turn, from the next card and this one last, each within half of its budget. A small cell on four cards (24 engines of 0.09 GB) had put all of them on this card, which has room for them, where rc176 spread them; now 6/6/6/6. A large cell on four 15.6 GB cards keeps T+1/T+2/T+2/T+2. Layout: packing was refused wherever it cost any card an engine, which kept 4nwv plain on four T4s although the packed rotation function on ONE card (15.1 s) already beat plain over four (18.5 s). Now the two layouts' plans are compared on a small cost model, decided once before the search from sizes and counts alone: seconds = rotation interpolations (orientations x reflections x operators) x s/interpolation / cards with a copy + rigid-body fits at once x engine GB x s/(fit GB) / engines Constants measured on 4nwv on Tesla T4 (17255 x 244183 x 4 interpolations): 75.8 s plain and 15.1 s packed on one card; rescoring + null 28.5 s with one engine and 11.6 s with seven, 9 fits of 2.10 GB, so 19.7 s shared among the engines. Packing must still keep the search, the scores and (where plain has it) the rigid body on the GPU, and the budget must hold both grids while the packed one is made. Expected on four T4s: packed T+1/T+1/T+1/T+1, 8.7 s against 21.8 s plain. One card is unchanged (the engines are the same either way, so packed). The layout line now logs both expected times and the search's size. Layout and engine count change where the work runs, never a number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 10:34:09 +02:00
leonarski_fandClaude Opus 5.5	3fa17b9385	Model validation: the rigid-body engines on every card in turn, even where this card has room for all ... The engines past this card's first now go one card after the other, from the next card and this one last, each within half of its own budget, as MR's do. Before, this card took every extra engine it had room for and the other cards only the rest, so a small cell on four cards ran all its fits on one. Where this card has no room for a second engine (a large cell on 15.6 GB cards) the plan is unchanged: 1/3/3/3. This card's scale fits take the room its engines leave. Engines are interchangeable and deterministic, so only where fits run changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 10:34:05 +02:00
leonarski_fandClaude Opus 5.5	46c44e04d9	Rotation scaling: a host-memory budget decides what runs side by side ... The run's budget (ProcessConfig::host_memory_budget_bytes, else 75% of the machine's RAM) is decided once and logged, and sizes the host worker pool. Three concurrencies of the scaling chain are decided on it from counts known before each starts (RotationScaleMerge::HostIngestBytes, HostMergeBytes, HostBytesBound), each logged where it serialises: - the quality guard's merge beside the main ingest: where they would not fit together, the guard merges first; - the canonical pass's post-refinement beside the merge chain: done before the first merge; - the all-observation P1 search merge beside the main engine's (GPU path): started where the search first reads it, after the main engine's merges and the release of their scratch. None of these reads what the other writes while it runs, so only the order changes: a [large] test runs the rotation set at a 1 TB and a 1 MB budget and compares the .hkl byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:31:24 +02:00
leonarski_fandClaude Opus 5.5	3ebf085e4c	Host memory: the machine's total, a budget decided from it, and a pool for large per-worker buffers ... get_host_total_memory (sysconf / hw.memsize / GlobalMemoryStatusEx), DefaultHostMemoryBudget at 75% of the total - decided from the total, never from what is free, as the GPU plans are - and HostWorkerPool, a process-wide pool (a third of the budget) that workers reserve their large buffers against before making them, so how many run at once is decided by bytes rather than threads. Only scheduling reads any of it. [HostMemory] tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:31:04 +02:00
leonarski_fandClaude Opus 5.5	b004a846dd	RigidBodyGPU_CallsLeaveTheCallerOnItsCard: one engine on each card with the per-card Reserve ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:24:40 +02:00
leonarski_fandClaude Opus 5.5	fbdf036e6f	Rigid-body engines leave the calling thread on its own card; the MR search runs on its transform's card ... Every RigidBodyGPUEngine call did cudaSetDevice(engine's card) and never set the caller back. Molecular replacement's group threads refine copies on the pool's engines - on any card where every engine on this one is taken - and then run the next translation function on the same thread. MolecularReplacementGPU ran its calls on whatever card the thread was on, so that function's buffers and cuFFT plan were made on the engine's card, beside the transform and the stream on the search's card: an illegal memory access, 9s02 with four cards visible, every run (the further-copies loop: refine a trial on another card's engine, then translation_function on the group thread). - Each engine call (and the constructor) now holds a DeviceGuard: on the engine's card for the call, back on the caller's after it. - MolecularReplacementGPU remembers the card its transform was uploaded to and runs every call (and its destructor) there, whichever card the calling thread is on. - RigidBodyGPU_CallsLeaveTheCallerOnItsCard: one engine per card, all taken from one thread; the thread's card is unchanged after each call (meaningful with two or more cards). No number changes: where every call runs is the same card as before for each object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:24:25 +02:00
leonarski_fandClaude Opus 5.5	a3f30d03c1	Rigid-body engines leave the calling thread on its own card; the MR search runs on its transform's card ... Every RigidBodyGPUEngine call did cudaSetDevice(engine's card) and never set the caller back. Molecular replacement's group threads refine copies on the pool's engines - on any card where every engine on this one is taken - and then run the next translation function on the same thread. MolecularReplacementGPU ran its calls on whatever card the thread was on, so that function's buffers and cuFFT plan were made on the engine's card, beside the transform and the stream on the search's card: an illegal memory access, 9s02 with four cards visible, every run (the further-copies loop: refine a trial on another card's engine, then translation_function on the group thread). - Each engine call (and the constructor) now holds a DeviceGuard: on the engine's card for the call, back on the caller's after it. - MolecularReplacementGPU remembers the card its transform was uploaded to and runs every call (and its destructor) there, whichever card the calling thread is on. - RigidBodyGPU_CallsLeaveTheCallerOnItsCard: one engine per card, all taken from one thread; the thread's card is unchanged after each call (meaningful with two or more cards). No number changes: where every call runs is the same card as before for each object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:24:16 +02:00
leonarski_fandClaude Opus 5.5	38e75c8e5b	HDF5Writer_Socket: wait for the subscription before the first file is announced ... The test connected its SUB socket and wrote at once; file 1 is closed and announced on the PUB socket right after the second image, and PUB drops whatever it sends before the subscription has reached it (slow joiner). On a loaded CI runner the first notification was lost and the test read file 2 where it expected file 1. Wait 2 s after subscribing, as the preview and metadata socket tests already do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-17 09:06:54 +02:00
leonarski_fandClaude Opus 5.5	0f461a65a2	Merge mr-transform-layout: the corner-packed molecular transform beside MR on every card ... Combines mr-multigpu (per-card engine budgets, a transform copy on each other card, the rotation/translation functions split over the cards) with mr-transform-layout (the GPU transform optionally held corner-packed, same bits, 2.2x faster rotation function). Conflict (MolecularReplacement.cpp): the two new functions side by side (PackMolecularTransformGPU next to SplitRotationFunction/SplitTranslationFunction). On top of the textual resolution: - One layout decision for all cards, not one per card: the functions are split into equal slices, so a mix of packed and plain cards would run at the speed of the plain ones; either layout gives the same bits. - PackMolecularTransformGPU now plans with the other cards' budgets too and packs only where every card keeps what the plain plan gives it: this card's transform, engines and score engine, and on every other card its copy and its engine count. A packed copy is made after the card's engines are reserved; copy and engines within half of that card's budget leave room for the plain grid it is made from. - The copies on the other cards are made in the chosen layout, and the per-card plan (copies, engines, room) is made with the bytes actually placed. - Test: four 15.6 GB cards stay plain (packed, each other card would fall from T+2 to T+1); with 24 GB cards beside, packed. So on a 4x T4 4nwv runs split and plain, on one card packed; whether giving up 3 of 7 engines for the packed rotation function would pay there is left to a server measurement. Workstation (1x RTX 5080, --gpulock), against rc176: battery timing --benchmark standard (9zlo, 9s02, 9gjx, 4nwv) and --benchmark inhouse (myob, cytc, 4nwv) on GPU, inhouse also --no-gpu: p.hkl, p_model.pdb and the MR_ report block byte-identical on every set. 4nwv GPU: rotation 5.53 -> 2.50 s, MR step 20.1 -> 16.5 s (packed, 2.28 GB in place of 0.84 GB). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-16 18:46:58 +02:00
leonarski_fandClaude Opus 5.5	3aa67a6059	MR GPU: corner-packed molecular transform for the rotation function ... Each sample of the molecular transform interpolates over 8 grid corners that lie in four u-rows of the half-l grid, 4-5 32-byte sectors per sample. On the GPU the transform can now be held packed: every point (i, j, l) stores its four (v, w) corners, so the two neighbouring records hold all eight, read from two places. The packed grid is cut to |u| <= reach as the plain one already is in v, and has the conjugate plane l = -1 in front; below l = -1 the eight corners are read mirrored about -i0 - 1 and conjugated. Same floats, same accumulation order: rotation and translation functions give the same bits (checked in MolecularReplacement_GPUMatchesCPU). The layout is chosen up front from the budget (PackMolecularTransformGPU): packed where the answer merge's plan holds every part beside it that it holds beside the plain transform, and the budget holds both grids while the packed one is made from the plain one on the device; logged either way. The upload no longer stages a host copy of the grid. The CPU At() is unchanged: a microbenchmark gained 1.3x single-threaded and ~1.0-1.1x at 16-32 threads, not worth 2.7x the host memory. 4nwv (moved model, timing benchmark), RTX 5080: rotation 5.56 -> 2.48 s, MR step 20.9 -> 16.7 s; transform on the card 0.84 -> 2.28 GB. p.hkl, p_model.pdb and the MR_ report block byte-identical on the timing sets (GPU and --no-gpu) and on 4nwv, 8tyy, 6oel, 3inp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-16 17:55:30 +02:00
leonarski_fandClaude Opus 5.5	bfc6ca8487	MR and model validation: use every card's memory, split MR's functions over the cards ... On a multi-GPU host the structure stage ran essentially on one card: the rotation function's launches all went to the search's card, and the rigid-body engines of model validation and MR were admitted only within half of that one card's budget, although RigidBodyGPUPool already placed engines on other cards. On a 4x T4 server 4nwv's MR rotation function took 75.8 s on one card, validation queued its 10 fits on one engine and MR's rescoring and null got one engine. - PlanModelValidationGPU / PlanMolecularReplacementGPU: this card is planned exactly as on a machine with it alone (GPU-vs-CPU placement unchanged: 80% of the total, less what is resident). The rigid-body engines it has no room for go on the other cards in turn, each against its own budget (80% of its total, less the resident bytes, counted on every card) within half of it. MR also puts a copy of the molecular transform on each other card within half of its budget. - RigidBodyGPUPool::Reserve takes per-card engine counts. - Validation forwards the most it holds on any card as gpu_bytes to a validation beside, and may run one beside only where twice that fits every card's budget. - MR splits every rotation and translation function into contiguous slices of the orientation list over the cards holding a transform copy (and room for the call's buffers), one thread per card, gathered back in order. Each orientation is scored by itself (one block / one map), nothing is summed across orientations, so N cards give the bytes of one. - ScoreEngines keeps a room per card; score engines always go on the search's own card; function buffers are held on the card they run on. Byte-identical at N=1 (workstation, one RTX 5080): timing benchmark (myob, cytc, 4nwv) and 4nwv, 8tyy, 6oel, 3inp with --model give the same p.hkl, MTZ data, p_model.pdb, maps and report against rc176. New tests: per-card plans on four 15.6 GB cards (validation 1/3/3/3 engines, MR T+1/T+2/T+2/T+2), and the split driven over 1-4 virtual cards on one device, bitwise equal to one card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4	2026-10-16 17:16:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00