58 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	b736c4cefb	RotationScaleMerge::DeviceBytes: the resident ingest's peak as measured ... The ingest term was 90 bytes per reflection and 64 per run. Measured with cudaMemGetInfo at each step of IngestUpload/IngestSort (relative to the free memory before the observation arrays): - the second radix sort is the peak where runs are few: the 66 B of observation arrays, by_image, perm, key, key_sorted and cub's alternate key/value buffers of a plain SortPairs - 102.3-103.0 B per reflection on 7qij (51.1 M, 52.2 M, 78.1 M reflections) and 4nwv (32.7 M); - the cut is the peak where runs are many: 98 B per reflection plus 56 per run (4nwv, 15.4 M runs: 125.8 B per reflection, 4106 MB against 3923 MB from the old formula); - the upload holds the 66 B beside the two 64 MB staging buffers (~150 MB measured). Now max(103 B per reflection + 58 B per run, 66 B per reflection + 160 MB): 2-4% above every measured peak, where the old formula was 10% under on 7qij (7170 MB against 7986 MB for the main engine) and 4.5% under on 4nwv. Its only consumer is whether the all-observation arm goes beside the main engine, which changes when that merge runs, not what it gives. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 08:27:36 +02:00
leonarski_fandClaude Opus 5.5	eb7a65d051	rotation merge: the fulls' frame and group CSRs built on the card ... Every merge downloaded the fulls' frame and group keys and built both CSRs by serial counting sorts on the host - four passes over the fulls, two of them scattering into an array as long as the distinct hkl (15 M on 4nwv) - and uploaded them again. They are now stable radix sorts of the resident fulls on the card, with the counts summed there; the host gets the counts per frame and the group CSR it walks itself. The same permutations (stable sorts list each frame's and each group's fulls in index order, as the counting sorts did): p.hkl identical on cytc and 4nwv. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-13 02:01:30 +02:00
leonarski_fandClaude Opus 5.5	095814baed	rugnux: place the quality guard from the pre-pass engine's size, before either ingest ... On a set too large for the quality guard's engine to sit beside the main one (8qaw: 123 M observations; 8a1a), the main engine was ingested, found not to leave room, dropped and ingested again after the guard - several seconds of ingest for nothing. The pre-pass's engine merged the guard's very frames earlier in the run, so its card memory per reflection is now kept (prepass_engine_bytes_ / prepass_engine_reflections_) and both engines are reckoned from it before either ingests: the guard at its frames' reflections, the main engine at all of them (observations add in proportion; rocking events and distinct hkl at most in proportion). Beside where the two fit the card less the 3 GB reserve, otherwise the guard first. The re-ingest path is gone. 7qij, 9v16, cytc: beside; 8qaw, 8a1a: guard first. Same merges, same numbers. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 23:31:49 +02:00
leonarski_fandClaude Opus 5.5	d5bf32763f	rotation merge: the resident ingest smooths the geometry on the card (last-bit change) ... The rest of the resident ingest moves to the card: the delta_phi re-solve against each frame's smoothed lattice, the one exact-Bragg angle per rocking event, the partiality recomputed from the smoothed mosaicity (with corr brought to it) and the histogram of the frames an event spans. The host keeps the per-frame work (the smoothed lattices, the mosaicity window) and reads back only corr; nothing per observation crosses back, and the host arrays the smoothing worked on are no longer built on this path. The integer work is the host's to the bit (the event-span histogram). The floating-point work differs in the last bits: every rounding on the card is explicit (no contraction, so the same bits on any GPU), the host's compiler fuses some of the same products into sums, and erf/atan2 come from the CUDA library. This is the one place in the lever where results change; see the hand-back report for the sets and metrics it moves. Measured on 7qij (main engine, 78 M reflections): the whole ingest 6.5 s host + 1.3 s device before this lever -> 2.9 s, of which 2.4 s is the upload (memory-bound, beside the post-refinement gather) and 0.25 s this smoothing. Same space group, error model, resolution and reflection count; IMEAN differs at the 1e-7 level except on near-zero intensities. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 22:46:16 +02:00
leonarski_fandClaude Opus 5.5	f88ff0440d	rotation merge: the resident ingest sorts and cuts the reflections on the card ... With the GPU resident, the host ingest swept the source reflections four times (background and h histogram, sort keys, the in-range build, the device staging), bucket-sorted the keys and cut the raw-hkl runs: 5-6 s of host time per engine on a 78 M-reflection sweep, run twice beside the quality guard. Now the frames' reflections are copied once, through two pinned buffers, to the card, where the per-frame background is summed, the reflections are ordered by two stable radix passes ((image, index), then raw hkl - the one order the host's total comparator gives), cut into runs by run-length encoding, filtered by the resolution range and compacted. The observations, the runs and their order are the host ingest's to the bit; a new GPU test checks the layout against the host's sort and cut. The geometry smoothing still runs on the host in this step, on arrays read back. The all-observation arm's engine is cloned after the main engine's device ingest (its DeviceBytes are known only then) and copies the runs on the card as well. DeviceBytes now also covers the ingest's own peak (90 bytes per reflection, 64 per run before the cut). Measured: p.hkl and MTZ data identical on cytc and 7qij. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 22:38:25 +02:00
leonarski_fandClaude Opus 5.5	7d656c13de	rugnux merge engine: the immutable fields the scaling gathers in one 16-byte record ... The group reduction reaches every observation through the group permutation, and so fetched one 32-byte sector per field for I, sigma, partiality and corr. I, sigma, partiality and prescaling_corr now sit together in one aligned record, so it fetches two sectors per observation; the grouping kernels' finiteness test and the combine read the same record. The per-observation fit coefficient, its flag and 1/sigma are no longer stored: the per-frame fit forms them as it reads each observation (the same expressions, so the same bits), which removes PrepScaleObsKernel and 13 bytes per observation of device memory. DeviceBytes' per-observation figure drops by those 13 bytes. Every sum keeps its order: p.hkl md5 identical to the base on cytc, 8a1a and 8qaw. RTX 5080, nsys, merge-engine kernels: 8a1a 2459 -> 1924 ms, 8qaw 3624 -> 3025 ms, cytc 501 -> 476 ms (ReduceGroupMeans 1.5-1.6x faster, PrepScaleObs gone, FitPerFrameG absorbs it). Measured and not taken: storing all per-observation arrays in group order makes the reduction 3-4x faster on its own, but the per-frame fit then has to gather, and moving 15 arrays at every grouping costs as much as the reduction saves (net 0 on the 5080); a record for the combine-only fields did not speed the combine. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 11:21:49 +02:00
leonarski_fandClaude Opus 5.5	e55750e1e0	rugnux merge ingest: keep d beside zeta only under an energy bandwidth ... The partiality recompute reads d only to broaden the rocking curve by the energy bandwidth. On a large set the final ingest sits at the process's peak resident memory (8a1a: max RSS rose by the array's 4 bytes x 68 M observations), so the array is kept only where a bandwidth is set; with zeta in and the predicted-partiality copy out, the ingest's peak is then where it was. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 12:38:04 +02:00
leonarski_fandClaude Opus 5.5	b46b4a4657	rugnux all-observation arm: observations copied on the card, not staged from the reflections again ... The all-observation arm of the space-group search merges on a copy of the engine (CloneIngest), and its IngestDevice staged every observation from the ~92-byte source reflections a second time - concurrently with the main engine's own staging of the same reflections. Now the main engine stages, and the copy takes its card arrays with cudaMemcpyPeer (a device copy when both engines share the card, as on a one-card machine). CloneIngest no longer copies the predicted partiality, which only that staging read. Output identical (cytc p.hkl md5, and with --dump-observations the dump md5). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 10:52:57 +02:00
leonarski_fandClaude Opus 5.5	42ed7261d3	rugnux merge ingest: four sweeps of the source reflections instead of seven ... The ingest read every ~92-byte Reflection seven times per engine: the background/h-range sweep, the h histogram, the key scatter, the observation build, the delta_phi re-solve against the smoothed geometry, the partiality recompute, and the device staging. On a box with several sockets these are remote, latency-bound loads (the owner's 4-socket profile puts RotationScaleMerge's ingest lambdas at the top of the CPU time on most large sets). Now: - the histogram of h is counted in the background sweep, per chunk of frames over the whole int16 range the sort key holds, so the h range needs no sweep of its own; - delta_phi is re-solved against the smoothed per-frame lattice while each kept observation is built (SmoothedFrameGeometry computes the per-frame bases first, SmoothedDeltaPhi solves one); - the build keeps zeta and d beside delta_phi, so the partiality recompute reads compact arrays, and it brings corr_ingested to the recomputed partiality in the same loop, which retires the observation-length copy of the predicted partiality and its separate pass. Output identical (p.hkl md5 on 9gdj and cytc; cytc also with --dump-observations, dump md5 equal, and CPU-only). Transient host memory: +4 bytes per kept observation inside IngestHost (zeta and d in, the predicted-partiality copy out) and nthreads x 256 KiB of histograms. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 09:42:00 +02:00
leonarski_fandClaude Opus 5.5	9d6a3f1e04	rugnux merge engines: count-based device-memory estimate for the all-observation arm beside ... The gate for merging the space-group search's all-observation arm on a copy of the engine, beside the first arm, judged an engine at a fixed 256 B per observation. It now counts what the ingest knows - observations, rocking events (at most one full each) and distinct hkl (at most one ASU group each) - at 80 / 260 / 200 B, the envelope of the measured per-engine peaks (helpers included: error-model fit, Wilson outlier test, pair sort), plus a fifth, and keeps 3 GB of the card for what else stays resident (context, the run's indexers: measured up to 2.3 GB at the gate with no speculative pass running). Measured per-engine peaks (scratch build tracking each engine's device allocations): 4nwv 7.98 GB, 8a1a 8.15, 8tyy 9.36, 8qaw 13.3 - two engines do not fit a 16 GB card on any of these, so they stay serial. The old constant was not an over-estimate there (fulls cost ~260 B each at the surface fit). What changes: 9 battery sets of 33-53 M observations at high multiplicity (8s38, 8sqt, 7ph1, 9min, 9i80, 9sl0, 9h0q, 8v4o, 6ze4) now merge the arm beside; none goes the other way. Their merge-step VRAM peak is 11.0-14.1 GiB of 15.9, no out-of-memory, no failed speculative probe. Every engine peak measured on 11 sets is at most 81% of the estimate. Output identical (p.hkl md5 and MTZ data, 40 sets incl. ci tier). Gain is small: the two merges share one GPU, so the arm beside slows the main P1 merge by ~0.5 s while saving ~0.7-1 s of wait at the search; alternating A/B on 5 of the sets: -1.1 s output pass in total. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 08:09:58 +02:00
leonarski_fandClaude Opus 5.5	21f76679c4	rugnux: merge and search the all-observation arm beside the first one ... The space-group search made its two P1 merges one after the other on one engine: the Lorentz-filtered one, its search, then the all-observation one and its search. Since every merge now starts from the same state, the all-observation merge is made on a copy of the engine (CloneIngest: the host half of the ingest copied, the device half staged on the copy's card), started right after the ingest and read where the search needs it, so it runs beside the P1 merge, the re-scaled search merge and the first arm. Its search then runs beside the first arm's. Placement: from three cards up the copy gets card 2 to itself (card 0 has the main engine, card 1 the quality guard's). With one or two cards it shares the last card, and only where two engines of this size fit the card's total memory (DEVICE_BYTES_PER_OBSERVATION = 256; two engines measured at ~150 B/observation each), so the choice never depends on what is free at the moment. On a 16 GB card that admits sets up to ~32M observations; larger ones keep the serial order. Nothing of the copy reaches the run before the first arm says the arm is wanted: its log, its result fields and its per-frame write-back onto the outcomes are taken at the point where the serial merge would have been made. A copy that failed is merged again in series. Results are bit-identical to the serial order (p.hkl md5 and MTZ data), at default -N and -N 8. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 21:40:03 +02:00
leonarski_fandClaude Opus 5.5	6e8cf93668	RotationScaleMerge: every merge starts from G = 1 ... Each merge restored corr to the ingest state but kept g_partial from the merge before it. A frame the new merge did not fit kept the G an earlier merge had fitted it with, and the fulls' CC1/2 weight (frame_scale_fulls) read it, so a merge depended on which merges ran before it on the same engine. RunArm now resets g_partial to 1 at its start, as the first merge on an ingest already had it. That is what lets two merges run side by side on two engines and give what they give in series. The fits themselves never warm-started (FitPerFrameG and the device kernel both solve each frame afresh), so the change only reaches frames that a merge leaves unfitted after an earlier one fitted them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:48:11 +02:00
leonarski_fandClaude Opus 5.5	4120b0daa8	Merged mmCIF: drop the scaled unmerged block ... The unmerged observations already leave a run as the unmerged MTZ (and the scaled fulls as the .hkl), and deposition does not come straight from rugnux, so the _diffrn_refln block made the .cif up to ten times larger for no consumer. The .cif is one data block again; the keys, software rows, metadata, R_merge/R_pim/CC* and provenance stay. ScaledFull loses the scale and frame it carried only for that block. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:51:56 +02:00
leonarski_fandClaude Opus 5.5	fc6b91555f	Merged mmCIF: deposition keys, R_merge/R_pim/CC*, experiment metadata, scaled unmerged block ... The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on _reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/ _diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a loop naming Rugnux for 'data reduction' and for 'data scaling'. The experiment as the input states it: _diffrn.ambient_temp from the sample temperature, the source's general class (from NXmx /entry/source/type, which the HDF5 reader now reads), facility and beamline, the detector make/model in _diffrn_detector.type (where .detector used to carry it) and its class in .detector where the name settles it, and the collection date from /entry/start_time. R_merge and R_pim are formed beside R_meas from the same per-reflection sums (stills and rotation merges) and written with CC* overall and per shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the _reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as three new columns of the shell tables. R_pim following Weiss (2001), CC* following Karplus & Diederichs (2012). A second data block, data_sample_unmerged, carries the scaled unmerged observations of a rotation merge as _diffrn_refln (measured index, frame, scaled I and sigma, pdbx_scale_value) - the same records as the .hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now carries the scale and the frame. The unmerged MTZ stays unscaled. Reflection data are unchanged: the merged _refln rows, the .mtz columns and the .hkl are written as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:50:11 +02:00
leonarski_fandClaude Opus 5.5	b43844ae06	rugnux: the merging engines sit on their card's NUMA node while they drive it ... The image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two threads that drive the merging tail were not: the quality guard took its card with set_gpu(1), and the main engine runs on the pipeline's own thread, which goes on to everything else. On a multi-socket server their pageable copies to the card went through a remote socket. RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost, IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask themselves; the new test checks both that and that a pinned caller keeps its own mask. No flag, no environment variable. On one NUMA node (this workstation) only the device setting happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base = rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa, cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs), -N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise, e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 12:51:38 +02:00
leonarski_fandClaude Opus 5.5	4be71702ec	RotationScaleMerge: hand back the surface buffers before the merge, so the GPU Wilson test fits ... The device correction-surface fit (gpu-tail2) keeps its per-term arrays - the terms, their parity, the group CSR, the three subsets' orders and the per-term weights, about 60 bytes a term - as engine members, and nothing freed them after the last surface. They stayed on the card through the merge, where the GPU Wilson outlier test (gpu-tail) then asks for ~96 bytes per full. On a large rotation set (139.8 M partials, 13.35 M fulls, 16 GB card) the engine's resident arrays left 1242 MiB free for the first Wilson test and 998 MiB for the one after the automatic resolution cut, against 1222 MiB needed: the run stopped with "Wilson outlier test ... run the CPU build" although nothing ran beside it. The surfaces' buffers are released once the last surface is fitted (the next pass's surfaces build their own), which gives back 905 MiB there. The up-front checks of the three dataset-sized tail steps (Wilson test, pair sort, error-model fit) also read free memory as cudaMemGetInfo reports it, which counts what the allocation pool holds with no buffer in it as taken. They now hand that back and, only if it is still short, wait for GPU work running beside the merge (GPUWorkBeside, the speculative probe) to end before giving up - as the engine's own allocations already do. That changes when a step runs, never what it computes; a set that does not fit still fails at once with the same "run the CPU build" message. p.mtz: the large set completes (and is identical with the Wilson test forced onto the host); myob 9bdd6916 and 8a1a 309ea9a8 unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 20:48:01 +02:00
leonarski_f	1abe52cb64	Merge branch 'par-merge' into rc175	2026-10-08 15:58:29 +02:00
leonarski_fandClaude Opus 5.5	3fa95eea29	RotationScaleMerge: the R_meas staging as members, its group pass on all threads ... The R_meas accumulators per group and the device's per-group sums were eight group-length locals plus a 56-byte record per group, value-initialised on one thread for every merge that reports statistics. They are now members, reset in parallel, and the device branch's pass that assigns each group its shell and copies its sums runs on all threads; the per-shell observation counts are integers, gathered per block. The shells' R_meas sums are still added in group order, as before. p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 12:37:29 +02:00
leonarski_fandClaude Opus 5.5	308d1ed258	RotationScaleMerge: export the merged reflections on all threads ... Which groups are written is decided first (KeptIndices, in group order), then each written group's MergedReflection is formed on its own, on all threads, into a result sized once - where the serial loop pushed back up to millions of records, growing a vector that each merge allocated afresh. The same records in the same order. p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 12:37:29 +02:00
leonarski_fandClaude Opus 5.5	76509be727	RotationScaleMerge: MergeAndStats' per-pass arrays as members, reset on all threads ... The rejected flags and the observation keys and halves (per full), the merged intensities (per group) and the Wilson observation records were locals: on a large cell every one of the run's merges allocated, page-faulted and zeroed them again on one thread - hundreds of megabytes per merge on 8qaw. They are now members, only resized, and reset with ParallelChunks where a pass needs them reset. The error-model samples are no longer cleared before the device download overwrites them (the host path still clears before it appends). p.mtz md5 unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 12:37:29 +02:00
leonarski_fandClaude Opus 5.5	08b318f939	RotationScaleMerge: build the correction surfaces' terms on the device ... With the fulls resident (the GPU combine ran), ApplyCellSurface no longer builds its terms on the host and uploads them: SurfaceBuildTerms makes them from the resident fulls, this surface's cell per full and the current corr (the only two uploads) - which fulls give a term, the terms in fulls order, the group CSR over them and each subset's per-(block, cell) segments - and SurfaceShellWeights the resolution gauge's inputs: the shell edges (order statistics of 1/d^2 read off a sort), each group's shell and each (cell, shell) bin's weight. Every order the host made is a stable counting sort, which a stable radix sort of the indices reproduces; the per-bin weights are summed one bin per thread in fulls order with the host's rounding. The existing exact surface kernels read them as before. Without resident fulls (the observation dump) the host path runs. p.mtz md5 and the surfaces' log lines unchanged on myob, cytc, thau, kdp, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 11:41:15 +02:00
leonarski_fandClaude Opus 5.5	ef90b933da	rugnux: the quality guard's merge on a second GPU, beside the main engine ... With two GPUs or more the two-pass quality guard's merge of the pre-pass frames gets a scaling engine of its own on device 1 (RotationScaleMerge:: SetGpuDevice), and the main engine's device ingest and first merge no longer wait for it to hand the card back. The first merge waits only for the guard's ingest, the last time the guard reads the outcomes: that merge writes per-frame fields (mosaicity among them) back into them. Neither merge reads what the other writes, and a merge gives the same bits on any card of one model, so only when the guard runs changes. With one GPU, and in the CPU build, the order is the one it was. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 11:15:29 +02:00
leonarski_fandClaude Opus 5.5	50006eff88	Thread count from the CPUs the process may use, not the machine's ... std::thread::hardware_concurrency() counts every CPU of the node, so a job given a CPU set (taskset, a cpuset, a Slurm allocation) or a container CPU quota started one thread per node CPU: the default -N, the shared worker pool and every "0 = all threads" default. AvailableCpus() (ThreadAffinity) counts the start affinity mask, capped by the cgroup CPU quota (v2 cpu.max, v1 cfs_quota/period, the process's own cgroup and then the mounted root), read once. Outside Linux it is hardware_concurrency(), so the viewer tree stays portable. It replaces every hardware_concurrency() call outside the tests and the vendored pocketfft. Only how many threads run changes; the passes split their work by n alone, so the results do not depend on it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 11:15:29 +02:00
leonarski_f	af3ea9ccad	Merge branch 'gpu-tail' into rc175	2026-10-08 10:22:03 +02:00
leonarski_fandClaude Opus 5.5	87f0bdc870	RotationScaleMerge: fit the error model on the device in the GPU build (tier F) ... FitErrorModelGPU puts the samples in the host's rank order with four stable radix passes, so the equal-count bins hold the same samples; computes every normalised deviation with the host's rounding, so every bin median is the host's to the bit; and hands the per-bin terms to the host's own update step, now ErrorModelUpdate (a pure extraction from FitErrorModel). The one thing that differs is the order each bin's two sums are added in: a fixed tree on the device, the order nth_element happened to leave the bin in on the host. a and b^2 can therefore differ from the host fit in their last bits - measured on two synthetic pools: relative difference 2e-16 and 3e-13 in a, 5e-16 in b^2. The device fit is deterministic (same pool, same bits). p.mtz md5 nevertheless unchanged on myob, cytc, 8a1a and 8qaw (GPU build): the difference does not survive the float rounding of the merged sigmas there. A knife-edge decision elsewhere could still see it, which is why this is kept as its own commit. The host fit took 0.4-0.6 s per fit on the large merges, about half of it the initial equal-count split, with only 16-way parallelism in the iterations. Device peak about 72 bytes per sample (the rank sort; each stage's buffers are freed when it is done); the free memory is checked first and a device without it stops with a message naming the CPU build. Kept last on the branch so that it can be dropped on its own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 08:52:04 +02:00
leonarski_fandClaude Opus 5.5	6e17ae36c8	RotationScaleMerge: the CC1/2 half-sets assigned on the device ... Every full's observation key and its half - the parity of the key's rank among its group's usable fulls - are integer work, so the device gives the host's halves exactly: one thread per full for the keys, one per group for the ranks. The host receives both (the anomalous statistics rank the same keys again within each Bijvoet cell) instead of walking every full's record for them. p.mtz md5 and p.cif unchanged on myob, cytc, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 08:51:56 +02:00
leonarski_fandClaude Opus 5.5	e59d227294	RotationScaleMerge: the ISa asymptote's group sums on the device ... The strong-reflection asymptote is read from per-group sums of I*corr, its square and the counting variance at the group mean - a walk over every full's record on the host. In the GPU build a kernel takes them, one thread per group in index order with the host's rounding (CountingVarianceExact); the host keeps the strong-group selection and the median. p.mtz md5, p.cif and the error-model lines (which print the asymptote) unchanged on myob, cytc, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 08:51:56 +02:00
leonarski_fandClaude Opus 5.5	8ed898d5a1	RotationScaleMerge: sort the outlier band's log-spread pairs on the device ... The multiplicative spread of the outlier band is the weighted median of (|ln(I/median)|, weight) pairs, one per usable full, and its sort was the bulk of that step. SortPairsGPU sorts them on the GPU (by the second key, then stably by the first) and leaves the sequence std::sort leaves: two pairs that compare equal are the same two numbers. The values themselves (a logarithm) and the running sum stay on the host. p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build); a Catch2 case checks the device order against std::sort. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:27:25 +02:00
leonarski_fandClaude Opus 5.5	1ad41c875f	RotationScaleMerge: the Bijvoet widening of the pooled outlier test measured on the device ... The per-group sums the widening is measured from - each hand's inverse-variance mean under the error model's sigmas, the group's d and the d range - are a walk over every full's record on the host. In the GPU build a kernel takes them, one thread per group in index order, with the host's model sigma to the bit (ModelSigmaExact: every product rounded on its own); the host keeps the shells and their medians. The host path is the same sums, now kept per group so both paths hand the shell step one array. p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:21:47 +02:00
leonarski_fandClaude Opus 5.5	8f605449f2	RotationScaleMergeGPU: hand the error-model samples down compacted and complete ... MergeEmSamples now forms each sample whole on the device - the counting variance at the reflection's expected intensity included, every product rounded on its own as the host build rounds it - and compacts them in fulls order with a scan, so the host receives exactly the vector it used to build. That replaces the download of three full-length double arrays and a flag array, and the host pass that compacted them while reading every full's eighty-byte record. p.mtz md5 and the error-model lines unchanged on myob, cytc, 8a1a and 8qaw (GPU build). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:14:07 +02:00
leonarski_fandClaude Opus 5.5	b6f2096015	RotationScaleMerge: the Wilson outlier test on the device in the GPU build ... WilsonOutliersGPU runs the test of WilsonOutliers.cpp step for step on the device and returns the same result to the bit: the two orders the host sorts into are total orders, so a stable radix sort of the indices reaches the same sequence; every sum is taken by one thread over its shell or reflection in the host's order; each product the host rounds on its own (-ffp-contract=off) is rounded with __dmul_rn; medians are read off sorts; and the logarithms (t, z, u, the tail scale) are taken on the host from downloaded counts and medians. MergeAndStats calls it where the merge itself is on the device. 8a1a final P61 merge, "rsm merge and statistics" 4.35 -> 2.82 s (the test runs twice there, once per error-model fit). p.mtz md5 unchanged on myob, cytc, 8a1a and 8qaw (GPU build). A Catch2 case checks the device result against the host one on a population that reaches every branch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 07:07:32 +02:00
leonarski_fandClaude Opus 5.5	729dd1706d	RotationScaleMerge: the surfaces' resolution-shell edges by selection, not a sort ... ApplyCellSurface copied and sorted every term's 1/d^2 to read 19 order statistics off it, for each of the four surfaces of the final merge. The edges are now selected: one pass counts the values under each top-16-bit pattern (a non-negative float orders as its bits), a second collects the few bins the ranks fall in, and nth_element picks the value inside each. An order statistic is a value, so the edges are the ones the sort gave. 8a1a, GPU build: the shell set-up of the four surfaces 0.55 -> 0.26 s (instrumented run), the correction-surface window 6.32 -> 6.12 s and the final merge 11.03 -> 10.82 s (two interleaved warm runs each); whole run 55.78 -> 55.45 s. 8qaw surfaces 2.95 -> 2.83 s. Tried in the same window and not kept, as they measured nothing: the serial stable counting sorts (surface group order, decay terms, fulls CSRs, delta-CC1/2 frame order) on all threads; delta-CC1/2 workers uncapped from 8; measuring two rounds of the edge search at once. The delta-CC1/2 window (2.4 s) is bound by its longest chain of ranges, one serial accumulation, not by the worker count. p.mtz byte-identical on myob, cytc, thau, kdp, 8a1a and 8qaw, GPU and CPU builds. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 06:46:28 +02:00
leonarski_fandClaude Opus 5.5	9ced4c4448	RotationScaleMerge: gather the SHELX file's scaled fulls on all threads ... The unmerged scaled fulls the final merge hands back for the .hkl were gathered by one thread pushing back over every full. Which fulls is now KeptIndices (the same list, in the same order), and each record is made side by side. The same records in the same order: p.mtz and the .hkl byte-identical (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob). 8a1a, no other load, interleaved: the P61 merge's statistics stage 3.94 -> 3.71 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 21:31:29 +02:00
leonarski_fandClaude Opus 5.5	9cc8172da5	rugnux: ingest the scaling engine's host half beside the post-refinement ... RotationScaleMerge::Ingest is now IngestHost (read the outcomes, sort, build every host array, smooth the geometry and recompute the partiality) followed by IngestDevice (allocate the GPU's arrays and stage them). The host half reads the outcomes and nothing else, as the post-refinement and the quality guard's merge do, so on the canonical pass it runs on its own thread beside both; the device half runs where the ingest used to, after the guard has handed the GPU back, so the two engines never hold the card at once. The device feed now stages the recomputed partiality in the same slices as every other field, straight from the ingest array, instead of in a pass of its own after the rest; the values on the device are the ones they were. 8a1a, no other load, interleaved runs: the first merge (P1) ends at 10.9 s after the image pass against 11.3 with only the guard overlapped and 12.7 on the rc175 base. p.mtz byte-identical on the gate sets (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a, 8tyy, CPU build myob); --model, stills, calibration, --mode scale (GPU and CUDA_VISIBLE_DEVICES="") outputs identical; --no-merge / --mode azint / --write-process-h5 _process.h5 differ only in the datasets that differ base-vs-base (azint sums, bkgEstimate, profiling times, dates, git). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 21:31:29 +02:00
leonarski_fandClaude Opus 5.5	bcfa0b11c6	RotationScaleMerge: solve the fulls-scale smoothing's cross-validation side by side ... ChooseLogScaleSmoothing picks the smoothing of the fulls' per-frame log scale by leaving out every fifth block of frames: 25 lambdas x 5 folds, each a banded solve over the whole sweep, run one after the other on every iteration of every fulls scaling loop. The 125 solves are independent, so they now run on the engine's threads; the held-out error is still added up per lambda over the folds and frames in the order the serial loop added it, from the same curves, so the lambda chosen is the same. nthreads defaults to 1 for other callers (the test). 8a1a: the six "combine and scale fulls" stages of the tail 4.7 -> 3.6 s. p.mtz byte-identical (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a); [RotationScale] and the scale_merge test files pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 21:31:29 +02:00
leonarski_fandClaude Opus 5.5	de9b2a7cd6	rugnux: merge the quality guard's pre-pass frames beside the post-refinement ... The guard's merge of the first prepass_images frames read nothing the post-refinement writes and the post-refinement reads nothing the guard writes, but the two ran one after the other at the start of the tail. The guard now merges the first frames of the outcomes in place (RotationScaleMerge::SetFrameLimit) with the per-frame scale not written back, instead of moving the reflections into a copy whose write-back was thrown away, and it runs on its own thread from just before the post-refinement; its log lines are held and replayed where the guard's result is read. Same observations in the same order, same settings: p.mtz is byte-identical (myob, thau, cytc, kdp, cytc -N 4 / -N 8, --prepass-fraction 1 on four sets, 8a1a). On 8a1a the 3.7 s guard is now inside the post-refinement, which itself goes from 4.5 to 6.0 s under the shared load: about 2 s off the tail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 21:31:29 +02:00
leonarski_fandClaude Opus 5.5	5421c569ed	rugnux: log the time of each section of the run's tail at debug level ... A TimingMark (common/Logger.h) logs "TIMING <section> <seconds>" at debug level when a section ends, so the lines go to <prefix>_log.txt and stay off the terminal. Marked: each pass of RunPipeline (pre-pass, probe, output) and its first-pass indexing, image pass, scale/merge/symmetry phase, output files and census wait; inside the scale/merge phase every merge (labelled by its role: P1, P1 space-group search, P1 all observations, the adopted group, P1 cross-check, quality guard), the post-refinement, the space-group search and its two SearchSpaceGroup arms, adoption, analyses, the P1 cross-check wait, model validation and writing; inside RotationScaleMerge the ingest and, per arm, partial scaling, combine and fulls scaling, correction surfaces and the merge; the console's steps; and the results report. tools/rugnux_timing.py tabulates the tail of a log. Only the log changes: the outputs are byte-identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 16:13:17 +02:00
leonarski_fandClaude Opus 5.5	5282dd6199	RotationScaleMerge::RunArm: free the fulls before growing them ... When a chain needs more fulls than the arrays hold, fulls and fulls_staging are released before the resize. The gather overwrites every field, so growing in place only copied the previous chain's records into the new allocation to be overwritten. The expected saving (a few tenths of a second on 8tyy) is below the run-to-run noise of the wall clock: 8tyy median of 3, 127.0 s without vs 129.6 s with, under different machine load. 8qaw max RSS 21.9 -> 21.7 GB. p.mtz md5-identical to rc175 on myoglobin, cytochrome C (also at -N 4 and -N 8), thaumatin, 8a1a, 8tyy, 9gdj and 8qaw. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:11:21 +02:00
leonarski_fandClaude Opus 5.5	c8e2e6ef56	RotationScaleMerge (tier F): decay / relative-B / radiation-damage sums over the fulls in fixed blocks ... Floating-point summation order only (tier F): the seven global sums of RefineDecay (slope fits, held-out disagreements), the per-batch sums of FitRelativeBCurve, and the per-shell and per-(batch, shell) sums of MeasureRadiationDamageB were single-threaded walks over every full. They are now FixedBlockSum: the split depends on the number of fulls alone (ReductionBlocks, ParallelBlocks), the block partials are merged in block order, so the sums have the same bits at every thread count and on every machine - but not those of the serial walk. Checked: p.mtz, p_P1.mtz, p.cif, p.hkl and the report identical to production on myob/cytc/thau (GPU) and myob (CPU), and identical between -N 4/8/default (GPU cytc, myob; CPU myob). On these sets the decay correction is not adopted (negligible or not cross-validated) and the monitor's printed values do not move, so the F-tier change is not visible in their outputs; a set where the decay correction is adopted will move in the last bits of corr. Clean branch: p.mtz md5 identical on myob/cytc/thau/8a1a (GPU), cytc -N 8, myob -N 4 (CPU). Measured (prototype with spans, 8a1a, 2 interleaved pairs, load 8-10): decay sections 2.70 -> 1.08 s, radiation-damage monitor 1.04 -> 0.86 s, RSM total 37.1 -> 35.6 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 06:30:20 +02:00
leonarski_fandClaude Opus 5.5	ea8132ceca	RotationScaleMerge: with the GPU resident, build the per-space-group grouping on the device ... Exact (tier E): the device gets the same group ids and the same group-ordered permutation (ascending observation index within a group) the host's counting sort produced. ComputeAsuGroups still finds the groups on the host (one ASU reduction per raw-hkl run, the (key, run) sort, the dense ids, the representatives - now unpacked from the packed key instead of a second ASU reduction). With the GPU resident it then hands the device only the runs in group order (two arrays as long as the runs, not the observations): BuildGroups stamps every observation's group from its run (the device evaluates the same finiteness test on the same uploaded floats), counts each group's observations, and fills each group's segment and puts it in index order. The host no longer builds or uploads the two observation-length arrays; the device keeps its per-observation group array across Runs instead of reallocating it beside the old one. SetGroups (host-built upload) is gone; the CPU path keeps the host histogram CSR. Measured (prototype, GPU RTX 5080, Run-span sections): ComputeAsuGroups summed over a run's merges 8tyy 9.0 -> 4.2 s, 8a1a 2.2 -> 0.85 s, cytc 0.51 -> 0.19 s. md5 of p.mtz identical to production on myob/cytc/thau/8a1a/8tyy and cytc -N 4. Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 06:30:20 +02:00
leonarski_fandClaude Opus 5.5	1f5c63f13c	R-free flags: none on a space-group search merge; the assignment on all threads ... Exact (tier E) for every written file. - A search merge (for_search) is never written - the run always merges again in the chosen group - and nothing reads rfree_flag except the writers and --model validation of the written merge, so RotationScaleMerge no longer assigns a free set to it. - AssignRfreeFlags takes a thread count: the per-reflection key (the lattice holohedry walk) runs on fixed chunks, and the distinct Laue-key count is taken off a parallel sort (equal keys are equal values, so every sort gives the same sequence) instead of an unordered_set. Rugnux passes its thread count at its two direct call sites; the default stays 1. Measured (prototype, GPU, 8tyy F432 cubic): the R-free/chi2 section of the merges 4.5 -> 0.65 s; 8a1a 0.76 -> 0.37 s; cytc -0.13 s. md5 of p.mtz/p_P1.mtz/p.cif/p.hkl identical to production on myob/cytc/thau/8a1a/8tyy. Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 06:30:20 +02:00
leonarski_fandClaude Opus 5.5	437f8729fe	RotationScaleMerge: MergeAndStats walks over a group CSR of the fulls, on all threads ... Exact (tier E): every per-group and per-pair sum keeps its index-order summation, so p.mtz and every written file are bit-identical; nothing depends on the thread count. - The fulls' group CSR (index order within a group) is kept per arm: on the GPU path it is the one RunArm already builds for SetFullsGroups, on the CPU path BuildFullsGroupCSR. - The four "every thread walks every full, keeps its own groups" loops (error-model sums on the CPU path, Bijvoet sums, ISa asymptote, anomalous pair sums) become ParallelBlocks over groups / pairs walking the CSR - O(n_full) instead of O(threads x n_full) traffic. - The serial walks that build per-group/per-pair data (outlier-median sets, band spread pairs, group d, anomalous-variance shells, per-hand R_meas rows, per-frame counts, the error-model sample list, the Wilson observation list) are ordered-parallel: fixed-block compaction (KeptIndices), CSR fills, per-block lists whose consumer is order-free (medians). - CC1/2 and CCanom halves are assigned cell by cell from the CSR (the rank is a set property). - Friedel pairs are numbered by first appearance through a parallel sort instead of an unordered_map of millions of keys (NumberByFirstAppearance). - CPU path: the fulls scaling loop's group means sum over the CSR on all threads (ReduceFullGroupMeans replaces the serial ReduceGroupMeans), the rescale runs on all threads. - The error-model sample download goes into uninitialised arrays (no serial zeroing). Measured (perf-rsm-explore-wip prototypes, GPU RTX 5080, box load 6-14, Run-span medians): MergeAndStats-side sections 8tyy -3.5 s, 8a1a -2.0 s, cytc -0.25 s. md5 of p.mtz identical to production rc174 on myob/cytc/thau/8a1a/8tyy (GPU), myob/cytc (CPU), and at -N 4/8 (GPU cytc, CPU myob). Clean branch rebuilt from scratch (GPU and CPU) and re-verified: p.mtz md5 identical to production on myob/cytc/thau/8a1a/8tyy/8qaw/9gdj (GPU), cytc -N 4, myob/cytc and myob -N 4 (CPU). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 06:30:20 +02:00
leonarski_fandClaude Opus 5.5	76b3956e11	French-Wilson once, on the written merge; none for the P1 cross-check ... Every full-statistics rotation merge computed French-Wilson amplitudes - the search candidates, the pinned low/high pair, the final, and the P1 cross-check - although only the written merge's amplitudes are used, and those were recomputed anyway with the anisotropic prior. RotationScaleMerge gets SetFrenchWilson(); Rugnux turns it off for its merges and makes the amplitudes once after the anisotropy analysis (anisotropic prior when a tensor was fitted, isotropic otherwise). --mode scale keeps the engine's own amplitudes. p_P1.mtz is now intensities only (owner decision): the F/SIGF/F(+)/F(-) columns are omitted; every intensity column is bit-identical to before. French-Wilson was 29% of the CPU samples in the large-cell tail (8tyy P1 cross-check window, ~84 core-s). p.mtz md5 unchanged on myob, cytc, thau, kdp, 8tyy (GPU build). Wall: 8tyy 173.9 -> 150.9 s; cytc/thau -0.2..-0.3 s, myob unchanged (3 interleaved repeats). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-06 23:10:26 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00