v1.0.0-rc.175 #85
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mx/Jungfraujoch#85
Reference in New Issue
Block a user
rugnux <master.h5> <broker URL>).--no-gpu, macOS).<prefix>_twin_<SG>.mtz, and a wrong--modelno longer confirms the data or decides the hand.--modelaccepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental--mrforces it), settling space group, hand and twinning and writing a different answer as<prefix>_mr.mtz.<prefix>_log.txt;_P1.mtz,--estimate-beam-centerand--beam-center-searchare removed.rugnux <file>, and shows the results of processed CBF, marCCD and SMV sweeps.The space-group search was almost entirely single-threaded. The per-reflection work now runs in parallel over fixed chunks, while every floating-point accumulation stays serial and in the same order as before, so the output is bit-identical. - AnalyzeLTestUnderMerge takes nthreads (default 1): the usable test, the orbit means and the partner search run in parallel; the 20 resolution-shell passes become one walk. - Stage B ClassifyReflection: one byte per reflection, candidates scored in batches of nthreads. - FindRowModulation: cone test and cosines in parallel. - PrepareMerge: parallel flatten; the sorts use inline {d, index} keys with the same comparison. Measured: SearchSpaceGroup 14.7 -> 4.8 s on 8tyy (instrumented build). Wall clock, median of 3 interleaved runs vs rc175, timed inside the GPU lock (-march=x86-64-v3, CUDA): 8tyy 136.9 -> 127.0 s, 8a1a 81.8 -> 80.4 s, cytochrome C 18.8 -> 18.9 s (unchanged). 8qaw max RSS 21.7 -> 21.9 GB. p.mtz md5-identical to rc175 on myoglobin, cytochrome C, thaumatin, 8a1a, 8tyy, 9gdj and 8qaw (GPU build), and on myoglobin and cytochrome C in a CUDA-off build. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiA pure move: HotPixels.{h,cpp} and HotPixelsGPU.{h,cu} are compiled into JFJochImageAnalysis (like beam_stop/) instead of JFJochRugnux. Only include paths and the CMake source lists change; the objects are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiThe final master marks a finished collection for anyone watching the directory (rsync, autoprocessing, a reader following the collection), so it must not appear before a data file it links to. - FileWriter: at END every data file is closed (renamed into place) before the master is finalized and renamed. A data file that fails to close does not cost the master; the failure is reported after it. For NXmxIntegrated, file 0 is the master's own image dataset, so there a failure still stops the master. - With several writers, the master was written by the writer of the first socket as soon as it got END, while the others could still be closing their data files. END now goes to every other writer first: - TCP: the connections are taken in reverse order, and each END waits for its acknowledgement, which a writer sends only after closing its files; - ZeroMQ, with a writer notification socket: END to every other socket, then their notifications (the same ones Finalize collected before, now collected here), then END to the first socket. Without a notification socket there is nothing to wait on and the order is all it changes. Tests: TCPImageCommTest_MasterWriterGetsEndLast (three mock writers, the master's END after the other two acknowledged), StreamWriterTest_ZMQ_MasterAfterAllDataFiles (two real writers, the second started only after the collection was sent; the master's ctime is after every data file's). Both fail without this change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiSo that rugnux can process a collection while it is written, the broker reports each data file once a writer has closed it and renamed it into place: - Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the images the file holds. TCP protocol version 5. Files closed at END are reported before the END acknowledgement. - LiveCollection (image_pusher): the start message the writer of the master file received, the closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher has no back channel and offers none. - Two routes, protected by the dataset's bearer tokens like the other dataset routes: - GET /live/start.cbor: that start message; - GET /live/events: server-sent events (collection, file, end, superseded) with the history first and a keepalive every 10 s. One follower at a time; a second gets 409. Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged. - The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for the whole collection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiModelStructureFactorsGPU computes what compute_model_factors() and map_from_coefficients() compute on the CPU - F_calc from the model's density (IT92, Refmac-compatible blur, unblurred as prepare_asu_data() does) and F_mask from the Refmac bulk-solvent mask, both on the reflections prepare_asu_data(d_min) lists, in its order; and a map from ASU coefficients on the grid get_size_for_hkl(coef, 0, 3.0) sizes - on a device. Made once per cell, group, resolution and model, then evaluated as often as the coordinates change, so refinement or MR can call it in a loop. The device is an explicit parameter; every call leaves the calling thread's current device as it found it. Pieces: - ModelDensityGPU: the rigid body's deterministic brick gather, moved out of RigidBodyGPU.cu into a component of its own (ModelMaskGPU's pattern); the rigid body uses it unchanged. MAX_BRICKS_PER_AXIS 8 -> 16, so fine grids with high-B atoms (lysozyme at 1.2 A, a 0.9 A P1 cell) are no longer refused; existing zones are gridded identically. - One copy of the content is gridded and the symmetry composed in reciprocal space (SymmetryComposition), operators applied on the fly; the mask is ModelMaskGPU (every image of every atom, islands, shrink). - Maps: gemmi's get_f_phi_on_grid() in ZYX order on the host (the coefficients written are the same), in-place cuFFT c2r, transposed back to XYZ on the device. One map at a time, in the engine's buffers. Decided once, up front, per card, from its TOTAL memory: the engine's bytes (16 N + cuFFT work + reflections, N the larger of the structure- factor and map grids) must be at most half the card - the rigid body's engines take at most a quarter beside it. Otherwise, or where the gather cannot grid the cell, the CPU path runs, logged with needed vs total. Anything to a resolution other than d_min (the null's 3.5 A fits) stays on the CPU, so all replicates and the real model's side of the null are computed the same way. A CUDA failure takes the existing path: the validation restarts on the CPU. Measured, model validation total per run (CPU path -> GPU), 16 GB card: F432 215 A cubic, 1.30 A, 500^3 grid: 47.7 -> 15.6 s (two validations; 14.3 -> 2.4 and 33.4 -> 13.2, the rest of the second is writing the three 0.5 GB maps); F_calc + F_mask 5.7 s -> 0.05 s C2 1.11 A: 23.9 -> 13.6 s; P3_2 1.55 A: 18.2 -> 10.2 s; P2_1 1.25 A: 13.4 -> 6.5 s; F4_132 328 A: 13.0 -> 5.5 s; P6_5: 8.8 -> 4.2 s; P4_3 0.97 A: 4.6 -> 2.5 s; P1 0.92 A: 4.2 -> 2.2 s; small P1: 3.2 -> 1.3 s; P6_1: 8.8 -> 6.2 s; lysozyme: 1.8 -> 1.4 s. p.mtz md5-identical on all 13 sets. Against the CPU path: FC within 1e-4 of mean |F|, phases of the strong half within 0.003 deg, maps within 1e-4 (2mFo-DFc) and 7e-4 (mFo-DFc) of their rms; every logged R, CC, FOM, k_sol and anomalous site list identical at the printed precision, except where a rigid-body commit sat on an exact R-free tie (0.2155 -> 0.2155) and fell the other way (R-work 0.2127 vs 0.2129). The GPU result is bit-identical run to run and with -N 8 (maps, map MTZ, placed model). Peak device memory of the engine: 2.5 GB at 500^3 (process total peaked at 14.4 GB with what the merge still holds). Tests: ModelStructureFactorsGPU_MatchesCPU (five groups, 3.5 and 1.5 A: same reflections, F_calc <= 1e-5 of mean |F|, F_mask 2e-7 rms, repeat bit-identical), ModelStructureFactorsGPU_MapMatchesCPU (<= 5e-6 of rms). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiA CUDA error on the validation's GPU path (rigid body, structure factors, maps) used to start the whole validation again on the CPU - a migration part way through, which makes how long a run takes, and in principle what its rigid bodies converge to, depend on a device fault. It now ends the validation with ok = false and a logged reason ("the GPU failed during the validation (...)"), which the run reports as MODEL_NOT_VALIDATED. A validation that did not finish decides nothing, so the reflection files are those of a run without a model. Where there is no GPU at all, or the up-front rule sends the work to the CPU, nothing changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiThe image-loop workers are kept on the CPUs of their card's NUMA node (pin_gpu), but the two threads that drive the merging tail were not: the quality guard took its card with set_gpu(1), and the main engine runs on the pipeline's own thread, which goes on to everything else. On a multi-socket server their pageable copies to the card went through a remote socket. RotationScaleMerge now holds a ScopedGpuPin (new, CUDAWrapper) for each IngestHost, IngestDevice and Run: pin_gpu(gpu_device) for the call, and afterwards the thread's previous device and CPU mask back (GetThreadCpus/SetThreadCpus, new in ThreadAffinity). So the binding covers whichever thread drives an engine, and does not stay on the pipeline thread. The threads the engine starts itself (ComputeAsuGroups, Combine, FitErrorModel, ApplyFrenchWilson use std::async) call RestoreThreadAffinity first, as the pool's workers do, so they are not squeezed onto one node. The guard thread, which does nothing but the guard's merge, uses pin_gpu(1) instead of set_gpu(1). Pool workers are untouched: they restore the start mask themselves; the new test checks both that and that a pinned caller keeps its own mask. No flag, no environment variable. On one NUMA node (this workstation) only the device setting happens, so nothing can move. Measured on the 5080 workstation (one card, one node), base = rc175-next 41fab6156, --model dropped, under the GPU lock: p.mtz md5 identical on myob_x10sa, cytc_x10sa, thau_x10sa_16keV, 8a1a, 11if, 7qis, 6h5t, kdp_x10sa_20keV at default -N (2 runs), -N 8 and CUDA_VISIBLE_DEVICES=0. Tail ("scale, merge and symmetry") unchanged within noise, e.g. 8a1a 30.06 -> 29.37 s (base runs 29.75/30.36), cytc 5.33 -> 5.35 s. The gain is for multi-socket boxes (T4 server: guard copies 1.7-2.1 vs 3.5-4.3 GB/s) and is not measured here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiEach rugnux / reference pair is now rugnux right-aligned to a fixed width, "/", the reference left-aligned to the same width ("-" for a missing value), so every slash of a column sits on one line; the tick or cross follows the space-group pair in its own column. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiSupercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice reflection predicted on the same frame lies within r3, and each resolution shell is read against the median of the six other parity classes - which a single doubling leaves empty - as an excess in units of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z beside it ("Supercell probe to the limit" in the log). Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its noise and at more than twice the strongest of them, in at least three shells), the run integrates and merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both (DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT (the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the class reaches a quarter of the lattice's intensity in the finest shell both are measured in). The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed. Measured on forced doubled-cell merges (rc175b530c2dbase): the open-arm crystal deposited in the doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell (odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at 8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the neighbour correlation, the separation from the nearest lattice reflection after the r3 filter - separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call. Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no longer advises -C with the doubled cell, which the indexer reduces back to the half cell. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNiThe model scale searched b_sol only over 10-80 A^2. That is the range Fokine & Urzhumtsev report for models at ordinary B, but on low-resolution entries whose deposited B is 200-400 A^2 the fit sat on the 80 A^2 ceiling (and k_sol on its 0.1 floor): a solvent edge far sharper than the atoms it lies against, reaching too deep in resolution. R-free came out 0.05-0.10 above REFMAC (0 cycles) and phenix.model_vs_data on the same MTZ, model and flags. The ceiling is now max(80, 2 x the model's mean atomic B) (SolventBMax), for the reported fit and for the rigid-body placement's per-zone solvent (CPU, GPU and the host fallback alike). The grid keeps eight b_sol values across a wider box, so it costs no more points; the default box, and so every model with mean B <= 40 A^2, is bit-identical to before. Measured (10 open-arm sets with --model, GPU build, vs the same base): high-B set, 7.0 A: R-free 0.462 -> 0.364, b_sol 80 -> 379 (REFMAC 0.366, REFMAC's own b_sol ~371; model_vs_data 0.291); the rigid-body step now commits high-B set, 3.9 A: 0.410 -> 0.331, b_sol 80 -> 388, k_sol 0.10 -> 0.225 (REFMAC 0.328, model_vs_data 0.314); null 10.1 -> 14.2 sigma B~200 set, 3.2 A: 0.408 -> 0.402, b_sol 80 -> 120; enantiomorph label decision unchanged (27.2 -> 25.8 sigma) 7 other sets (mean B 16-57): bit-identical R, k_sol, b_sol, placement. p.hkl md5 and the MTZ reflection data are identical to the base on all 10. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>gemmi's table lists some groups a second time on another origin ("P 42 21 2a", "P 21212(a)", "I 2 2 2a", "C 1 21 1"). SameRotationGroups kept them as separate candidates, so the true group tied with itself in R-free and the choice came out UNDECIDED: five single-copy sets of the --af-model survey (and one partial-model set) ended there. A group equal to one already listed up to an origin shift on the twelfths is now left out. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>f3a405655into rc175: near-tie re-checks with the same first-pass inputs are not run twice d42d89b5f5The CC(model, data) table's rows ('# 4.13 ...') were read as section titles, so every key after the table in 5. MODEL VALIDATION counted as a decision outside it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3A UINT16_WIDENED frame (marCCD) was page-locked where it lay with PinInputRegion, which keeps the registration until the engine moves on to another frame. The pre-scan hands its kept frames to the spot measurement and to the defective-pixel search at once, each with its own engines, so one engine could unregister a frame while another engine's copy from it was still in flight. That takes the context down ("unspecified launch failure"), seen as a rare crash at the end of the pre-scan on a 3072x3072 marCCD sweep (1 run in 30 reproduced it). The stored bytes now go through the engine's own page-locked staging buffer, as the CBF and BSLZ4 decoders already do. The pixels decoded are unchanged. Test: ImagePreprocessorGPU_SharedStoredFramesManyEngines - four engines upload four shared marCCD frames concurrently; it aborted with the launch failure before the change and passes after it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3f853596a45135926ea597507df6e9b891ebe7d5964e75Changes results - not for merging without the owner's approval. The per-image min-pix scheme indexed every frame at min-pix 3, 2 and 1, each with the 30/80/all seed escalation, then indexed the winner again: 3.9 indexer calls per frame. Measured per frame on a serial lysozyme run (2000 frames, de novo and against the known cell): - where several lists index, the searches return the same lattice (97-98% of pairs); - the gain over a fixed min-pix 2 has two sources: (A) frames min-pix 2 indexes but whose min-pix 1/3 list indexes ~26 more spots (a better per-frame refinement), and (B) frames with almost no spot of two pixels (min-pix 3/2 lists of 1-3 spots, ~50 single pixels at min-pix 1) on which the min-pix 1 search fits 8-9 spots. De novo, 189 of 194 such B lattices had cells unrelated to the crystal's; against the known cell they index, but leaving them out raises I/sigma and lowers R_pim. Now: the lattice is searched for on the min-pix 2 list (min-pix 3 where that does not index); the first list that indexes hands its indexer answer to the others, which are only refined and scored (FrameIndexerCalls::lattice_found). The min-pix 1 list is searched on its own only where the min-pix 2 list holds at least ViableCellMinSpots spots. Ties keep the larger min-pix, as before. Measured on two full 24960-frame serial runs (RTX 5080, box loaded by other agents, load 9-20; loop rate is frames/s of the image loop; I/sig, R_pim, observations of the P41212 merge): known cell (ffbidx, -C): base 785 / 750 -> 1130 / 1017 frames/s (warm 6000-frame ABAB 860 -> 1453) run A: obs 5.37M -> 5.30M, I/sig 13.09 -> 13.02, R_pim 6.48 -> 6.46%, CC1/2 .970 -> .973 run B: obs 4.91M -> 4.89M, I/sig 12.00 -> 11.95, R_pim 6.50 -> 6.47%, CC1/2 .971 -> .970 de novo (FFT): base 89 / 91 -> 174 / 195 frames/s run A: obs 3.96M -> 3.91M, I/sig 10.18 -> 10.12, R_pim 6.94 -> 6.98%, CC1/2 .960 -> .967 run B: obs 3.91M -> 3.89M, I/sig 10.53 -> 10.52, R_pim 7.01 -> 7.03%, CC1/2 .965 -> .961 fixed min-pix 2, for scale: ffbidx 1496 / 1316 frames/s, obs 4.94M / 4.54M, I/sig 12.62 / 11.77. The indexing rate drops (ffbidx 59.5 -> 56.9%, 55.2 -> 52.4%; FFT 60.5 -> 52.1%, 60.2 -> 47.9%): the frames that go are the spurious B lattices; observations stay within 1.3%. Indexer calls per frame 3.89 -> 1.67 (de novo, 2000 frames); the min-pix 1 search is skipped on 58% of frames. De novo, 23 of 691 frames whose lattice was the crystal's now take a superlattice (13, same spots indexed) or a cell fitted to 6-12 spots of a weak frame (10) from the min-pix 2 search. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>Changes results - not for merging without the owner's approval. The 30/80/all seed escalation exists for the de-novo search, which a flooded frame's full spot list derails. ffbidx fits a known cell with a trimmed score that already discounts off-lattice spots, so for it the escalation only multiplied calls: up to three per searched list, ~2.2 per indexed frame with a fixed min-pix 2. The ffbidx path is GPU-bound (nsys: card 98.5% busy, ffbidx kernels 86%), so calls are the loop rate. Stills only: rotation never resolves to ffbidx. Measured on top of the one-search min-pix scheme (parent), two full 24960-frame serial runs, P41212 merge (box loaded by other agents, load 13-17): run A: 1130 -> 1532 frames/s; obs 5.30M -> 5.47M; I/sig 13.02 -> 13.12; R_pim 6.46 -> 6.43%; CC1/2 .973 -> .976; indexing rate 56.9 -> 57.3% (original scheme: 785 frames/s, 5.37M, 13.09, 6.48%, .970) run B: 1017 -> 1488 frames/s; obs 4.89M -> 4.99M; I/sig 11.95 -> 11.94; R_pim 6.47 -> 6.42%; CC1/2 .970 -> .972; indexing rate 52.4 -> 52.9% (original: 750, 4.91M, 12.00, 6.50%, .971) Warm 6000-frame ABAB in one lock hold: 1453/1419 -> 1734/1810 frames/s, same indexing rate. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>On a 16 GB card a probe pass (GPUWorkBeside) runs its image work beside the main line's image pass (GPUImagePass). Measured on a 16 Mpx detector with nsys, the two together held 8.6 GB of device buffers (two sets of 24 per-worker 67 MB image buffers plus five 0.6 GB FFT indexers), so on an 8 GB card the refined-geometry pass ran out of memory ("Allocating GPU memory (53.0 MB) failed"). Below 12 GB of card memory each of the two now waits, before it starts, for the other kind to end. Only when each runs depends on the card, never what it computes; neither waits once running, so nothing waits in a circle. On a 16 GB card nothing changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>86f17740finto stillsmerge c15ee18782Every TimingMark ("TIMING <section>") and every rugnux step ("TIMING step ...") now opens an NVTX v3 range of the same name, so Nsight Systems traces from different machines can be lined up by pipeline step. Start/end ranges, not push/pop, so a range may end on another thread and parallel arms may overlap. Header-only NVTX (CUDA::nvtx3), a no-op without a profiler attached and compiled out without CUDA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>On an 8.55 GB card (WDDM) the validation planned 6.81 GB of a 6.84 GB budget, but the card already held more than the 20% headroom allowed, and the driver paged GPU memory to system RAM: every rigid-body fit ran ~5x slower. Measured on 4nwv on a 16 GB card, at the start of model validation the device held 1.1 GB beyond a fresh context, and on 6oel 2.7 GB: - the stream-ordered allocation pool's free-but-reserved memory, left by the image passes and the scaling's pooled buffers (1.07 GB on 4nwv, 2.1 GB on 6oel); the validation's engines are allocated outside the pool and cannot reuse it; - the image pass's FFT indexers (IndexerThreadPool), alive for the whole run (0.8 GB on 4nwv, 0.5 GB on 6oel), although nothing indexes after the merge. Both are now released: the validation trims the pool (cuda_trim_allocation_pool) before it plans, and rugnux drops the indexer pool once the leftover-lattice census, its last user, has ended, before the model is validated or searched for. What is left beside the validation is 0.06 GB over a fresh context on both sets (streams, idle cuFFT plans). Extra engines are now added only within half of the budget. They change no number and bought little where measured (4nwv, 16 GB card: 13.6 s with one rigid-body engine, 13.1 s with two, 12.6 s with five), so the card keeps headroom instead. On 16 GB 4nwv now plans 2 rigid-body and 2 scale-fit engines, 6.56 GB (measured peak 7.3 GB with the context); on an 8.55 GB card one of each, 4.42 GB. Reflection files, maps and models byte-identical to rc175 HEAD on 4nwv, 8tyy, 6oel and 3inp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>bd27d802b) into rc175: CPU fulls scaling loop on a working copy, combine prefetches, output unchanged 9d8b06525fbd7464d70) into gpumem-run 8f722227e8011b98838) into rc175: molecular replacement plans its GPU parts once per merge on model validation's budget (one rigid-body pool for every fit, the transform cropped to the indices it is read at, the pool trimmed at each stage), output unchanged 323df1561e51706641f) into rc175: RESULT-CHANGING (owner approved, float rounding): molecular replacement's transform FFT on the GPU wherever it fits the search's budget d91fc7ca3d2881e6255) into rc175: CPU translation function builds each grid point's entries once per setup and sums straight into FFTW's input, output unchanged e185aedd988d17d0522) into rc175: RESULT-CHANGING (owner approved, float rounding): molecular replacement's CPU transform FFT on all threads, the result independent of the thread count 873880bba1c72e4e97c) into rc175: the viewer keeps "Known unit cell" unticked when a run's results are shown aa4d56d8a896388a389) into rc175: CPU spot finder's vertical window slide vectorises, output unchanged 16911a44bbc207012a1) into rc175: RESULT-CHANGING (owner approved): the beam-stop finder's rings follow 2-theta on a tilted detector, with the solid angle along a ring divided out; an untilted detector's mask is unchanged 2810b31994The owner's generator and its SVG, under docs/images/, shown on RUGNUX_OVERVIEW.md. Corrections against the code, with the owner's header and style kept: - Input: marCCD added; "stream" is not a ZeroMQ stream but the broker's live collection (closed data files read as they are reported, rotation only). - Pre-scan: also masks defective pixels; it MEASURES the beam centre and commits none - the first pass indexes at both centres - so the centre now goes to Index as its own arrow ("measured beam centre"), and the arrow into the per-image panel carries "masks, spot widths". - Index: a first pass on a frame sample (validation frames, rotated into one crystal frame), not a per-image indexer. - Refine: the per-image fit is the in-tree LM solver adapted from Ceres (LMSolver.h), not Ceres Solver itself. - Input arrow: pass 1 integrates only the first half of the sweep (>= 120 deg). - Post-refine: beam centre added (distance, beam, cell, axis; tilt is held); the hypotheses run between the passes (beam-centre arm, metric-symmetry arm, geometry walk, rotation-scale walk) noted; pass 2 is repeated while a round realises a gain. - Whole dataset: the P1 merge is where competing passes are judged (quality guard, beam-centre arms), so it carries "tested"; Symmetry adds the doubled-cell hypothesis; the in-group merge names corrections and error model; structure is drawn dashed with a bypass to the report (no model); report lists unmerged MTZ and the report file. - Legend: a CUDA step takes its CPU path without a GPU. - Taller boxes where a second text line needed room; the SVG is written next to the script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4Owner's review of the rotation figure: - The in-group merge names the correction surfaces it fits (decay, absorption, modulation; RotationScaleMerge) instead of the word "corrections". - New "autodiff" badge, on the fits whose derivatives are automatic: Refine and the rotation indexer's lattice refinement (XtalOptimizer/XtalRefine, forward- mode Dual numbers), Post-refine (ceres::AutoDiffCostFunction in PostRefine.cpp) and the stills geometry bundle adjustment (GeometryRefiner). Not on the rigid body (analytic plus forward differences), the stills partiality refinement (central differences), the beam-centre fit or the stills indexer's candidate refinement. The Ceres text on Refine is gone. - Text back to about the owner's budget and box sizes: the meaning moved to arrows ("pass 2, or pass 1 if it merges better" into the P1 merge, "beam centre" from the pre-scan to Index) and badges; "rival ..." phrases removed. New: the stills figure (rugnux_pipeline_stills.svg) from the same script, same style and legend. Stills = no goniometer axis or a stationary one (RugnuxRotationData). Geometry bundle adjustment first, only with a known cell (dashed); pre-scan masks the beam stop and bad pixels; per image: decode, profile, spots, index (1D FFT, or ffbidx with a cell; min-pix chosen per image by what indexes: tested), refine beam/orientation/cell, excitation-error prediction, profile fit; whole dataset: consensus cell and per-image scale (CPU), symmetry, merge with partiality post-refinement and error model, structure, report. Both figures are on RUGNUX_OVERVIEW.md, the stills one under "Stills instead". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PXc1zD1yxHqaR1DGdCTHL4e25876055(=3be5dcaac) 1508620246e25876055(=3be5dcaac) beside the kept workstation run 622b691f693be5dcaac- the timing benchmark on six platforms