14 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Fable 5.1	546566cff4	Refinement: the gather's pairs built in fixed-shape buffers, and each evaluation replayed as a CUDA graph ... ModelDensityGPU::Compute() no longer reads the pair count back to the host between its launches: the atoms are counted per brick with integer atomics, the counts scanned into the bricks' segments, the segments filled through atomics and then each brick put into model order by rank (one block per brick), which leaves the pairs exactly as the stable radix sort of the atom-major fill left them - the gather sums in the same order, bit for bit. The pair bound of the atoms set is checked on the host at SetAtoms() instead of on the device at every evaluation, and the buffers shrink from four to three ints per pair. With nothing coming back mid-evaluation, RefinementGPUEngine records the X-ray evaluation and the restraint evaluation each as a CUDA graph on its stream (cudaStreamBeginCapture, thread-local mode) and replays it, re-recording only where the shape changes (the atom count, a gradient asked or not, a restraint set re-uploaded). Two graphs, not one, so that the host still fills the restraints' positions while the X-ray half runs, as before. Where a stream cannot be captured the launches are queued as they were. Outputs byte-identical to 037c32d9f on the in-house cytochrome C live run (MTZ arrays, HKL, models, blobs, report). Workstation (RTX 5080): refinement 1.64 -> 1.50 s on that run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-27 05:57:57 +01:00
leonarski_fandClaude Fable 5.1	d122118d73	Refinement: the per-evaluation host chain on the threads and in place ... Byte-identical: every pass is element by element in the serial loop's order, or a copy that is no longer made. - The engine's gradients are read in place from its pinned buffers and widened to double on the threads (RefinementGPUEngine::HostGradient/HostRestraintGradient); the positions go up from a pinned vector of the engine's host side; the model's atom pointers are kept between calls (the model is the context's own). - Context: the riders moved and folded on the threads - a parent takes its riders in index order through a CSR list (SetRiderParents), as the serial walk added them. - AtomTarget: the parameters set and the gradient assembled on the threads; the per-atom gradient vectors kept between evaluations (7 MB each on a large model: made anew, their pages fault in every time). - RefinementMinimiser: the sweeps over the parameters (x + alpha d, s, y) on the threads it is given, the dot products the serial chains they were; Direction()'s working vector and a let-go pair's vectors reused. A/B against the frozen combined2 binary, one lock hold each, outputs identical but provenance lines: cytc refinement 1.600 -> 1.548 s (live), 1.724 -> 1.668 (CLI); thau 1.906 -> 1.885, 2.031 -> 2.027; 4nwv (with the commit before) 39.9 -> 35.5 s, the ladder 14.6 -> 12.7, each step 8.3 -> 7.2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:48:28 +01:00
leonarski_fandClaude Fable 5.1	b16873036c	Refinement: density records sent from pinned memory, and the L-BFGS direction in fused sweeps ... Two byte-identical cuts to the per-evaluation host chain, which on a large model is twice the GPU's work per evaluation (34 ms against 16 ms of kernels per weight-trial slot, measured in nsys) and so bounds the refinement on one card with three slots, and the whole of it on a card with one slot. - The gather's per-atom density records (38 MB on a 242k-atom model, sent on every B-factor evaluation) live in a PinnedVector and go up by DMA; the boxes go through a pinned staging buffer of the density object. ModelDensityGPU::SetAtoms and RefinementGPUEngine::SetDensities take a pointer and a count. - RefinementMinimiser::Direction takes each pair's dot product in the same sweep that applies the previous pair's update to q: the dot products are serial chains of n additions, and the update's memory traffic now hides under them (20 -> 13 ms per call at 365k parameters). The same operations per element in the same order; the loop-2 coefficient stays written as (a[i] - b) with b on its own, because a[i] - rho * dot contracts into a fused multiply-add and changes the bits (checked bitwise in a standalone harness). - A debug line with the GPU engine's bytes and the budget. 4nwv (open arm, 242k atoms), live flags, one lock hold, loaded box: refinement 42.35 -> 38.40 s; refine.gpu.upload 8.14 -> 3.94 s. All outputs identical but provenance lines. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:48:28 +01:00
leonarski_fandClaude Fable 5.1	3189f71e8f	Refinement: the engine polls for its evaluations and the pair count comes back pinned; the slots beyond one per card go to the cards the ladder's long trials leave free ... The GPU engine's waits on its stream (every evaluation, the refits' transforms) and the model density's wait for the (brick, atom) pair count were blocking event and stream synchronisations; where a weight trial has a card to itself, as on a box with a card per trial, the wake-up of the sleeping thread is on the critical path of every evaluation. Both now poll, and the pair count and overflow flag come back into pinned memory instead of a pageable int (a pageable copy back is staged and blocks the host for the whole stream). OwnSlotDevices places the slots beyond one per card on the cards the ladder's long trials (those refining B) leave free, so that on four cards the three B rungs have a card each and the three B-held rungs share the fourth, instead of two cards each carrying a B rung beside a B-held one. The outcome does not depend on the placement. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 22:41:43 +01:00
leonarski_fandClaude Fable 5.1	128e3bb9ae	RefinementGPU: the symmetry phases from a table instead of sincospi in the compose and scatter kernels ... exp(2 pi i h.t) depends only on the integer h.t (in units of 1/den), so the engine tabulates it once over the range the rows and operators make, with the same op_phase() at the same double argument - the same bits. The compose kernel reads the table at h.t; the scatter terms carry their table entry, and their row directly, so neither hot kernel divides: the rotations go to the device in whole units (gemmi's are den times an integer matrix), the term's row no longer has to be split from its operator. The table is a few thousand entries, the scatter terms one int wider. Outputs are byte-identical; the X-ray evaluation counter drops 7-10% on the demo-sized sets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 20:58:10 +01:00
leonarski_f	257bf2fe2b	Refinement: trial slots counted with their scale pool and both restraint sets; on the CPU no more than -N ... GPU: a slot was packed onto a card by its engine's bytes alone. The per-slot estimate now adds the slot's ModelScaleGPUPool with the robust fit's buffers (ModelScaleGPUPool::DeviceBytes(n, 1, true)), and RefinementGPUEngine::DeviceBytes counts the two restraint sets SetRestraints() holds, not one. The 80%-of-card budget is unchanged, so where these do not fit fewer slots are packed instead of an allocation failing inside a running trial. CPU: OwnSlotDevices made a slot per wanted trial whatever the thread count, so -N 1 ran six trial threads with six models, topologies and engines in memory. The main slot and those beside it are now at most -N. Side by side equals sequential, so no result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3c04c01158266c267f8e5ffaaaa98d6d8fd2a747)	2026-10-26 15:38:58 +01:00
leonarski_f	13da79b788	Refinement GPU: the atom gradient kernel walks its box without integer division ... Bitwise. One warp per atom, each lane taking the box's points t = lane, lane + 32, ...: every point cost three integer divisions/modulos to turn t into (u, v, w) and three more (imod) to wrap them into the grid. The lane now steps (iu, iv, iw) on by 32 points with adds and carries, and wraps from the box's first point, wrapped once per atom (a box is narrower than the cell, so one conditional subtraction per axis). The same points in the same order per lane, the same arithmetic. atom_gradient_kernel (nsys, side by side): 5reo 426 -> 237 us, 5f6m 887 -> 523, 9yl4 1436 -> 802, 7d98 692 -> 388 us per launch. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Wall: 5reo 1.43 -> 1.35, 5f6m 2.20 -> 2.00, 9yl4 6.88 -> 6.82, 7d98 3.58 -> 3.38, 5f6m MD start 4.16 -> 3.71 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6554e9dcbe4b448eda582eb76d6a24c59fe6ab0c)	2026-10-26 09:16:34 +01:00
leonarski_f	4a8e13ddfa	Refinement GPU engine: its waits on the stream asleep (blocking-sync event) rather than spinning ... Bitwise. cudaStreamSynchronize spins a core for the whole wait; the weight trials' engines wait on a shared card for most of a run, so about half of the user cycles of a refinement were the spin (PROFILE.md: 38-50%). A StreamWaiter (CUDAMemHelpers.h: an event made with cudaEventBlockingSync, recorded and waited for) now takes the refinement engine's waits - evaluation, Fcalc, Fmask, the uploads of positions, row model, blur and restraints. The short waits of the host-driven loops stay spinning: the scale fit's Levenberg-Marquardt (one launch of 66-120 us per step) and the gather's pair count; making those block too cost 2-3% of wall (wake-up latency), here the wall is unchanged. No process-wide cudaDeviceScheduleBlockingSync: that would change every GPU wait of the broker. CPU use (-N 8, time -v): 5reo 301-315% -> 190%, 5f6m 326% -> 181%, 9yl4 333% -> 222-228%, 7d98 307% -> 181%, 5f6m MD start 309-320% -> 151%. Wall, two interleaved runs each: 5reo 1.43-1.52 -> 1.43, 5f6m 2.19-2.20 -> 2.19-2.20, 9yl4 6.83-6.84 -> 6.88-6.89, 7d98 3.50-3.51 -> 3.58, 5f6m MD start 4.16-4.34 -> 4.16 s. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d23d2cf7bb99b30617a3f57c531b03db8bc03e05)	2026-10-26 09:16:34 +01:00
leonarski_f	bda90dbdd5	Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms ... - B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)	2026-10-26 01:38:24 +01:00
leonarski_f	8718065a7c	Refinement GPU: the bulk-solvent mask on its own fine grid on the card ... The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10)	2026-10-26 01:38:24 +01:00
leonarski_f	5569f3d1f2	Refinement GPU: the X-ray target and the restraints in one submission with one wait ... RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80)	2026-10-26 01:38:23 +01:00
leonarski_f	e3782cabe4	Refinement GPU: atom densities uploaded only when some B moved, one device copy, positions through a pinned buffer ... Every evaluation uploaded both copies of the atoms' densities (the gather's and the gradient's) and their boxes, although in a positions-only minimisation no B changes between two Refits. The engine now keeps one copy (the gradient kernel reads ModelDensityGPU's atoms and boxes), uploads it only where some atom's density was made again, and sends the positions from a pinned buffer without waiting. The densities that are made again (every atom in a B minimisation) and the grid positions are worked out on the refinement's threads. Same bits. 9yl4: upload 3.29 -> 0.56 s, host side 1.29 -> 0.61 s, refinement 22.3 -> 17.7 s; 5reo 4.8 -> 4.4 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 46c3b67c91aee0aec3473b0fb1a4945c00bda070) (cherry picked from commit 0aebc98110c8155932af647d6204891aec662294)	2026-10-26 01:38:22 +01:00
leonarski_fandClaude Opus 5.5	f077320efa	Refinement restraints: n_slots, not slots - Qt defines slots as a macro, which broke the viewer's build through Rugnux.h ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 10:22:02 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00