3d382ce0a09c24e0021e0c8288d8a9d08eb03e6b
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 leonarski_f
|
3d382ce0a0 |
Refinement: an exception in a trial slot or in RefineModel no longer ends the run
A slot builder thread that throws (gemmi's topology or hydrogen placement, bad_alloc) would call std::terminate - rugnux or the broker gone. The slot is now left out as one the card cannot take is, with a warning, and MakeOwnSlot's GPU engine catches std::exception, not only JFJochException. Both callers of RefineModel (after the merge, and --mode refine) catch std::exception: the refinement is reported NOT_PERFORMED with "refinement failed: <what>" and the run writes everything else (cell e.s.d.s, unmerged MTZ, report), as the refinement is processing-inert. Test: --mode refine on a cell too large for the bulk-solvent grid comes back with REFINE= NOT_PERFORMED and the reason, not an exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc4e99fe4b3da2a9a3fa4290b5f70bfcc8c7bdb8) |
||
|
leonarski_f
|
e406eb17ea |
docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits
- New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10) |
||
|
leonarski_f
|
f34d2f4cca |
Refinement: the weight trials' own slots made while the placement runs
Bitwise. MakeOwnSlot copies the model as it was before the topology and reads only what the placement does not change (cell, reflections, the engine's kind, the B ceiling), so the slots are now built on their threads right after the start score, beside the rigid-body and group-B placement, and joined where they were built before (also where the placement throws). The "slots" stage goes from 0.23 s to 0 on 9yl4; the placement is not slowed (0.59 -> 0.60 s). Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU); side by side = one at a time. Wall: 5reo 1.46-1.53 -> 1.54, 5f6m 2.26 -> 2.21, 9yl4 7.03 -> 6.83, 7d98 3.65 -> 3.56, 5f6m MD start 4.20 -> 4.15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0752a921a4293d4213894e3242c4ef36d8ce2a05) |
||
|
leonarski_f
|
1f8b5b1986 |
Refinement: a Refit's solvent grid starts around where the context's last one ended
Logic change (deterministic), kept on evidence. Every Refit searched the whole k_sol/b_sol box - an 88-point coarse pass of plain least squares, then the robust fine pass around its winner. ModelScaleBox (and ModelScaleGPUPool::Fit / ModelScaleGPU::FitSolvent) take an optional start; with one, the coarse pass is skipped and the fine pass is laid around it. Refinement passes the grid winner of the context's previous Refit (before MoveOverallB moves the overall B into b_sol); it travels with a trial's fit, and a trial's first Refit without one (Context::ForgetStarts, which also clears the likelihood's D start) searches the whole box, so a trial still depends on nothing its slot ran before (side by side = one at a time, md5). Evidence (GPU, -N 8): full refinement of the 14 parity sets from their placed models - 12 refined CIFs identical (md5), 9ig7 and 6iu9 differ, final R-work/R-free equal to 4 digits but 6iu9 R-work 0.2911 -> 0.2910; of the 5 timing cases 4 identical, 7d98 (cif start) R-free 0.3096 -> 0.3095; 5f6m MD starts 01-04 median R-free 0.1692 -> 0.1692; 6o2h on the CPU engine identical (md5). The score-only parity table is not touched (one Refit, no start). refit.scale (TIMING sum): 5f6m 1.52 -> 1.10 s. Wall: 5reo 1.68 -> 1.53, 5f6m 2.43 -> 2.24, 9yl4 7.19 -> 7.02, 7d98 3.70 -> 3.65, 5f6m MD start 4.57 -> 4.22 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff285fb525ecf4b480698d4aafe212a34301a8fe) |
||
|
leonarski_f
|
ca0a6dd343 |
Refinement: each slot's likelihood fit on all of the run's threads
Bitwise (the bins are fitted independently: the same bits at any thread count). The weight-trial slots Refit in lockstep while the GPU idles, each with its 1/slots share of -N for the ML bins; the bins now go to the shared worker pool with -N tasks per slot, so the threads of slots that are not fitting help the ones that are. After the Newton searches (previous commit) the ML fit is 0.1-0.5 s summed over a run, so the wall gain is within noise: 5reo 1.63-1.75 -> 1.72, 5f6m 2.60 -> 2.57, 9yl4 7.34 -> 7.28, 7d98 3.78 -> 3.78, 5f6m MD start 4.82 -> 4.76 s. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 56815f5390cc01e8da5d420fff73cdf4394c385c) |
||
|
leonarski_f
|
2c1df0ebfa |
Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's
Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves 6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a trial depends on nothing its slot ran before (side by side = one at a time, md5). 14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186, D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s, 5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60, 9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free 0.1693 -> 0.1692. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44) |
||
|
leonarski_f
|
3b06bcee78 |
Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win
Three changes to how the trajectory of commit 2bfb84bc4 moves and ends: - A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking on while R-work falls. - Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6). - The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds, R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement that on deposited models cost R-free. Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016 (e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this: 5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 -> 0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857) |
||
|
leonarski_f
|
bda90dbdd5 |
Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms
- B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144) |
||
|
leonarski_f
|
6369a263b1 |
Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start
Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519) |
||
|
leonarski_f
|
4424d8884d |
Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature
Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825) |
||
|
leonarski_f
|
6075239be0 |
Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them
The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec) |
||
|
leonarski_f
|
bcb71136ab |
Refinement restraints: metal ions bound to their donors by the model's LINK records
Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68) |
||
|
leonarski_f
|
9da3851844 |
Refinement: at most n weight trials at a time, and a cancel flag the caller can set
- RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c) |
||
|
leonarski_f
|
8718065a7c |
Refinement GPU: the bulk-solvent mask on its own fine grid on the card
The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10) |
||
|
leonarski_f
|
46ad2c10ed |
Refinement GPU: the robust scale fit on the card, the pool back in every trial slot
The score's robust fine pass (pseudo-Huber IRLS of ModelScaleBox::robust) on the device, so that refinement on a GPU no longer fits its scale on the host every cycle - with the trials side by side that was a robust grid search on one thread per slot (13-133 s per refinement on the seven timing sets, against 1.7-8 s before the score commit). - ModelScaleGPU: the weights multiply the WSSR and J^T J / J^T r in scale_sums (a weight row per fit, reserved at the first robust fit); robust_residuals / robust_weights kernels; the median of each fit's first residuals taken on the host as FitModelScale takes it (nth_element at n/2, on the fits' threads); LevMarBatch split out of FitBatch so that the rounds restart from where a fit stands. FitSolvent(..., robust) refits the coarse winner in the fine pass, as FitModelScale's robust fit does; the plain fit is unchanged. - ModelScaleGPUPool::Fit(..., robust); its streams at the highest priority: a fit is a chain of small launches the host waits on, and on a card shared with the trials' long kernels each wait queued behind them (refit.scale 58-87 ms -> 14-45 ms per cycle, the result the same to the bit). - Refinement: the main context and every trial slot fit on their pool (robust); the CPU engine keeps FitModelScale's robust fit. The card's fit is FitModelScale's to rounding (float residuals and weights, the steps' anisotropic factor in float), deterministic run to run and whichever engine a fit takes: R-work/R-free of the 14 parity sets on the GPU equal the CPU engine's to 4 digits (largest difference 1.4e-6 on the 3.6 A set). Test: ModelScaleGPUPool_RobustMatchesFitModelScale (one reflection in fifty a 4x outlier, five cells). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d15f048cbd7ed2d6b430f9f9cb5018ca284da8f0) |
||
|
leonarski_f
|
c4d2d419ce |
Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles
Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8) |
||
|
leonarski_f
|
8933a72a89 |
Refinement: the weight trials side by side, each on a model, topology and engine of its own
Every rung of the weight ladder starts from the same model and depends on no other, so the six position trials, and then the three B trials at the chosen weight, run at the same time: on the main model and engine and on up to five slots beside it, each a copy of the model before the topology was made (the topology is built again on it, hydrogens included) with its own engine. The slots go on the GPUs in turn from the main engine's, as many as each card's budget takes, so a machine with several cards spreads them; a slot whose engine cannot be made is left out, and with none the trials run one after another on the main engine as before. The threads are shared out among the slots. The log lines and the choice are made afterwards in ladder order, so the log, the trial table and the refined model are the same, bit for bit (GPU and CPU engines checked). 5reo 3.9 -> 2.5 s, 9yl4 13.4 -> 7.8 s, 6o2h 3.1 -> 1.9 s, 8dyz 4.6 -> 2.7 s, 9rcs 2.9 -> 2.2 s (one RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0ea6a6db8aba640afc5249fe748922ddfaa2537a) (cherry picked from commit 0c0176c8bf571ed26f8a68a9de40222487c6200e) |
||
|
leonarski_f
|
5569f3d1f2 |
Refinement GPU: the X-ray target and the restraints in one submission with one wait
RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80) |
||
|
leonarski_f
|
001e61c1a4 |
Refinement: the repulsion search on the refinement's threads, hydrogens skipped before the neighbour walk
BuildRepulsions walked the neighbours of every atom and only then dropped the pairs of a riding hydrogen; those atoms are now skipped before the walk. The atoms are searched in 64 blocks under ParallelFor and the blocks' pairs joined in atom order, so the restraint list is the same as before for any thread count. The model is no longer copied for the NeighborSearch. Same bits. 9yl4: repulsions 3.37 -> 1.09 s (91 calls), refinement 17.7 -> 14.2 s; 5reo 0.38 -> 0.12 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ded11388a18f5a4138a73150c4bb073570152727) (cherry picked from commit 9c96836a90c9f170008d343af2cd885a4e59b321) |
||
|
leonarski_f
|
e3782cabe4 |
Refinement GPU: atom densities uploaded only when some B moved, one device copy, positions through a pinned buffer
Every evaluation uploaded both copies of the atoms' densities (the gather's and the gradient's) and their boxes, although in a positions-only minimisation no B changes between two Refits. The engine now keeps one copy (the gradient kernel reads ModelDensityGPU's atoms and boxes), uploads it only where some atom's density was made again, and sends the positions from a pinned buffer without waiting. The densities that are made again (every atom in a B minimisation) and the grid positions are worked out on the refinement's threads. Same bits. 9yl4: upload 3.29 -> 0.56 s, host side 1.29 -> 0.61 s, refinement 22.3 -> 17.7 s; 5reo 4.8 -> 4.4 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 46c3b67c91aee0aec3473b0fb1a4945c00bda070) (cherry picked from commit 0aebc98110c8155932af647d6204891aec662294) |
||
|
leonarski_f
|
dd49d4e2f2 |
Refinement: the likelihood's D and beta fitted bin by bin on the refinement's threads
EstimateLikelihoodParameters fits each resolution bin on its own and writes only that bin's values, so the bins run under ParallelFor; the result is the same, bit for bit. It was the largest single stage of a refinement (107 calls, one core): 5reo 2.42 -> 0.56 s, 8dyz 2.46 -> 0.57 s, 9yl4 3.54 -> 0.88 s; whole refinement 5reo 6.3 -> 4.8 s, 8dyz 6.9 -> 5.3 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85aa34d3031c8e36be4462fca544bc801af45d5a) (cherry picked from commit d84ba55252adf455d5b4ba62052dfbafa6290a64) |
||
|
leonarski_f
|
24b1711d64 |
Refinement: time per stage - an NVTX range per call and one TIMING line per stage with its sum and calls
RefinementStage sums the time of each stage over a refinement (Refit and its parts, the evaluations and their X-ray, restraint, host and upload shares, the repulsion build, the riding hydrogens, the rungs) and logs it at debug level at the end; rugnux_refine_test -v shows it. No change to any result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9491f54e0beccf36c65ebcec9afff90d08485f86) (cherry picked from commit 0ce206a88df960754476ce2f75ffaed90fbf3123) |
||
|
leonarski_f
|
9cb5cc6237 |
Refinement: placement steps kept only where R-free drops; a wider weight ladder
On deposited models the per-residue B shift raised R-free at every resolution tried (2.56 A: 0.320 -> 0.327) and the rigid bodies rarely helped; each is now kept only where it lowers R-free by more than the 0.001 tie, otherwise the model goes on as it was. The positions' weight ladder runs from 1/1024 to 1/32 of the gradient-balanced weight: the R-free optimum sat near 1/500 at 2.6 A and near 1/50 at 1.1-1.7 A. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d444fb03f610db70903de04f53f0489a71540b37) |
||
|
leonarski_f
|
b4b68d368f |
Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs
RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2) |