9 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	2be88e11e3	Refinement restraints: contacts with lattice translates are kept, and an atom's own symmetry contact once ... Contacts() dropped every pair found in image 0 with a nonzero lattice shift ("the identity term covers the contact"), but that pair is a contact with a translated copy of j, not with j itself: in P1 no intermolecular repulsion or B-contact restraint was made at all, and in every group the contacts along a, b and c were missing (MeasureGeometry's clash count with them). Such a pair is now kept like a symmetry mate's, from the lower atom of the two. An atom against its own image x -> T x is also met through T^-1 at the same distance (another operator, or the same one at another lattice shift for a screw), and both were kept: atoms near a 3-, 4- or 6-fold or a screw axis were repelled twice. For i == j the image first in fractional coordinates of T x and T^-1 x is kept; an involution is kept once as before. Test: in P1 two waters 2.4 A apart across a cell face get one repulsion and one B contact (none before); a water 1.4 A from a 3-fold axis gets one repulsion with its images (two before). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d6e4d0491e655a49d59b709e1398843506d162a)	2026-10-26 15:38:59 +01:00
leonarski_f	00662c4110	Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records ... The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)	2026-10-26 13:56:29 +01:00
leonarski_f	6369a263b1	Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start ... Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519)	2026-10-26 01:38:24 +01:00
leonarski_f	4424d8884d	Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature ... Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)	2026-10-26 01:38:24 +01:00
leonarski_f	6075239be0	Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them ... The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec)	2026-10-26 01:38:24 +01:00
leonarski_f	bcb71136ab	Refinement restraints: metal ions bound to their donors by the model's LINK records ... Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68)	2026-10-26 01:38:24 +01:00
leonarski_f	9da3851844	Refinement: at most n weight trials at a time, and a cancel flag the caller can set ... - RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c)	2026-10-26 01:38:24 +01:00
leonarski_f	d3673d17fc	RefinementTest: the GPU restraint check on an engine made for a cell its gather takes ... The restraints do not read the cell; the 14 A cell that makes the symmetry contacts is too narrow for the gather's boxes, so the engine was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a6e00e10ce1798a97d1f054c7553cc9eb6b5cf4b)	2026-10-25 07:20:32 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00