128e3bb9aed76abb5de7a03a39793b3e8564d159
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 leonarski_f
|
e406eb17ea |
docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits
- New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10) |
||
|
leonarski_f
|
2c1df0ebfa |
Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's
Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves 6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a trial depends on nothing its slot ran before (side by side = one at a time, md5). 14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186, D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s, 5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60, 9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free 0.1693 -> 0.1692. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44) |
||
|
leonarski_f
|
6075239be0 |
Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them
The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec) |
||
|
leonarski_f
|
8718065a7c |
Refinement GPU: the bulk-solvent mask on its own fine grid on the card
The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10) |
||
|
leonarski_f
|
c4d2d419ce |
Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles
Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8) |
||
|
leonarski_f
|
5569f3d1f2 |
Refinement GPU: the X-ray target and the restraints in one submission with one wait
RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80) |
||
|
leonarski_f
|
dd49d4e2f2 |
Refinement: the likelihood's D and beta fitted bin by bin on the refinement's threads
EstimateLikelihoodParameters fits each resolution bin on its own and writes only that bin's values, so the bins run under ParallelFor; the result is the same, bit for bit. It was the largest single stage of a refinement (107 calls, one core): 5reo 2.42 -> 0.56 s, 8dyz 2.46 -> 0.57 s, 9yl4 3.54 -> 0.88 s; whole refinement 5reo 6.3 -> 4.8 s, 8dyz 6.9 -> 5.3 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85aa34d3031c8e36be4462fca544bc801af45d5a) (cherry picked from commit d84ba55252adf455d5b4ba62052dfbafa6290a64) |
||
|
leonarski_f
|
24b1711d64 |
Refinement: time per stage - an NVTX range per call and one TIMING line per stage with its sum and calls
RefinementStage sums the time of each stage over a refinement (Refit and its parts, the evaluations and their X-ray, restraint, host and upload shares, the repulsion build, the riding hydrogens, the rungs) and logs it at debug level at the end; rugnux_refine_test -v shows it. No change to any result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9491f54e0beccf36c65ebcec9afff90d08485f86) (cherry picked from commit 0ce206a88df960754476ce2f75ffaed90fbf3123) |
||
|
leonarski_f
|
b4b68d368f |
Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs
RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2) |