215 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	d8c14046b2	rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report ... Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal metric, so the components of G* the point group leaves free (projected from the group's rotation parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000), so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second estimate; the written e.s.d. is the larger of the two per parameter. Stills get none. Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals; report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS, CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT). On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated .ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k lysozyme observations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)	2026-10-26 14:33:38 +01:00
leonarski_f	8a123e794d	Stills: the run's rocking curve goes to _process.h5 and --mode scale merges with it; MeasureRunWidth takes the window at each reflection's resolution ... --mode scale re-measured the rocking width on the stored reflections, so on any file whose frames were predicted through the run's curve it merged with a different curve than the mx run did. The curve the post-refinement used (formula, table, radial scale) is now carried in the EndMessage (StillsRunWidth) and written to the master: /entry/MX/stillsRockingWidthFormula [gamma0 1/A, gamma_e rad, bandwidth sigma], /entry/MX/stillsRadialScale, /entry/MX/stillsRockingWidthTableDstar and /entry/MX/stillsRockingWidthTable (1/A). The reader returns it, and --mode scale passes it to the post-refinement. A file without it is re-measured as before. MeasureRunWidth grouped each frame's reflections by the frame's window, taken as the largest excitation error it predicted. With a window that follows sigma(d*) that is the high-resolution window, applied to the frame's low-resolution reflections too. Given the curve the frames were predicted through, the window is now that curve at the reflection's own resolution; without one, the frame's largest excitation error as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 992343d0206489991880bb927efc79131134a1d9)	2026-10-26 14:09:32 +01:00
leonarski_f	00662c4110	Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records ... The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)	2026-10-26 13:56:29 +01:00
leonarski_f	e406eb17ea	docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits ... - New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10)	2026-10-26 11:07:43 +01:00
leonarski_f	4dbcb35708	Stills partiality: the rocking width measured in shells of equal width in d*, used as a table ... MeasureRunWidth measured the width in eight shells of equal count in d*^2 and fitted the three-term formula gamma0^2 + (gamma_e d*)^2 + (bw |q_z|)^2 to them. On runs predicting ~1000 reflections a frame the equal-count shells all sat at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the fit's form flipped with the number of frames (to a pure bandwidth at 50k frames). The same window-group moment is now also measured in eight shells of equal width in d*, made non-decreasing (pool adjacent violators, weighted by each shell's summed signal) and kept in RunWidth as a table that the partiality interpolates linearly in d*. The formula is still fitted on the equal-count shells and still sets the prediction window. With fewer than three measurable shells the formula is used as before. Still battery, 12 non-float sets, together with the sigma-following prediction window (the previous commit; run 20261026-1021_1b28d6_still-part-comb against rc180 20261026-0857_3882b0_still-part-base1, 0-cycle REFMAC R-free vs deposited, both --no-refine): 6 better, 3 unchanged, 3 worse (by 0.002), sum of changes -0.027 (the window alone: -0.007); largest gains pal90 -0.014, alvra103_int16 -0.011, lcls222 -0.005; largest losses euxfel152 +0.005, alvra213_int16 +0.005 (+0.010 with the window alone), euxfel83 +0.003. Win/draw/loss against the depositions unchanged at 9/2/1. Rotation: p.hkl byte-identical on 5reo, lyso_x06da_ref and thau_x10sa_16keV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit de779bb10b1813eeae625e3994b1ca73d6079d7a)	2026-10-26 10:50:52 +01:00
leonarski_f	0829b9a72d	Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale ... Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its indexed spots): one width at every resolution, differing eightfold between frames. Against the run's measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated). Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way, scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run. Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited): 9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall +13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three rotation sets. Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc)	2026-10-26 10:50:52 +01:00
leonarski_f	13da79b788	Refinement GPU: the atom gradient kernel walks its box without integer division ... Bitwise. One warp per atom, each lane taking the box's points t = lane, lane + 32, ...: every point cost three integer divisions/modulos to turn t into (u, v, w) and three more (imod) to wrap them into the grid. The lane now steps (iu, iv, iw) on by 32 points with adds and carries, and wraps from the box's first point, wrapped once per atom (a box is narrower than the cell, so one conditional subtraction per axis). The same points in the same order per lane, the same arithmetic. atom_gradient_kernel (nsys, side by side): 5reo 426 -> 237 us, 5f6m 887 -> 523, 9yl4 1436 -> 802, 7d98 692 -> 388 us per launch. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Wall: 5reo 1.43 -> 1.35, 5f6m 2.20 -> 2.00, 9yl4 6.88 -> 6.82, 7d98 3.58 -> 3.38, 5f6m MD start 4.16 -> 3.71 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6554e9dcbe4b448eda582eb76d6a24c59fe6ab0c)	2026-10-26 09:16:34 +01:00
leonarski_f	4a8e13ddfa	Refinement GPU engine: its waits on the stream asleep (blocking-sync event) rather than spinning ... Bitwise. cudaStreamSynchronize spins a core for the whole wait; the weight trials' engines wait on a shared card for most of a run, so about half of the user cycles of a refinement were the spin (PROFILE.md: 38-50%). A StreamWaiter (CUDAMemHelpers.h: an event made with cudaEventBlockingSync, recorded and waited for) now takes the refinement engine's waits - evaluation, Fcalc, Fmask, the uploads of positions, row model, blur and restraints. The short waits of the host-driven loops stay spinning: the scale fit's Levenberg-Marquardt (one launch of 66-120 us per step) and the gather's pair count; making those block too cost 2-3% of wall (wake-up latency), here the wall is unchanged. No process-wide cudaDeviceScheduleBlockingSync: that would change every GPU wait of the broker. CPU use (-N 8, time -v): 5reo 301-315% -> 190%, 5f6m 326% -> 181%, 9yl4 333% -> 222-228%, 7d98 307% -> 181%, 5f6m MD start 309-320% -> 151%. Wall, two interleaved runs each: 5reo 1.43-1.52 -> 1.43, 5f6m 2.19-2.20 -> 2.19-2.20, 9yl4 6.83-6.84 -> 6.88-6.89, 7d98 3.50-3.51 -> 3.58, 5f6m MD start 4.16-4.34 -> 4.16 s. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d23d2cf7bb99b30617a3f57c531b03db8bc03e05)	2026-10-26 09:16:34 +01:00
leonarski_f	f34d2f4cca	Refinement: the weight trials' own slots made while the placement runs ... Bitwise. MakeOwnSlot copies the model as it was before the topology and reads only what the placement does not change (cell, reflections, the engine's kind, the B ceiling), so the slots are now built on their threads right after the start score, beside the rigid-body and group-B placement, and joined where they were built before (also where the placement throws). The "slots" stage goes from 0.23 s to 0 on 9yl4; the placement is not slowed (0.59 -> 0.60 s). Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU); side by side = one at a time. Wall: 5reo 1.46-1.53 -> 1.54, 5f6m 2.26 -> 2.21, 9yl4 7.03 -> 6.83, 7d98 3.65 -> 3.56, 5f6m MD start 4.20 -> 4.15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0752a921a4293d4213894e3242c4ef36d8ce2a05)	2026-10-26 09:16:34 +01:00
leonarski_f	1f8b5b1986	Refinement: a Refit's solvent grid starts around where the context's last one ended ... Logic change (deterministic), kept on evidence. Every Refit searched the whole k_sol/b_sol box - an 88-point coarse pass of plain least squares, then the robust fine pass around its winner. ModelScaleBox (and ModelScaleGPUPool::Fit / ModelScaleGPU::FitSolvent) take an optional start; with one, the coarse pass is skipped and the fine pass is laid around it. Refinement passes the grid winner of the context's previous Refit (before MoveOverallB moves the overall B into b_sol); it travels with a trial's fit, and a trial's first Refit without one (Context::ForgetStarts, which also clears the likelihood's D start) searches the whole box, so a trial still depends on nothing its slot ran before (side by side = one at a time, md5). Evidence (GPU, -N 8): full refinement of the 14 parity sets from their placed models - 12 refined CIFs identical (md5), 9ig7 and 6iu9 differ, final R-work/R-free equal to 4 digits but 6iu9 R-work 0.2911 -> 0.2910; of the 5 timing cases 4 identical, 7d98 (cif start) R-free 0.3096 -> 0.3095; 5f6m MD starts 01-04 median R-free 0.1692 -> 0.1692; 6o2h on the CPU engine identical (md5). The score-only parity table is not touched (one Refit, no start). refit.scale (TIMING sum): 5f6m 1.52 -> 1.10 s. Wall: 5reo 1.68 -> 1.53, 5f6m 2.43 -> 2.24, 9yl4 7.19 -> 7.02, 7d98 3.70 -> 3.65, 5f6m MD start 4.57 -> 4.22 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff285fb525ecf4b480698d4aafe212a34301a8fe)	2026-10-26 09:16:34 +01:00
leonarski_f	1f1aa18b67	ModelScaleGPU: the robust fit's median of the residuals on the device ... Bitwise. The first robust round copied every fine-pass fit's residuals to the host (up to 25 x n floats, 9 MB at 92k reflections) and took std::nth_element of each there. The median is now a radix selection on the card (one block per fit, four byte-wide passes of integer counts over the float bits - the residuals are not negative, so the bits order as the values): the element nth_element puts at n / 2, the same value; only the fits' c come back. FitSolvent and RobustBatch lose the host threads they took for it. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start. Wall (-N 8, quiet): 5reo 1.72 -> 1.68, 5f6m 2.57 -> 2.43, 9yl4 7.28 -> 7.19, 7d98 3.78 -> 3.70, 5f6m MD start 4.76 -> 4.57 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc53896c335fdabdf4d417323038e05b8b63535d)	2026-10-26 09:16:34 +01:00
leonarski_f	ca0a6dd343	Refinement: each slot's likelihood fit on all of the run's threads ... Bitwise (the bins are fitted independently: the same bits at any thread count). The weight-trial slots Refit in lockstep while the GPU idles, each with its 1/slots share of -N for the ML bins; the bins now go to the shared worker pool with -N tasks per slot, so the threads of slots that are not fitting help the ones that are. After the Newton searches (previous commit) the ML fit is 0.1-0.5 s summed over a run, so the wall gain is within noise: 5reo 1.63-1.75 -> 1.72, 5f6m 2.60 -> 2.57, 9yl4 7.34 -> 7.28, 7d98 3.78 -> 3.78, 5f6m MD start 4.82 -> 4.76 s. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 56815f5390cc01e8da5d420fff73cdf4394c385c)	2026-10-26 09:16:34 +01:00
leonarski_f	2c1df0ebfa	Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's ... Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves 6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a trial depends on nothing its slot ran before (side by side = one at a time, md5). 14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186, D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s, 5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60, 9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free 0.1693 -> 0.1692. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44)	2026-10-26 09:16:33 +01:00
leonarski_f	a8d3a8dbc2	Refinement: the mask grid sized without allocating it; the L-BFGS pairs keep their s.y ... BulkSolventMask::MaskGrid, documented as "sizes and spacing set, values not allocated", allocated and zeroed the whole fine mask grid (up to 2^26 floats) - in BulkSolventMask's constructor and in every GPU engine's, i.e. once more for each weight-trial slot, only for the grid's size. It now takes good_grid_size() as set_size_from_spacing() does, and the CPU engine's Fmask() allocates its copy. rugnux_refine_test, RTX 5080, -N 8: 9yl4 8.6 -> 7.8 s, 7d98 4.85 -> 4.0 s (engine 0.66 -> 0.16 s, slots 0.60 -> 0.23 s on 9yl4); the high-resolution sets, whose mask grid is small, are unchanged. RefinementMinimiser keeps s.y of every pair it stores and Direction() takes rho and gamma from it instead of two more dot products per pair per call (the same Dot() of the same vectors): Direction's CPU time -22% on 9yl4; no wall change with the trials side by side, where other slots fill the card meanwhile. Bitwise: refined CIF identical on 5reo, 5f6m, 9yl4, 7d98, a 5f6m MD start (GPU) and 6o2h (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6f5caf5e1b1183a752acc5cca78bfb7ef002b12a)	2026-10-26 09:16:33 +01:00
leonarski_f	3882b0f062	Serial stills with a reference or model: crystals their own cells anchor keep that indexing ... The per-image reference/model choice of indexing ignored the crystals' refined cells. On a pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as indexed than after a twin law, the cells decide that law - the rule the data-only resolution already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets the reference choose only among the indexings left open. Exactly merohedral lattices (no metric signal) are unaffected. Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets): pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16 0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged; lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against cells that decide all three laws (96/96/100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f)	2026-10-26 06:49:20 +01:00
leonarski_f	3b06bcee78	Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win ... Three changes to how the trajectory of commit 2bfb84bc4 moves and ends: - A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking on while R-work falls. - Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6). - The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds, R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement that on deposited models cost R-free. Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016 (e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this: 5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 -> 0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857)	2026-10-26 01:38:24 +01:00
leonarski_f	bda90dbdd5	Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms ... - B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)	2026-10-26 01:38:24 +01:00
leonarski_f	6369a263b1	Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start ... Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519)	2026-10-26 01:38:24 +01:00
leonarski_f	4424d8884d	Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature ... Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)	2026-10-26 01:38:24 +01:00
leonarski_f	626afc49dd	Refinement likelihood: the searches over D alone with the variance taken once ... EstimateLikelihoodParameters' golden-section searches over D at a fixed beta - four per bin on the free reflections, and the one on the working reflections the D/FOM change added, 50 evaluations over 95 % of the data - evaluated -ln L reading every row through the index and taking log(var) at every evaluation. The rows are now gathered per search with their variance and its logarithm; the sum is the same expressions in the same order, so D, beta and everything after are the same to the bit (refined model and report identical on the seven timing sets). refit.ml 200 -> 140 ms per cycle on the 1.1 A set, one thread per trial slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 23ad3bfc254d10af310f17dbb8f85616f666a783)	2026-10-26 01:38:24 +01:00
leonarski_f	6075239be0	Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them ... The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec)	2026-10-26 01:38:24 +01:00
leonarski_f	bcb71136ab	Refinement restraints: metal ions bound to their donors by the model's LINK records ... Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68)	2026-10-26 01:38:24 +01:00
leonarski_f	9da3851844	Refinement: at most n weight trials at a time, and a cancel flag the caller can set ... - RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c)	2026-10-26 01:38:24 +01:00
leonarski_f	8718065a7c	Refinement GPU: the bulk-solvent mask on its own fine grid on the card ... The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10)	2026-10-26 01:38:24 +01:00
leonarski_f	46ad2c10ed	Refinement GPU: the robust scale fit on the card, the pool back in every trial slot ... The score's robust fine pass (pseudo-Huber IRLS of ModelScaleBox::robust) on the device, so that refinement on a GPU no longer fits its scale on the host every cycle - with the trials side by side that was a robust grid search on one thread per slot (13-133 s per refinement on the seven timing sets, against 1.7-8 s before the score commit). - ModelScaleGPU: the weights multiply the WSSR and J^T J / J^T r in scale_sums (a weight row per fit, reserved at the first robust fit); robust_residuals / robust_weights kernels; the median of each fit's first residuals taken on the host as FitModelScale takes it (nth_element at n/2, on the fits' threads); LevMarBatch split out of FitBatch so that the rounds restart from where a fit stands. FitSolvent(..., robust) refits the coarse winner in the fine pass, as FitModelScale's robust fit does; the plain fit is unchanged. - ModelScaleGPUPool::Fit(..., robust); its streams at the highest priority: a fit is a chain of small launches the host waits on, and on a card shared with the trials' long kernels each wait queued behind them (refit.scale 58-87 ms -> 14-45 ms per cycle, the result the same to the bit). - Refinement: the main context and every trial slot fit on their pool (robust); the CPU engine keeps FitModelScale's robust fit. The card's fit is FitModelScale's to rounding (float residuals and weights, the steps' anisotropic factor in float), deterministic run to run and whichever engine a fit takes: R-work/R-free of the 14 parity sets on the GPU equal the CPU engine's to 4 digits (largest difference 1.4e-6 on the 3.6 A set). Test: ModelScaleGPUPool_RobustMatchesFitModelScale (one reflection in fifty a 4x outlier, five cells). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d15f048cbd7ed2d6b430f9f9cb5018ca284da8f0)	2026-10-26 01:38:24 +01:00
leonarski_f	c4d2d419ce	Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles ... Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8)	2026-10-26 01:38:23 +01:00
leonarski_f	8933a72a89	Refinement: the weight trials side by side, each on a model, topology and engine of its own ... Every rung of the weight ladder starts from the same model and depends on no other, so the six position trials, and then the three B trials at the chosen weight, run at the same time: on the main model and engine and on up to five slots beside it, each a copy of the model before the topology was made (the topology is built again on it, hydrogens included) with its own engine. The slots go on the GPUs in turn from the main engine's, as many as each card's budget takes, so a machine with several cards spreads them; a slot whose engine cannot be made is left out, and with none the trials run one after another on the main engine as before. The threads are shared out among the slots. The log lines and the choice are made afterwards in ladder order, so the log, the trial table and the refined model are the same, bit for bit (GPU and CPU engines checked). 5reo 3.9 -> 2.5 s, 9yl4 13.4 -> 7.8 s, 6o2h 3.1 -> 1.9 s, 8dyz 4.6 -> 2.7 s, 9rcs 2.9 -> 2.2 s (one RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0ea6a6db8aba640afc5249fe748922ddfaa2537a) (cherry picked from commit 0c0176c8bf571ed26f8a68a9de40222487c6200e)	2026-10-26 01:38:23 +01:00
leonarski_f	5569f3d1f2	Refinement GPU: the X-ray target and the restraints in one submission with one wait ... RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80)	2026-10-26 01:38:23 +01:00
leonarski_f	001e61c1a4	Refinement: the repulsion search on the refinement's threads, hydrogens skipped before the neighbour walk ... BuildRepulsions walked the neighbours of every atom and only then dropped the pairs of a riding hydrogen; those atoms are now skipped before the walk. The atoms are searched in 64 blocks under ParallelFor and the blocks' pairs joined in atom order, so the restraint list is the same as before for any thread count. The model is no longer copied for the NeighborSearch. Same bits. 9yl4: repulsions 3.37 -> 1.09 s (91 calls), refinement 17.7 -> 14.2 s; 5reo 0.38 -> 0.12 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ded11388a18f5a4138a73150c4bb073570152727) (cherry picked from commit 9c96836a90c9f170008d343af2cd885a4e59b321)	2026-10-26 01:38:23 +01:00
leonarski_f	e3782cabe4	Refinement GPU: atom densities uploaded only when some B moved, one device copy, positions through a pinned buffer ... Every evaluation uploaded both copies of the atoms' densities (the gather's and the gradient's) and their boxes, although in a positions-only minimisation no B changes between two Refits. The engine now keeps one copy (the gradient kernel reads ModelDensityGPU's atoms and boxes), uploads it only where some atom's density was made again, and sends the positions from a pinned buffer without waiting. The densities that are made again (every atom in a B minimisation) and the grid positions are worked out on the refinement's threads. Same bits. 9yl4: upload 3.29 -> 0.56 s, host side 1.29 -> 0.61 s, refinement 22.3 -> 17.7 s; 5reo 4.8 -> 4.4 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 46c3b67c91aee0aec3473b0fb1a4945c00bda070) (cherry picked from commit 0aebc98110c8155932af647d6204891aec662294)	2026-10-26 01:38:22 +01:00
leonarski_f	dd49d4e2f2	Refinement: the likelihood's D and beta fitted bin by bin on the refinement's threads ... EstimateLikelihoodParameters fits each resolution bin on its own and writes only that bin's values, so the bins run under ParallelFor; the result is the same, bit for bit. It was the largest single stage of a refinement (107 calls, one core): 5reo 2.42 -> 0.56 s, 8dyz 2.46 -> 0.57 s, 9yl4 3.54 -> 0.88 s; whole refinement 5reo 6.3 -> 4.8 s, 8dyz 6.9 -> 5.3 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85aa34d3031c8e36be4462fca544bc801af45d5a) (cherry picked from commit d84ba55252adf455d5b4ba62052dfbafa6290a64)	2026-10-26 01:38:22 +01:00
leonarski_f	24b1711d64	Refinement: time per stage - an NVTX range per call and one TIMING line per stage with its sum and calls ... RefinementStage sums the time of each stage over a refinement (Refit and its parts, the evaluations and their X-ray, restraint, host and upload shares, the repulsion build, the riding hydrogens, the rungs) and logs it at debug level at the end; rugnux_refine_test -v shows it. No change to any result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9491f54e0beccf36c65ebcec9afff90d08485f86) (cherry picked from commit 0ce206a88df960754476ce2f75ffaed90fbf3123)	2026-10-26 01:38:22 +01:00
leonarski_f	07ae590e54	Serial stills: the indexing ambiguity resolved from the data when no reference or model is given ... Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without -z or --model used to merge the crystals' random indexings together (and warned). The crystals are now made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities, weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent, four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest crystals. Deterministic (fixed summation order, sequential passes). A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals' own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as indexed, those crystals are anchors and only the others are decided by the intensities. Without this, two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350). With a reference or model the per-image choice stands; the same resolution runs as a check, reported as AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS, AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev). --mode scale is not changed. Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with --model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213 0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t 0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes 0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36)	2026-10-25 23:26:32 +01:00
leonarski_f	c982ba4f65	FFBIDX on the GPU: the same spots index to the same cells, whatever order the thread blocks finish in ... Each block of the candidate-vector and the cell search sorts its own candidates and merges them into one shared top list under a lock, in the order the blocks happen to get there. The merge compared values only, so where two candidates scored the same - common on the 30-spot seed the stills path tries first, since the score is a sum of trimmed terms - the place went to whichever block merged first. The same frame then indexed to a slightly different lattice, or not at all, from one run to the next. Ties are now broken by the candidate's identity - the sample index for a vector; the cell vector (with its flip), candidate and rotation sample for a cell - which makes the order total, so the merged lists are the top-N of everything whatever the merge order. The blocks' own lists are already in that order: the block radix sort is stable and its samples ascend with the thread index. Measured on a 100k-frame serial set: 4-21 of ~33k indexed frames came out with a different lattice in every pair of runs, which the stills scaling and partiality chain turned into 0.97-0.99 CC1/2 and thousands of observations in or out of the merge. With the fix, three runs at the default thread count and one at -N 8 give the md5-identical merged .hkl; a per-frame dump (spot list, indexer answer, refined lattice) of 20k frames is identical across three runs where it differed in 4-6 frames before. FFBIDXIndexerCPU_MatchesGPU and Portable_FFBIDXIndexerCPU_SyntheticStill pass. No regression test: a repeat-and-compare test on the library's spot lists, threaded, passed before the fix too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9cff12d652088851b81f6a353ac52aa8318799f2)	2026-10-25 19:58:38 +01:00
leonarski_f	64bbc44e11	fast-feedback-indexer vendored in image_analysis/indexing/fast-feedback-indexer instead of fetched ... The GPU known-cell indexer behind FFBIDX is now built from an in-tree copy of upstream commit a94fd4f5c55289e70860feea358182d2072e02b5 (the archive the FetchContent block pinned), so it can carry local fixes. No behaviour change. Copied: indexer/src/{indexer.cpp, indexer_gpu.cu, log.cpp} and the ffbidx/ headers the build uses (indexer.h, indexer_gpu.h, refine.h, log.h, exception.h, envvar.h), each with a header line naming the upstream URL, commit and licence; LICENSE.md beside them; and the library's test spot lists (data/simple/files/image{0..8}_{local,peakfinder8,radial}.txt) that FFBIDXIndexerCPU_MatchesGPU reads, now found in-tree. ffbidx/version.h is written once ("1.1.3-a94fd4f") instead of generated - the generator ran git in the parent build, so it used to stamp Jungfraujoch's own hash. The in-tree target keeps the name fast_indexer_static and what upstream's indexer/src/CMakeLists.txt gives the static library: the source list, CUDA_RUNTIME_LIBRARY Static, cxx_std_17 (PUBLIC), the source directory as PUBLIC include, CUDA::cudart_static and Threads::Threads (PRIVATE), Eigen3::Eigen (PUBLIC, the project's), the CUDA-architecture default only where none is set, and -Xcompiler=/Zc:preprocessor for the CUDA sources under MSVC (CUB needs the conforming preprocessor). Left out: the shared library, the C API (c_api.cpp, unused), install/pkg-config rules, the version_txt generator target, the top-level project's data/tests/python/examples subdirectories, and -DVECTOR_CANDIDATE_REFINEMENT=1 (no source reads it). Licences: COLLECT.sh takes the text from the vendored LICENSE.md; the THIRD_PARTY_NOTICES row moves from the fetched to the vendored table. CLAUDE.md's list of FetchContent downloads no longer names it. Verified: clean configure and build of rugnux and jfjoch_test; FFBIDXIndexerCPU_MatchesGPU and the Portable_FFBIDXIndexerCPU tests pass; a 3000-image stills run gives the md5-identical merged .hkl with the fetched and the vendored library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0e4fcb87de0fbb7c0022750ff4d62d8c5a5c3c18)	2026-10-25 19:58:38 +01:00
leonarski_f	cf7676f5d5	Stills merge: outlier rejection on by default (10 sigma), the band linear in the error model's sigma ... The stills merge had no working outlier rejection. --reject-outliers was 0 for stills, and when set its band took the systematic part as a factor exp(+-n s), s the spread of ln(I / median) - on stills a reflection's observations are partials and per-crystal scales, many near zero, so s is wide and the band let through observations hundreds of sigma out. On a high-multiplicity broad-bandpass run with a spotty powder contaminant, single grains in a weak reflection's box were that reflection's whole merged intensity (half-set means -2 vs 2284 on 141 observations); the shells holding the powder rings and the lowest shell merged at CC1/2 0.6-0.75 beside 0.99 elsewhere, at multiplicity ~400-1000. Stills now test the plain linear band, n sqrt(a sigma^2 + (b <I>)^2) about the median - the error model's b is the spread as it is - at n = 10 by default (rotation keeps 6 and its multiplicative band). The stills merge statistics now report the rejected count (OBSERVATIONS_REJECTED was always 0 there). Measured (REFMAC 0-cycle R-free of an isomorphous room-temperature model, depositor's free set, scale mode on stored integrations, vs the same code with --reject-outliers 0): - broad-bandpass in-house stills, 50k frames: 0.318 -> 0.265; 20k: 0.274 -> 0.244; R_split 0.29 -> 0.06, CC1/2 0.79 -> 0.98. 0.1% of the observations are rejected. - monochromatic in-house stills, 30k frames: 0.216 -> 0.212; R_split 0.126 -> 0.087. - n = 6 gains ~0.003 more there but costs the public XFEL sets 0.003-0.006 on three of four; 20 keeps most of the gain. Still battery, 12 non-float sets, against rc180: R_split better on all 12; R-free change mean -0.0005 (range -0.006..+0.003, plus pal86 +0.009 that is within that set's run-to-run spread of ~0.007 - a rerun of rc180's behaviour gave 0.192 against the baseline's 0.185); one draw becomes a win (lcls242_s), nothing else changes class. Rotation is untouched (the stills band and default are gated on the experiment not being a rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3a7142801056f21c720956255f1207e0bf1a8f43)	2026-10-25 19:58:38 +01:00
leonarski_fandClaude Opus 5.5	f077320efa	Refinement restraints: n_slots, not slots - Qt defines slots as a macro, which broke the viewer's build through Rugnux.h ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 10:22:02 +01:00
leonarski_f	789bd37606	Refinement: ten short macro-cycles instead of five long ones ... The scale, the bulk solvent and the likelihood's parameters are held within a macro-cycle; refitting them twice as often, with 10 L-BFGS iterations on the positions and 5 on B per cycle (was 25 and 15), lowered R-free on a 1.65 A deposited model from 0.2218 to 0.2189 with B held, and halved what B refinement cost there, at the same number of evaluations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8021d0cfc0bd56b1cc149a40df72ed78f35367c8)	2026-10-25 07:20:32 +01:00
leonarski_f	9cb5cc6237	Refinement: placement steps kept only where R-free drops; a wider weight ladder ... On deposited models the per-residue B shift raised R-free at every resolution tried (2.56 A: 0.320 -> 0.327) and the rigid bodies rarely helped; each is now kept only where it lowers R-free by more than the 0.001 tie, otherwise the model goes on as it was. The positions' weight ladder runs from 1/1024 to 1/32 of the gradient-balanced weight: the R-free optimum sat near 1/500 at 2.6 A and near 1/50 at 1.1-1.7 A. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d444fb03f610db70903de04f53f0489a71540b37)	2026-10-25 07:20:32 +01:00
leonarski_f	a3082c1500	DensityBlobs: an ion is called by its anomalous peak alone; ligand size scales gentler with resolution ... Measured by stripping the deposited ligands and ions of 12 open-arm models (placement-refined, then the blobs of mFo-DFc): ligands with a blob 34/39, called LIGAND 26/39, LIGAND blobs on a removed ligand or ion 24/33; ions with a blob 23/40. The ligand-size threshold was four resolution spheres, which called no glycan at 3.3 A and every ion at 1.1 A; it is now one and a half, and never under 8 A^3. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8e66d58ba87c35119df3ba8779ca193053f5cf00)	2026-10-25 07:20:32 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00
leonarski_f	e58453a693	Refinement restraints: the CCP4 Monomer Library subset embedded in rugnux, read through gemmi's MonLib ... A restraint dictionary carried in the binary, so refinement reads nothing from disk and needs no CLIBD_MON: the 20 amino acids and MSE, the DNA/RNA nucleotides, ATP/ADP/AMP/GTP/GDP/GMP, DMSO, PEG fragments (PEG, PGE, PG4, 1PE), GOL, EDO, SO4, PO4, ACT, MPD, water and common ions - copied verbatim from the CCP4 Monomer Library (LGPL-3.0) - plus links_and_mods.cif cut down to the links and modifications those compounds can take (peptide, cis-peptide, proline, N-methyl, disulfide, nucleic-acid backbone, termini, metal-Cys/His) and ener_lib.cif cut to its atom-type table. monomers/update_monomers.py holds the compound list and regenerates the subset; CMake embeds every .cif of the directory as string pieces under MSVC's literal limit; EmbeddedMonomerLibrary() parses them from memory into a gemmi::MonLib once per process. About 0.7 MB of text in the binary. gemmi v0.7.5's monlib.cpp, topo.cpp and riding_h.cpp vendored verbatim (prepare_topology). Licence text in licenses/, rows and a note in THIRD_PARTY_NOTICES.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 38034718052c4a300691393b5b817f476c83dfc2)	2026-10-25 07:20:32 +01:00
leonarski_f	3c73b469f5	rugnux: TIMING marks for the rest of the rotation tail - ingest sub-steps, the analyses, the pass's teardown and the pinned-pool release ... The tail after the last image had unmarked gaps: the device ingest's second upload, geometry smoothing and partiality recompute were one "smoothing" mark; the analyses step was one mark although the anisotropy future, not tNCS, is what it waits on; and the time between "Processed N images" and the end of the run (the pass's locals going, the pinned pool handed back) had none. Logging only: outputs unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4ce9e5d6d489324a759f155ad9718271c52ba86b)	2026-10-25 07:20:25 +01:00
leonarski_f	78d3b7ef75	Stills geometry: fit only the given cell's crystal form; a stray frame is no cell population ... RefineStillsGeometry fitted distance, beam and cell on the strongest sampled frames whatever their cell, so a sample holding two crystal forms put the cell, and with it the distance, between them - while the merge later keeps only the given cell's population (GivenCellPopulation). The sampled frames are now filtered by the same population before the bundle and the module placement are formed, the population told from the strong frames (>= 20 indexed spots), whose cells show the valley between two forms; a weak frame's cell at the start geometry spreads across it. A run of one form has no valley and keeps every frame. GivenCellPopulation: a stray frame or two far out in a population's tail is a local maximum of the histogram with no valley of its own, so its range was everything, both forms included, and its mean - the mean of all frames - could sit nearer the given cell than the given form's own and win. A population is now represented only by its highest peak. Test CellPopulation_StrayFramesBeyondTheTail (fails before, passes after). Still battery, against the same code without the change: the two runs of a two-form sample -0.0019 and -0.0015 R-free (0-cycle REFMAC; the second repeated three times, -0.0013..-0.0021, run noise ~0.001), R_split 10.6 -> 9.8 % on the first; single-form controls fit the identical geometry (R-free within 0.0003). Rotation does not call either path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ada1b08003fa9748175cce381fc8f28569f28296)	2026-10-25 07:20:25 +01:00
leonarski_f	c62a67975e	Polarization plane: measured on the pooled background, refused beyond what a source can give, logged, and kept across passes ... The pre-scan's plane test read per-pixel sector medians of the mean projection. On a large photon-counting detector with a background of a fraction of a count per pixel over the 60 frames read, those medians were zero and the test returned no harmonic at all: on 21 of 255 battery sweeps (EIGER2 16M and JUNGFRAU 4M sweeps, the laboratory EIGER sweeps, the small-molecule sweeps at 20-25 keV) the correlation was NaN or within 0.01 of zero, so the plane could not be decided there. The background is now pooled over 9 x 9 pixels first, as the mask itself already is; 216 of 244 synchrotron sweeps then show their (horizontal) plane past both gates, against 159 of 229 before. Pooling also lifted one long-distance sweep (500 mm, 2theta under 21 deg, polarization can move its background by at most 4 %) over the correlation gate: its background carries a 10 % cos 2phi harmonic along image y, 2.9 times what a fully polarized source would give, on a horizontally polarized beamline. No polarized source exceeds a scale of 1, so a harmonic of twice that or more now decides nothing (PLANE_MODULATION_MAX). The pre-scan logs the correlation and the scale on every run that detects the beam stop (GetPlaneEvidence). A turned plane is kept on the run and re-applied where a pass starts again from an experiment taken before the decision while the shadow is already in the mask (the rotation measured-centre arm, the no-lattice pass), which previously went on with the stated plane. No battery sweep turns; where the plane is not turned the applied coefficient is unchanged. The degree is still never fitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 50a145a4f1ddf6d58332a8ab745627dc36a2dc3d)	2026-10-25 07:20:25 +01:00
leonarski_f	fca2a77b0c	Scaling: the low-resolution limit bounds the scale fit, not the merge ... Reflections coarser than --scaling-low-resolution (default 50 A) are now merged and written - what maps and refinement want of the lowest resolutions - but stay out of every fit of the scale model, by the same path an ice-ring observation takes (excluded from scaling, kept for merging), under a name of its own: RotationScaleMerge's per-observation flag is now `unscaled` (UNSCALED_ICE | UNSCALED_LOW_RES; on the device OBS_UNSCALED = OBS_ON_ICE_FLAG | OBS_LOW_RES_FLAG), set from the reflection on the host and in the device ingest, and only its ice bit marks a merged reflection as on an ice ring. The stills merge drops its low-resolution cut; ScaleOnTheFly, StillsPartialityRefine and the reindexing correlation keep it. The merge statistics run over the whole sphere unless --report-resolution declares a range, so the reflections the beam stop hides count as missing. Which coarse reflections exist at all is the beam-stop shadow test's decision (a7610d847). Why not simply lift the limit: on 5nw5, 35 reflections beyond 50 A bent the scale model enough to move the merged 20-50 A intensities by +61 % and 10-20 A by +19 % (Wilson B 319 -> 382, REFMAC R-free +0.006). Kept out of the fit, the merged intensities at 10-50 A are unchanged to 0.2 % and Wilson B is 307. CPU subset battery against the previous commit (lp-cpu-v2 / lp-cpu-v1): REFMAC 0-cycle R-free 5nw5 +0.0004, 6oel +0.0005, others within 0.0001; rugnux's own R-free/R_model rise where the new >50 A terms enter it (lyso R_model 0.2714 -> 0.2739; the model is poorest there). Kept beyond 50 A: 7qij 61, 5nw5 35, 6oel 14 unique. In-house photon-counting sets unchanged, except insu_I_x06da_ref, where the delta-CC1/2 frame exclusion now removes 449 instead of 78 frames: refres R_meas 0.246 -> 0.167, ISa 33.5 -> 35.4, R_free 0.2443 -> 0.2427, anomalous signal 3.31 -> 3.38 (XDS: R_meas 0.349, ISa 25.1). Stills (lcls242_s/t, lcls222, euxfel83, sacla236) within 0.0002 of the previous commit. GPU (device ingest) and CPU give the same 5nw5 merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 32a9fbcb3ca67879ceb051c13f58d61fe85530f4)	2026-10-25 07:20:21 +01:00
leonarski_f	e97b902603	Beam stop: a reflection whose signal disk reaches the shadow is not integrated; no-data pixels are dead, not shadow ... The profile fit keeps a signal disk that lost pixels to the mask and fits the pixels that remain. That holds for a gap or a dead pixel, but not for the beam-stop shadow: the mask is drawn where the background is dimmed, and the Bragg half-shadow does not end there, so the rest of such a disk is attenuated too. Measured on summed rocking events against their symmetry mates (stratified by resolution and zeta, the detector edge excluded), observations whose disk touches the shadow read low by 12-60 % on 5uth, 7qij, 5nw5, 3r6o, 3meb (CCD and PILATUS alike) and level on the in-house photon-counting sweeps; observations with a clean disk are unbiased except on 5nw5, whose half-shadow reaches ~20 px past the mask. The test sits in MXAnalysisWithoutFPGA::IntegrateOnImage, so rotation, stills, the _process.h5 and --mode scale see the same reflections; with no shadow in the mask (the broker) it does nothing. The shadow finder also reports pixels holding a detector's no-data mark (PILATUS CBF module gaps at -1): they are the darkest pixels there are. The pre-scan now masks those as dead (bit 10) instead of beam-stop shadow (bit 9), so that the shadow test does not drop every reflection clipped by a module gap and the reported shadow share is the shadow's. A mean of zero is not taken as dead: a deep shadow on a still counts nothing in the projected frames. CPU subset battery vs rc180 (REFMAC 0-cycle R-free): 3meb -0.0004, 5uth -0.0004, 6h2p -0.0003, 6oel -0.0002, 3r6o +0.0001, 7qij/7rji 0, 5nw5 +0.002 (its lowest-shell CC1/2 0.78 -> 0.89, ISa 8.4 -> 9.3); in-house refres ISa up 0.1-0.3 on lyso/thau/insu/cytc/myob, half_image refres R_meas 75 -> 64 %, no set lost quality. Stills (lcls242_s/t, lcls222, euxfel83, sacla236): R-free within +-0.002, the noise of a perturbed run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a7610d847694b5540dacc9ef609344ea7c7ae8fd)	2026-10-25 07:20:21 +01:00
leonarski_f	2dbb9d5951	Stills partiality: merge only partials of p >= 0.5, adopt the refined model on every crystal, centre the rocking curve on the run's radial scale ... Three changes to the stills post-refinement and merge, decided on 0-cycle REFMAC R-free against the deposited models (diff = ours - deposited data, noise ~0.002): - Stills merge only observations whose model partiality is at least 0.5 (RugnuxDefaultScalingSettings; rotation keeps 0.02, and --min-partiality still overrides). Diagnosis on per-observation dumps: the empirical rocking curve has a floor of 5-15 % of its peak out on the tail (mis-oriented crystals), and dividing such observations by a small model p makes them biased, not noisy. Variance terms in (1-p) or (1-p)/p recovered a third of what the cut does; a narrower (correct-looking angular) width or a Lorentzian form was worse. On one set in scale mode the optimum is flat over 0.3-0.6. - The CC adoption gate in RefineOne is gone: it left 25-33 % of crystals at p = 1 next to the refined ones, and collapsed to 3 % on a run whose curve is off-centre. - The run's radial scale of the reciprocal lattice (a wavelength or cell a little off, invisible to spot positions since they fix only lambda d*) is measured from where the curve is centred against d*^2 (MeasureRadialScale, iterated three times) and applied to q in the width measurement and the post-refinement. One run had its curve centred at +2-3e-3 d*^2 (scale -1.6e-3 measured); the others measure below 7e-4. --mode scale on stills now starts where the full pipeline starts: p = 1, the excitation error recomputed from the stored lattice (the window was read as zero before, so no run width was ever measured), and collapsed scales rejected after the refinement. Still battery, the 12 non-float sets, change of diff vs the previous build: -0.0035 mean, 9 better, three within +-0.0006 except one +0.0053 (an edge win becoming a draw); the off-centre run -0.005 here and -0.017 in an earlier run of the same algorithm (that set varies by ~0.01 between runs). Rotation p.hkl byte-identical on three sets. Test StillsPartialityRefine_MeasureRadialScale added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d85e5b975ba863828bd17ebc3b5308e9aa294ce)	2026-10-25 07:20:21 +01:00
leonarski_f	789e064e6a	Stills scaling: a model error in the per-crystal scale fit, and the rocking width measured per window group ... Two changes to the stills scale model; rotation is untouched (p.hkl md5 identical on two rotation sets). The per-crystal scale fits (ScaleOnTheFly and the partiality post-refinement's G and tilt) measured residuals in counting sigmas. The strong reflections of a still scatter about the model by tens of percent (robust sd of ln(I/<I>) 0.45-0.8 by shell on one run), so with a Cauchy loss at 3 counting sigma every strong reflection was an outlier and the scale was set by the weak ones: re-fitting each frame's scale against the final merge moved it by a factor 0.45-1.5 (5-95%). The fit's variance is now the expected one at the model, background + counting + (0.3 * model)^2. The coefficient is not delicate: 0.1-1.0 fit alike in an offline replay; what matters is that the term is there. The run's rocking width came from the intensity-weighted moment of the excitation error over all frames pooled. Frames predict through windows of their own that differ by an order of magnitude, and the widest windows dominated both moments: on a run whose windows are mostly narrower than the curve at high resolution the measurement read the window at every resolution, 2.8e-4 1/A with no angular term, several times too wide at low resolution (low-partiality observations there merged at 0.1-0.3 of their reflection's mean). The moment is now taken per group of frames of similar window (4 per shell), each against its own window, and the shell takes the median of the groups whose curve is narrower than their window. On that run the width becomes 6.7e-5 1/A + 0.018 deg angular; where the pooled measurement was already right (a SASE-bandwidth run), it stays where it was. 0-cycle REFMAC R-free against the deposited models, 9 still battery sets (6 used while developing, 3 held out), change from rc179: -0.0100, -0.0095, -0.0057, -0.0055, -0.0051, -0.0045, -0.0030, -0.0024, +0.0009 (held out: -0.0055, -0.0051, +0.0009). The width change alone accounts for most of the largest gain; the scale-fit variance for 0.000-0.003 per set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ccc0e01f56827bbe0648ed7c5b8049912d80650f)	2026-10-24 16:57:12 +02:00
leonarski_f	b00c668e3a	v1.0.0-rc.179 (#90) ... * Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer. Reviewed-on: #90 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-24 14:52:29 +02:00