128e3bb9aed76abb5de7a03a39793b3e8564d159
229
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Fable 5.1
|
128e3bb9ae |
RefinementGPU: the symmetry phases from a table instead of sincospi in the compose and scatter kernels
exp(2 pi i h.t) depends only on the integer h.t (in units of 1/den), so the engine tabulates it once over the range the rows and operators make, with the same op_phase() at the same double argument - the same bits. The compose kernel reads the table at h.t; the scatter terms carry their table entry, and their row directly, so neither hot kernel divides: the rotations go to the device in whole units (gemmi's are den times an integer matrix), the term's row no longer has to be split from its operator. The table is a few thousand entries, the scatter terms one int wider. Outputs are byte-identical; the X-ray evaluation counter drops 7-10% on the demo-sized sets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Fable 5.1
|
dfb1536fe2 |
Sweep ends, overloads and ice flags: one walk over the reflections instead of two
Before the merge, the integrated reflections were walked twice - once per frame for what the blank sweep ends are decided on, once more for the overload counts and the ice flags - and each walk reads the whole reflection set from memory. The frame's sweep-end tally is now a function of its own (TallySweepEnd), and the merge step makes it in the same per-frame walk that counts the overloads and flags the ice, then adds the counts up over the frames still in, so the trimmed and left-out frames count nothing, as before. Per frame the sums are the same sums in the same order; the totals are integers. TrimBlankSweepEnds keeps its one-argument form for the callers that walk the frames for nothing else. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Fable 5.1
|
093ac1612c |
Space-group search and frame disposition: three host-side waits shortened
The first arm's space-group search now runs beside the wait for the all-observation merge and the search started on it, instead of after both. Classifying a reflection for a candidate group asks gemmi twice which operators fix it (once for the absence, once for the epsilon factor); one pass over the operators answers both, with the absence read off a fixing operator's own translation, which is what gemmi's test reduces to once the centring has been checked. The delta-CC1/2 frame disposition builds its per-reflection totals on all threads, each taking a range of reflections and walking every observation so the sums keep their order, and measures up to sixteen candidate chains side by side with a worker's scratch made on first use. Same numbers, same bits; only the waiting changed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Fable 5.1
|
bbcae8ae7f |
RotationScaleMerge: the resident ingest uploads the reflections once
The device ingest staged every reflection to the card twice - once for what the sort and the cut read, once again for the observations the cut kept - and the second staging was a full copy of the reflections into pinned memory and over the bus. Where the whole set of reflections and the ingest's own peak beside it fit a quarter of the card (decided on the card's total memory, so the answer depends on the card and not on the moment), the first upload now goes into one resident copy, and the observations are taken from it in a single extract; the second staging stays as the fallback where the copy does not fit, and where the card reads host memory in place. The extract kernel also hands over the predicted positions, which the host had gathered itself from a downloaded index array, and the two pinned staging buffers are kept across the ingest instead of being allocated per staging. Same observations, same order, same bits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Fable 5.1
|
00f8c8ba4b |
Refinement: the final ladder takes the rungs the step from the best state tried already; the longest trials to the slots first
The trajectory ends where a step gains nothing, and that step ran from the best state at the weights the final ladder then tries from the same state: on a positional step every rung, on a B step the middle one. A trial of the same weights from the same model is the same computation (RunTrials: the outcome is the same on any slot), so the final takes those outcomes from the step and runs only the rungs it has not got. The maps' Refit() then starts from the fit the main slot would hold had it run its rung (Context::Restore of that rung's fit) - the solvent grid and the likelihood's D it starts from otherwise depend on which trial the main slot ran last, and that showed as last-bit differences in the map coefficients. RunTrials gives the trials that refine B, the longest, to the slots first: on a machine whose cards hold a slot each the ladder's two B rungs no longer share one card while a positional rung has one of its own. Byte-identical (p.mtz, p.hkl, p.cif, p_model, p_refined_model, p_refined_maps.mtz, p_blobs.json, report minus timing) on three rotation sets run live-path, and on two of them in --mode refine. Refinement wall on one RTX 5080: 1.86 -> 1.71 s, 2.42 -> 2.17 s, 50.0 -> 40.8 s (one stage of the trajectory each). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
5338e1b535 |
Centre-of-symmetry statistics and anisotropy: hashed lookups and the simulation and cell assignment on all threads, output unchanged
AnalyzeCentreOfSymmetry ran on one thread between the adopted group's merge and the analyses: its orbit fold and L-test partner lookup were std::map (now hashed - the orbits are sorted by a total order right after, and the lookups only test membership), and its 20 simulated populations took their log/cos per reflection serially. The generator's draws are now taken in the same order on one thread and turned into intensities on all of them (the Box-Muller operands in the order GCC evaluated them; bit-exact against the old function on five groups at 1/4/32 threads). AnalyzeAnisotropy's three BuildCells and the observation cells of the systematic floor find each copy's nearest of 60 directions on all threads and fill the cells in the old order; the observation sort is a ParallelSort (index-unique pairs, a total order). AnisotropyRunInfo carries the worker count. Live path, workstation, 3 A/B pairs, outputs identical (mtz/hkl/cif/model/maps/report): thau centre stats 0.18 -> 0.06 s, analyses 0.57 -> 0.43 s; cytc 0.04 -> 0.016 s, 0.37 -> 0.32 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_fandClaude Opus 5.5
|
7c49fefb54 |
Tail: the cell e.s.d.s and the unmerged MTZ table beside the refinement; the refined difference and anomalous maps side by side
The cell e.s.d.s and the unmerged MTZ read only the integrated observations and the group, both final once the model has been read, and the refinement reads neither: both now start before the refinement instead of after it. In MakeMaps the difference map is made on its own thread while the anomalous map is made. Outputs identical (MTZ data, CIF/HKL/INS, refined model, maps, blobs, report minus provenance) on the thaumatin and cytochrome C live runs. Live path without the unmerged MTZ (as the broker runs it), step Refinement: thaumatin 2.76 -> 2.40 s, cytochrome C 1.99 -> 1.93 s. With the unmerged MTZ (CLI), Refinement + Writing files: 3.21 -> 2.67 s and 3.33 -> 2.80 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu |
||
|
leonarski_f
|
2e296e6bed |
jfjoch_writer: the live refinement's model, maps and blobs are written beside the master file
In broker live mode the refined model, its map coefficients and the difference-map blobs stayed in the broker for the viewer routes and were lost at the next run. They now go to the writer in the END message as the placed model and maps already do - EndMessage::refined_model (PlacedModelMessage), refined_map_coefficients (MtzTable) and blobs (DensityBlobsMessage, one vector per field) - and the writer writes <prefix>_refined_model.cif (+ .pdb), <prefix>_refined_maps.mtz and <prefix>_blobs.json, the names the CLI gives them. The blobs' JSON is made by one function (DensityBlobsJson) that the CLI now writes through as well, so the two files are byte-identical. Optional CBOR fields, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c65220d96a96d2b1897f3c6632c94f9379b670a3) |
||
|
leonarski_f
|
13bd9db55c |
Refinement: a B factor is held at its bound by the whole target's gradient, restraints included
The held test read the X-ray gradient alone. At the floor, an atom whose X-ray term pushed outward but whose restraints pulled it back in more strongly was held fixed against the total target, and one whose restraints pushed it out was handed a gradient along the direction Set() clamps away - a flat direction for L-BFGS, failed line searches and s.y <= 0 pairs. On the 14 parity sets (--mode refine, GPU) the refined models are md5-identical; on the four 5f6m MD starts R-free 0.1707/0.1676/0.1720/0.1669 -> 0.1707/0.1690/0.1719/0.1668 (median 0.1692 -> 0.1699). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2f4b0b5ea54fdac54de7ee6c5a1476d2fc019fd8) |
||
|
leonarski_f
|
2be88e11e3 |
Refinement restraints: contacts with lattice translates are kept, and an atom's own symmetry contact once
Contacts() dropped every pair found in image 0 with a nonzero lattice shift ("the identity term
covers the contact"), but that pair is a contact with a translated copy of j, not with j itself:
in P1 no intermolecular repulsion or B-contact restraint was made at all, and in every group the
contacts along a, b and c were missing (MeasureGeometry's clash count with them). Such a pair is
now kept like a symmetry mate's, from the lower atom of the two.
An atom against its own image x -> T x is also met through T^-1 at the same distance (another
operator, or the same one at another lattice shift for a screw), and both were kept: atoms near
a 3-, 4- or 6-fold or a screw axis were repelled twice. For i == j the image first in fractional
coordinates of T x and T^-1 x is kept; an involution is kept once as before.
Test: in P1 two waters 2.4 A apart across a cell face get one repulsion and one B contact (none
before); a water 1.4 A from a 3-fold axis gets one repulsion with its images (two before).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 0d6e4d0491e655a49d59b709e1398843506d162a)
|
||
|
leonarski_f
|
470fc4e479 |
ffbidx (vendored): INDEXER_GPU_DEVICE is not read, the indexer stays on the thread's pinned GPU
The environment variable overrode the current device that ScopedGpuPin sets, so a stray value
moved an indexing thread to another card. Recorded in the file header as a local modification,
together with the deterministic tie-break of
|
||
|
leonarski_f
|
257bf2fe2b |
Refinement: trial slots counted with their scale pool and both restraint sets; on the CPU no more than -N
GPU: a slot was packed onto a card by its engine's bytes alone. The per-slot estimate now adds the slot's ModelScaleGPUPool with the robust fit's buffers (ModelScaleGPUPool::DeviceBytes(n, 1, true)), and RefinementGPUEngine::DeviceBytes counts the two restraint sets SetRestraints() holds, not one. The 80%-of-card budget is unchanged, so where these do not fit fewer slots are packed instead of an allocation failing inside a running trial. CPU: OwnSlotDevices made a slot per wanted trial whatever the thread count, so -N 1 ran six trial threads with six models, topologies and engines in memory. The main slot and those beside it are now at most -N. Side by side equals sequential, so no result changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3c04c01158266c267f8e5ffaaaa98d6d8fd2a747) |
||
|
leonarski_f
|
3d382ce0a0 |
Refinement: an exception in a trial slot or in RefineModel no longer ends the run
A slot builder thread that throws (gemmi's topology or hydrogen placement, bad_alloc) would call std::terminate - rugnux or the broker gone. The slot is now left out as one the card cannot take is, with a warning, and MakeOwnSlot's GPU engine catches std::exception, not only JFJochException. Both callers of RefineModel (after the merge, and --mode refine) catch std::exception: the refinement is reported NOT_PERFORMED with "refinement failed: <what>" and the run writes everything else (cell e.s.d.s, unmerged MTZ, report), as the refinement is processing-inert. Test: --mode refine on a cell too large for the bulk-solvent grid comes back with REFINE= NOT_PERFORMED and the reason, not an exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc4e99fe4b3da2a9a3fa4290b5f70bfcc8c7bdb8) |
||
|
leonarski_f
|
48bf9ebe3a |
CellEsd: PI from JFJochMath.h instead of M_PI, and the Herbstein (2000) credit at the block e.s.d.s
M_PI is not defined by MSVC without _USE_MATH_DEFINES, and CellEsd.cpp is reachable from the viewer build. The citation, so far only in CellEsd.h, now also sits at the algorithm it credits. The other files new in rc180 under common/, image_analysis/, reader/, writer/, rugnux/ and viewer/ use no M_* constants or POSIX-only calls (grep of the rc180 diff). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a019bc5fd2084907af9379068af3265ffe6c9ef5) |
||
|
leonarski_f
|
d8c14046b2 |
rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report
Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated
observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed
centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal
metric, so the components of G* the point group leaves free (projected from the group's rotation
parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the
GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal
e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000),
so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second
estimate; the written e.s.d. is the larger of the two per parameter. Stills get none.
Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the
group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder
organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where
fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals;
report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS,
CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT).
On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets
written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated
.ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k
lysozyme observations.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)
|
||
|
leonarski_f
|
8a123e794d |
Stills: the run's rocking curve goes to _process.h5 and --mode scale merges with it; MeasureRunWidth takes the window at each reflection's resolution
--mode scale re-measured the rocking width on the stored reflections, so on any file whose frames were predicted through the run's curve it merged with a different curve than the mx run did. The curve the post-refinement used (formula, table, radial scale) is now carried in the EndMessage (StillsRunWidth) and written to the master: /entry/MX/stillsRockingWidthFormula [gamma0 1/A, gamma_e rad, bandwidth sigma], /entry/MX/stillsRadialScale, /entry/MX/stillsRockingWidthTableDstar and /entry/MX/stillsRockingWidthTable (1/A). The reader returns it, and --mode scale passes it to the post-refinement. A file without it is re-measured as before. MeasureRunWidth grouped each frame's reflections by the frame's window, taken as the largest excitation error it predicted. With a window that follows sigma(d*) that is the high-resolution window, applied to the frame's low-resolution reflections too. Given the curve the frames were predicted through, the window is now that curve at the reflection's own resolution; without one, the frame's largest excitation error as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 992343d0206489991880bb927efc79131134a1d9) |
||
|
leonarski_f
|
00662c4110 |
Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records
The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28) |
||
|
leonarski_f
|
e406eb17ea |
docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits
- New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10) |
||
|
leonarski_f
|
4dbcb35708 |
Stills partiality: the rocking width measured in shells of equal width in d*, used as a table
MeasureRunWidth measured the width in eight shells of equal count in d*^2 and fitted the three-term formula gamma0^2 + (gamma_e d*)^2 + (bw |q_z|)^2 to them. On runs predicting ~1000 reflections a frame the equal-count shells all sat at 3.8-1.8 A, so the low-resolution width was the formula's extrapolation, and the fit's form flipped with the number of frames (to a pure bandwidth at 50k frames). The same window-group moment is now also measured in eight shells of equal width in d*, made non-decreasing (pool adjacent violators, weighted by each shell's summed signal) and kept in RunWidth as a table that the partiality interpolates linearly in d*. The formula is still fitted on the equal-count shells and still sets the prediction window. With fewer than three measurable shells the formula is used as before. Still battery, 12 non-float sets, together with the sigma-following prediction window (the previous commit; run 20261026-1021_1b28d6_still-part-comb against rc180 20261026-0857_3882b0_still-part-base1, 0-cycle REFMAC R-free vs deposited, both --no-refine): 6 better, 3 unchanged, 3 worse (by 0.002), sum of changes -0.027 (the window alone: -0.007); largest gains pal90 -0.014, alvra103_int16 -0.011, lcls222 -0.005; largest losses euxfel152 +0.005, alvra213_int16 +0.005 (+0.010 with the window alone), euxfel83 +0.003. Win/draw/loss against the depositions unchanged at 9/2/1. Rotation: p.hkl byte-identical on 5reo, lyso_x06da_ref and thau_x10sa_16keV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit de779bb10b1813eeae625e3994b1ca73d6079d7a) |
||
|
leonarski_f
|
0829b9a72d |
Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale
Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its indexed spots): one width at every resolution, differing eightfold between frames. Against the run's measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated). Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way, scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run. Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited): 9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall +13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three rotation sets. Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc) |
||
|
leonarski_f
|
13da79b788 |
Refinement GPU: the atom gradient kernel walks its box without integer division
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m0s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m28s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 18m17s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m17s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 23m45s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 23m41s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 27m32s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 28m36s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 29m38s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 31m39s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 24m10s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 23m51s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 28m4s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 29m44s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 23m45s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 25m39s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 28m32s
Build Packages / Unit tests (push) Successful in 1h35m16s
Build Packages / Validation Battery (push) Successful in 41m34s
Bitwise. One warp per atom, each lane taking the box's points t = lane, lane + 32, ...: every point cost three integer divisions/modulos to turn t into (u, v, w) and three more (imod) to wrap them into the grid. The lane now steps (iu, iv, iw) on by 32 points with adds and carries, and wraps from the box's first point, wrapped once per atom (a box is narrower than the cell, so one conditional subtraction per axis). The same points in the same order per lane, the same arithmetic. atom_gradient_kernel (nsys, side by side): 5reo 426 -> 237 us, 5f6m 887 -> 523, 9yl4 1436 -> 802, 7d98 692 -> 388 us per launch. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Wall: 5reo 1.43 -> 1.35, 5f6m 2.20 -> 2.00, 9yl4 6.88 -> 6.82, 7d98 3.58 -> 3.38, 5f6m MD start 4.16 -> 3.71 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6554e9dcbe4b448eda582eb76d6a24c59fe6ab0c) |
||
|
leonarski_f
|
4a8e13ddfa |
Refinement GPU engine: its waits on the stream asleep (blocking-sync event) rather than spinning
Bitwise. cudaStreamSynchronize spins a core for the whole wait; the weight trials' engines wait on a shared card for most of a run, so about half of the user cycles of a refinement were the spin (PROFILE.md: 38-50%). A StreamWaiter (CUDAMemHelpers.h: an event made with cudaEventBlockingSync, recorded and waited for) now takes the refinement engine's waits - evaluation, Fcalc, Fmask, the uploads of positions, row model, blur and restraints. The short waits of the host-driven loops stay spinning: the scale fit's Levenberg-Marquardt (one launch of 66-120 us per step) and the gather's pair count; making those block too cost 2-3% of wall (wake-up latency), here the wall is unchanged. No process-wide cudaDeviceScheduleBlockingSync: that would change every GPU wait of the broker. CPU use (-N 8, time -v): 5reo 301-315% -> 190%, 5f6m 326% -> 181%, 9yl4 333% -> 222-228%, 7d98 307% -> 181%, 5f6m MD start 309-320% -> 151%. Wall, two interleaved runs each: 5reo 1.43-1.52 -> 1.43, 5f6m 2.19-2.20 -> 2.19-2.20, 9yl4 6.83-6.84 -> 6.88-6.89, 7d98 3.50-3.51 -> 3.58, 5f6m MD start 4.16-4.34 -> 4.16 s. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d23d2cf7bb99b30617a3f57c531b03db8bc03e05) |
||
|
leonarski_f
|
f34d2f4cca |
Refinement: the weight trials' own slots made while the placement runs
Bitwise. MakeOwnSlot copies the model as it was before the topology and reads only what the placement does not change (cell, reflections, the engine's kind, the B ceiling), so the slots are now built on their threads right after the start score, beside the rigid-body and group-B placement, and joined where they were built before (also where the placement throws). The "slots" stage goes from 0.23 s to 0 on 9yl4; the placement is not slowed (0.59 -> 0.60 s). Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU); side by side = one at a time. Wall: 5reo 1.46-1.53 -> 1.54, 5f6m 2.26 -> 2.21, 9yl4 7.03 -> 6.83, 7d98 3.65 -> 3.56, 5f6m MD start 4.20 -> 4.15 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0752a921a4293d4213894e3242c4ef36d8ce2a05) |
||
|
leonarski_f
|
1f8b5b1986 |
Refinement: a Refit's solvent grid starts around where the context's last one ended
Logic change (deterministic), kept on evidence. Every Refit searched the whole k_sol/b_sol box - an 88-point coarse pass of plain least squares, then the robust fine pass around its winner. ModelScaleBox (and ModelScaleGPUPool::Fit / ModelScaleGPU::FitSolvent) take an optional start; with one, the coarse pass is skipped and the fine pass is laid around it. Refinement passes the grid winner of the context's previous Refit (before MoveOverallB moves the overall B into b_sol); it travels with a trial's fit, and a trial's first Refit without one (Context::ForgetStarts, which also clears the likelihood's D start) searches the whole box, so a trial still depends on nothing its slot ran before (side by side = one at a time, md5). Evidence (GPU, -N 8): full refinement of the 14 parity sets from their placed models - 12 refined CIFs identical (md5), 9ig7 and 6iu9 differ, final R-work/R-free equal to 4 digits but 6iu9 R-work 0.2911 -> 0.2910; of the 5 timing cases 4 identical, 7d98 (cif start) R-free 0.3096 -> 0.3095; 5f6m MD starts 01-04 median R-free 0.1692 -> 0.1692; 6o2h on the CPU engine identical (md5). The score-only parity table is not touched (one Refit, no start). refit.scale (TIMING sum): 5f6m 1.52 -> 1.10 s. Wall: 5reo 1.68 -> 1.53, 5f6m 2.43 -> 2.24, 9yl4 7.19 -> 7.02, 7d98 3.70 -> 3.65, 5f6m MD start 4.57 -> 4.22 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff285fb525ecf4b480698d4aafe212a34301a8fe) |
||
|
leonarski_f
|
1f1aa18b67 |
ModelScaleGPU: the robust fit's median of the residuals on the device
Bitwise. The first robust round copied every fine-pass fit's residuals to the host (up to 25 x n floats, 9 MB at 92k reflections) and took std::nth_element of each there. The median is now a radix selection on the card (one block per fit, four byte-wide passes of integer counts over the float bits - the residuals are not negative, so the bits order as the values): the element nth_element puts at n / 2, the same value; only the fits' c come back. FitSolvent and RobustBatch lose the host threads they took for it. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start. Wall (-N 8, quiet): 5reo 1.72 -> 1.68, 5f6m 2.57 -> 2.43, 9yl4 7.28 -> 7.19, 7d98 3.78 -> 3.70, 5f6m MD start 4.76 -> 4.57 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc53896c335fdabdf4d417323038e05b8b63535d) |
||
|
leonarski_f
|
ca0a6dd343 |
Refinement: each slot's likelihood fit on all of the run's threads
Bitwise (the bins are fitted independently: the same bits at any thread count). The weight-trial slots Refit in lockstep while the GPU idles, each with its 1/slots share of -N for the ML bins; the bins now go to the shared worker pool with -N tasks per slot, so the threads of slots that are not fitting help the ones that are. After the Newton searches (previous commit) the ML fit is 0.1-0.5 s summed over a run, so the wall gain is within noise: 5reo 1.63-1.75 -> 1.72, 5f6m 2.60 -> 2.57, 9yl4 7.34 -> 7.28, 7d98 3.78 -> 3.78, 5f6m MD start 4.82 -> 4.76 s. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 56815f5390cc01e8da5d420fff73cdf4394c385c) |
||
|
leonarski_f
|
2c1df0ebfa |
Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's
Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves 6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a trial depends on nothing its slot ran before (side by side = one at a time, md5). 14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186, D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s, 5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60, 9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free 0.1693 -> 0.1692. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44) |
||
|
leonarski_f
|
a8d3a8dbc2 |
Refinement: the mask grid sized without allocating it; the L-BFGS pairs keep their s.y
BulkSolventMask::MaskGrid, documented as "sizes and spacing set, values not allocated", allocated and zeroed the whole fine mask grid (up to 2^26 floats) - in BulkSolventMask's constructor and in every GPU engine's, i.e. once more for each weight-trial slot, only for the grid's size. It now takes good_grid_size() as set_size_from_spacing() does, and the CPU engine's Fmask() allocates its copy. rugnux_refine_test, RTX 5080, -N 8: 9yl4 8.6 -> 7.8 s, 7d98 4.85 -> 4.0 s (engine 0.66 -> 0.16 s, slots 0.60 -> 0.23 s on 9yl4); the high-resolution sets, whose mask grid is small, are unchanged. RefinementMinimiser keeps s.y of every pair it stores and Direction() takes rho and gamma from it instead of two more dot products per pair per call (the same Dot() of the same vectors): Direction's CPU time -22% on 9yl4; no wall change with the trials side by side, where other slots fill the card meanwhile. Bitwise: refined CIF identical on 5reo, 5f6m, 9yl4, 7d98, a 5f6m MD start (GPU) and 6o2h (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6f5caf5e1b1183a752acc5cca78bfb7ef002b12a) |
||
|
leonarski_f
|
3882b0f062 |
Serial stills with a reference or model: crystals their own cells anchor keep that indexing
The per-image reference/model choice of indexing ignored the crystals' refined cells. On a pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as indexed than after a twin law, the cells decide that law - the rule the data-only resolution already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets the reference choose only among the indexings left open. Exactly merohedral lattices (no metric signal) are unaffected. Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets): pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16 0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged; lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against cells that decide all three laws (96/96/100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f) |
||
|
leonarski_f
|
3b06bcee78 |
Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win
Three changes to how the trajectory of commit 2bfb84bc4 moves and ends: - A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking on while R-work falls. - Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6). - The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds, R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement that on deposited models cost R-free. Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016 (e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this: 5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 -> 0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857) |
||
|
leonarski_f
|
bda90dbdd5 |
Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms
- B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144) |
||
|
leonarski_f
|
6369a263b1 |
Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start
Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519) |
||
|
leonarski_f
|
4424d8884d |
Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature
Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825) |
||
|
leonarski_f
|
626afc49dd |
Refinement likelihood: the searches over D alone with the variance taken once
EstimateLikelihoodParameters' golden-section searches over D at a fixed beta - four per bin on the free reflections, and the one on the working reflections the D/FOM change added, 50 evaluations over 95 % of the data - evaluated -ln L reading every row through the index and taking log(var) at every evaluation. The rows are now gathered per search with their variance and its logarithm; the sum is the same expressions in the same order, so D, beta and everything after are the same to the bit (refined model and report identical on the seven timing sets). refit.ml 200 -> 140 ms per cycle on the 1.1 A set, one thread per trial slot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 23ad3bfc254d10af310f17dbb8f85616f666a783) |
||
|
leonarski_f
|
6075239be0 |
Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them
The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec) |
||
|
leonarski_f
|
bcb71136ab |
Refinement restraints: metal ions bound to their donors by the model's LINK records
Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68) |
||
|
leonarski_f
|
9da3851844 |
Refinement: at most n weight trials at a time, and a cancel flag the caller can set
- RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c) |
||
|
leonarski_f
|
8718065a7c |
Refinement GPU: the bulk-solvent mask on its own fine grid on the card
The score draws the mask on a grid of its own (min(d_min/3, 0.4 A), BulkSolventMask) on the host; on a GPU engine that was a CPU mask and transform every cycle, per trial slot on one thread. Now the engine masks on that grid itself: - RefinementGPUSetup::mask_grid (BulkSolventMask::MaskGrid, the grid the host mask uses); where it is not the structure factors' grid the engine reserves a grid, a spectrum and an R2C plan for it (counted in DeviceBytes and the FFT work area), and Fmask masks (ModelMaskGPU), transforms and reads the rows there. Refit takes the engine's Fmask on a GPU, BulkSolventMask on the CPU. - ModelMaskGPU's shrink: each edge point marks its stencil as solvent instead of every macromolecule point searching its stencil for an edge - the same idempotent stores to the same points, so the same mask to the bit (ModelMaskGPU_* tests unchanged), and a thin shell of points visited instead of half the grid. - BulkSolventMask finds its symmetry orbits at its first mask: a GPU refinement never draws one on the host (1.1 s of trial-slot setup on the 3.6 A set). The card's mask is gemmi's but at points exactly on an atom's radius (ModelMaskGPU.h): R-work/R-free of the 14 parity sets on the GPU are the CPU engine's to 4 digits. Refinement deterministic run to run on the GPU (7 timing sets, refined model and report identical over 5 runs of successive builds that did not change results). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 89b5d00907c9ad99e8b389f6272a3724f981cd10) |
||
|
leonarski_f
|
46ad2c10ed |
Refinement GPU: the robust scale fit on the card, the pool back in every trial slot
The score's robust fine pass (pseudo-Huber IRLS of ModelScaleBox::robust) on the device, so that refinement on a GPU no longer fits its scale on the host every cycle - with the trials side by side that was a robust grid search on one thread per slot (13-133 s per refinement on the seven timing sets, against 1.7-8 s before the score commit). - ModelScaleGPU: the weights multiply the WSSR and J^T J / J^T r in scale_sums (a weight row per fit, reserved at the first robust fit); robust_residuals / robust_weights kernels; the median of each fit's first residuals taken on the host as FitModelScale takes it (nth_element at n/2, on the fits' threads); LevMarBatch split out of FitBatch so that the rounds restart from where a fit stands. FitSolvent(..., robust) refits the coarse winner in the fine pass, as FitModelScale's robust fit does; the plain fit is unchanged. - ModelScaleGPUPool::Fit(..., robust); its streams at the highest priority: a fit is a chain of small launches the host waits on, and on a card shared with the trials' long kernels each wait queued behind them (refit.scale 58-87 ms -> 14-45 ms per cycle, the result the same to the bit). - Refinement: the main context and every trial slot fit on their pool (robust); the CPU engine keeps FitModelScale's robust fit. The card's fit is FitModelScale's to rounding (float residuals and weights, the steps' anisotropic factor in float), deterministic run to run and whichever engine a fit takes: R-work/R-free of the 14 parity sets on the GPU equal the CPU engine's to 4 digits (largest difference 1.4e-6 on the 3.6 A set). Test: ModelScaleGPUPool_RobustMatchesFitModelScale (one reflection in fifty a 4x outlier, five cells). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d15f048cbd7ed2d6b430f9f9cb5018ca284da8f0) |
||
|
leonarski_f
|
c4d2d419ce |
Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles
Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8) |
||
|
leonarski_f
|
8933a72a89 |
Refinement: the weight trials side by side, each on a model, topology and engine of its own
Every rung of the weight ladder starts from the same model and depends on no other, so the six position trials, and then the three B trials at the chosen weight, run at the same time: on the main model and engine and on up to five slots beside it, each a copy of the model before the topology was made (the topology is built again on it, hydrogens included) with its own engine. The slots go on the GPUs in turn from the main engine's, as many as each card's budget takes, so a machine with several cards spreads them; a slot whose engine cannot be made is left out, and with none the trials run one after another on the main engine as before. The threads are shared out among the slots. The log lines and the choice are made afterwards in ladder order, so the log, the trial table and the refined model are the same, bit for bit (GPU and CPU engines checked). 5reo 3.9 -> 2.5 s, 9yl4 13.4 -> 7.8 s, 6o2h 3.1 -> 1.9 s, 8dyz 4.6 -> 2.7 s, 9rcs 2.9 -> 2.2 s (one RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0ea6a6db8aba640afc5249fe748922ddfaa2537a) (cherry picked from commit 0c0176c8bf571ed26f8a68a9de40222487c6200e) |
||
|
leonarski_f
|
5569f3d1f2 |
Refinement GPU: the X-ray target and the restraints in one submission with one wait
RefinementEngine::EvaluateWithRestraints evaluates both targets at the model's atoms. On the GPU the X-ray chain is queued first, the restraints' coordinates are written into a pinned buffer while the device works on it, the restraint kernels follow on the same stream, and the host waits once; the sums and gradients come back through pinned buffers. Same kernels on the same inputs, same bits. The CPU engine keeps calling the two in turn. 9yl4: evaluation 5.1 -> 4.7 ms, refinement 14.2 -> 13.4 s; 5reo 4.0 -> 3.9 s (RTX 5080, -N 8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51434b98b62a9e6322ffe811bf272a45ff6ce003) (cherry picked from commit 0bcfa3f8041cec4bb4bc376b8559f30c36d3ac80) |
||
|
leonarski_f
|
001e61c1a4 |
Refinement: the repulsion search on the refinement's threads, hydrogens skipped before the neighbour walk
BuildRepulsions walked the neighbours of every atom and only then dropped the pairs of a riding hydrogen; those atoms are now skipped before the walk. The atoms are searched in 64 blocks under ParallelFor and the blocks' pairs joined in atom order, so the restraint list is the same as before for any thread count. The model is no longer copied for the NeighborSearch. Same bits. 9yl4: repulsions 3.37 -> 1.09 s (91 calls), refinement 17.7 -> 14.2 s; 5reo 0.38 -> 0.12 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ded11388a18f5a4138a73150c4bb073570152727) (cherry picked from commit 9c96836a90c9f170008d343af2cd885a4e59b321) |
||
|
leonarski_f
|
e3782cabe4 |
Refinement GPU: atom densities uploaded only when some B moved, one device copy, positions through a pinned buffer
Every evaluation uploaded both copies of the atoms' densities (the gather's and the gradient's) and their boxes, although in a positions-only minimisation no B changes between two Refits. The engine now keeps one copy (the gradient kernel reads ModelDensityGPU's atoms and boxes), uploads it only where some atom's density was made again, and sends the positions from a pinned buffer without waiting. The densities that are made again (every atom in a B minimisation) and the grid positions are worked out on the refinement's threads. Same bits. 9yl4: upload 3.29 -> 0.56 s, host side 1.29 -> 0.61 s, refinement 22.3 -> 17.7 s; 5reo 4.8 -> 4.4 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 46c3b67c91aee0aec3473b0fb1a4945c00bda070) (cherry picked from commit 0aebc98110c8155932af647d6204891aec662294) |
||
|
leonarski_f
|
dd49d4e2f2 |
Refinement: the likelihood's D and beta fitted bin by bin on the refinement's threads
EstimateLikelihoodParameters fits each resolution bin on its own and writes only that bin's values, so the bins run under ParallelFor; the result is the same, bit for bit. It was the largest single stage of a refinement (107 calls, one core): 5reo 2.42 -> 0.56 s, 8dyz 2.46 -> 0.57 s, 9yl4 3.54 -> 0.88 s; whole refinement 5reo 6.3 -> 4.8 s, 8dyz 6.9 -> 5.3 s (RTX 5080, -N 8, loaded box). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85aa34d3031c8e36be4462fca544bc801af45d5a) (cherry picked from commit d84ba55252adf455d5b4ba62052dfbafa6290a64) |
||
|
leonarski_f
|
24b1711d64 |
Refinement: time per stage - an NVTX range per call and one TIMING line per stage with its sum and calls
RefinementStage sums the time of each stage over a refinement (Refit and its parts, the evaluations and their X-ray, restraint, host and upload shares, the repulsion build, the riding hydrogens, the rungs) and logs it at debug level at the end; rugnux_refine_test -v shows it. No change to any result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9491f54e0beccf36c65ebcec9afff90d08485f86) (cherry picked from commit 0ce206a88df960754476ce2f75ffaed90fbf3123) |
||
|
leonarski_f
|
07ae590e54 |
Serial stills: the indexing ambiguity resolved from the data when no reference or model is given
Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without -z or --model used to merge the crystals' random indexings together (and warned). The crystals are now made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities, weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent, four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest crystals. Deterministic (fixed summation order, sequential passes). A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals' own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as indexed, those crystals are anchors and only the others are decided by the intensities. Without this, two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350). With a reference or model the per-image choice stands; the same resolution runs as a check, reported as AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS, AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev). --mode scale is not changed. Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with --model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213 0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t 0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes 0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36) |
||
|
leonarski_f
|
c982ba4f65 |
FFBIDX on the GPU: the same spots index to the same cells, whatever order the thread blocks finish in
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m27s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 10m19s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 19m23s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m8s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 23m47s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 24m24s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 27m33s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m33s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 29m21s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 30m3s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 23m43s
Build Packages / Generate python client (push) Successful in 27s
Build Packages / Build documentation (push) Successful in 1m19s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 24m49s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 30m37s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m42s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 28m43s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 32m54s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 35m41s
Build Packages / Unit tests (push) Successful in 2h7m57s
Build Packages / Validation Battery (push) Successful in 38m34s
Each block of the candidate-vector and the cell search sorts its own candidates and merges them into one shared top list under a lock, in the order the blocks happen to get there. The merge compared values only, so where two candidates scored the same - common on the 30-spot seed the stills path tries first, since the score is a sum of trimmed terms - the place went to whichever block merged first. The same frame then indexed to a slightly different lattice, or not at all, from one run to the next. Ties are now broken by the candidate's identity - the sample index for a vector; the cell vector (with its flip), candidate and rotation sample for a cell - which makes the order total, so the merged lists are the top-N of everything whatever the merge order. The blocks' own lists are already in that order: the block radix sort is stable and its samples ascend with the thread index. Measured on a 100k-frame serial set: 4-21 of ~33k indexed frames came out with a different lattice in every pair of runs, which the stills scaling and partiality chain turned into 0.97-0.99 CC1/2 and thousands of observations in or out of the merge. With the fix, three runs at the default thread count and one at -N 8 give the md5-identical merged .hkl; a per-frame dump (spot list, indexer answer, refined lattice) of 20k frames is identical across three runs where it differed in 4-6 frames before. FFBIDXIndexerCPU_MatchesGPU and Portable_FFBIDXIndexerCPU_SyntheticStill pass. No regression test: a repeat-and-compare test on the library's spot lists, threaded, passed before the fix too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 9cff12d652088851b81f6a353ac52aa8318799f2) |
||
|
leonarski_f
|
64bbc44e11 |
fast-feedback-indexer vendored in image_analysis/indexing/fast-feedback-indexer instead of fetched
The GPU known-cell indexer behind FFBIDX is now built from an in-tree copy of upstream commit
a94fd4f5c55289e70860feea358182d2072e02b5 (the archive the FetchContent block pinned), so it can carry
local fixes. No behaviour change.
Copied: indexer/src/{indexer.cpp, indexer_gpu.cu, log.cpp} and the ffbidx/ headers the build uses
(indexer.h, indexer_gpu.h, refine.h, log.h, exception.h, envvar.h), each with a header line naming the
upstream URL, commit and licence; LICENSE.md beside them; and the library's test spot lists
(data/simple/files/image{0..8}_{local,peakfinder8,radial}.txt) that FFBIDXIndexerCPU_MatchesGPU reads,
now found in-tree. ffbidx/version.h is written once ("1.1.3-a94fd4f") instead of generated - the
generator ran git in the parent build, so it used to stamp Jungfraujoch's own hash.
The in-tree target keeps the name fast_indexer_static and what upstream's indexer/src/CMakeLists.txt
gives the static library: the source list, CUDA_RUNTIME_LIBRARY Static, cxx_std_17 (PUBLIC), the source
directory as PUBLIC include, CUDA::cudart_static and Threads::Threads (PRIVATE), Eigen3::Eigen (PUBLIC,
the project's), the CUDA-architecture default only where none is set, and -Xcompiler=/Zc:preprocessor for
the CUDA sources under MSVC (CUB needs the conforming preprocessor). Left out: the shared library, the C
API (c_api.cpp, unused), install/pkg-config rules, the version_txt generator target, the top-level
project's data/tests/python/examples subdirectories, and -DVECTOR_CANDIDATE_REFINEMENT=1 (no source
reads it).
Licences: COLLECT.sh takes the text from the vendored LICENSE.md; the THIRD_PARTY_NOTICES row moves from
the fetched to the vendored table. CLAUDE.md's list of FetchContent downloads no longer names it.
Verified: clean configure and build of rugnux and jfjoch_test; FFBIDXIndexerCPU_MatchesGPU and the
Portable_FFBIDXIndexerCPU tests pass; a 3000-image stills run gives the md5-identical merged .hkl with
the fetched and the vendored library.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 0e4fcb87de0fbb7c0022750ff4d62d8c5a5c3c18)
|
||
|
leonarski_f
|
cf7676f5d5 |
Stills merge: outlier rejection on by default (10 sigma), the band linear in the error model's sigma
The stills merge had no working outlier rejection. --reject-outliers was 0 for stills, and when set its band took the systematic part as a factor exp(+-n s), s the spread of ln(I / median) - on stills a reflection's observations are partials and per-crystal scales, many near zero, so s is wide and the band let through observations hundreds of sigma out. On a high-multiplicity broad-bandpass run with a spotty powder contaminant, single grains in a weak reflection's box were that reflection's whole merged intensity (half-set means -2 vs 2284 on 141 observations); the shells holding the powder rings and the lowest shell merged at CC1/2 0.6-0.75 beside 0.99 elsewhere, at multiplicity ~400-1000. Stills now test the plain linear band, n sqrt(a sigma^2 + (b <I>)^2) about the median - the error model's b is the spread as it is - at n = 10 by default (rotation keeps 6 and its multiplicative band). The stills merge statistics now report the rejected count (OBSERVATIONS_REJECTED was always 0 there). Measured (REFMAC 0-cycle R-free of an isomorphous room-temperature model, depositor's free set, scale mode on stored integrations, vs the same code with --reject-outliers 0): - broad-bandpass in-house stills, 50k frames: 0.318 -> 0.265; 20k: 0.274 -> 0.244; R_split 0.29 -> 0.06, CC1/2 0.79 -> 0.98. 0.1% of the observations are rejected. - monochromatic in-house stills, 30k frames: 0.216 -> 0.212; R_split 0.126 -> 0.087. - n = 6 gains ~0.003 more there but costs the public XFEL sets 0.003-0.006 on three of four; 20 keeps most of the gain. Still battery, 12 non-float sets, against rc180: R_split better on all 12; R-free change mean -0.0005 (range -0.006..+0.003, plus pal86 +0.009 that is within that set's run-to-run spread of ~0.007 - a rerun of rc180's behaviour gave 0.192 against the baseline's 0.185); one draw becomes a win (lcls242_s), nothing else changes class. Rotation is untouched (the stills band and default are gated on the experiment not being a rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3a7142801056f21c720956255f1207e0bf1a8f43) |