Commit Graph
1220 Commits
Author SHA1 Message Date
leonarski_f 825b88c19d CHANGELOG: 1.0.0-rc.180 entry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 339c6cbaa4177964e5eb06d0878efb83d1da2dbc)
2026-10-26 15:39:07 +01:00
leonarski_f 13bd9db55c Refinement: a B factor is held at its bound by the whole target's gradient, restraints included
The held test read the X-ray gradient alone. At the floor, an atom whose X-ray term pushed
outward but whose restraints pulled it back in more strongly was held fixed against the total
target, and one whose restraints pushed it out was handed a gradient along the direction Set()
clamps away - a flat direction for L-BFGS, failed line searches and s.y <= 0 pairs.

On the 14 parity sets (--mode refine, GPU) the refined models are md5-identical; on the four
5f6m MD starts R-free 0.1707/0.1676/0.1720/0.1669 -> 0.1707/0.1690/0.1719/0.1668 (median
0.1692 -> 0.1699).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 2f4b0b5ea54fdac54de7ee6c5a1476d2fc019fd8)
2026-10-26 15:38:59 +01:00
leonarski_f 2be88e11e3 Refinement restraints: contacts with lattice translates are kept, and an atom's own symmetry contact once
Contacts() dropped every pair found in image 0 with a nonzero lattice shift ("the identity term
covers the contact"), but that pair is a contact with a translated copy of j, not with j itself:
in P1 no intermolecular repulsion or B-contact restraint was made at all, and in every group the
contacts along a, b and c were missing (MeasureGeometry's clash count with them). Such a pair is
now kept like a symmetry mate's, from the lower atom of the two.

An atom against its own image x -> T x is also met through T^-1 at the same distance (another
operator, or the same one at another lattice shift for a screw), and both were kept: atoms near
a 3-, 4- or 6-fold or a screw axis were repelled twice. For i == j the image first in fractional
coordinates of T x and T^-1 x is kept; an involution is kept once as before.

Test: in P1 two waters 2.4 A apart across a cell face get one repulsion and one B contact (none
before); a water 1.4 A from a 3-fold axis gets one repulsion with its images (two before).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 0d6e4d0491e655a49d59b709e1398843506d162a)
2026-10-26 15:38:59 +01:00
leonarski_f 470fc4e479 ffbidx (vendored): INDEXER_GPU_DEVICE is not read, the indexer stays on the thread's pinned GPU
The environment variable overrode the current device that ScopedGpuPin sets, so a stray value
moved an indexing thread to another card. Recorded in the file header as a local modification,
together with the deterministic tie-break of c982ba4f6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 89acdeef939ee82cf15ee9c5fb52a016fb1441f8)
2026-10-26 15:38:59 +01:00
leonarski_f 257bf2fe2b Refinement: trial slots counted with their scale pool and both restraint sets; on the CPU no more than -N
GPU: a slot was packed onto a card by its engine's bytes alone. The per-slot estimate now adds
the slot's ModelScaleGPUPool with the robust fit's buffers (ModelScaleGPUPool::DeviceBytes(n, 1,
true)), and RefinementGPUEngine::DeviceBytes counts the two restraint sets SetRestraints() holds,
not one. The 80%-of-card budget is unchanged, so where these do not fit fewer slots are packed
instead of an allocation failing inside a running trial.

CPU: OwnSlotDevices made a slot per wanted trial whatever the thread count, so -N 1 ran six trial
threads with six models, topologies and engines in memory. The main slot and those beside it are
now at most -N. Side by side equals sequential, so no result changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 3c04c01158266c267f8e5ffaaaa98d6d8fd2a747)
2026-10-26 15:38:58 +01:00
leonarski_f ac58921445 rugnux live: a score that is not finite is left out of the status, not sent as null
tf_z, rf_z, mr_d_min, the placed model's R-factors, CC, map and anomalous sigmas, the rigid
body's move, the refinement's R-factors and rmsds, and the headline d_min are set only where
finite (llg_z and null_sigma already were). nlohmann writes NaN/inf as null, which the page's
"!== undefined" checks let through to .toFixed(), and which the zod schema rejects for the
required blob distance - a blob with no atom within 12 A has an infinite one. That distance
is now optional (jfjoch_api.yaml; C++ model and frontend client regenerated), absent then.
The model cards test for != null, so a null that arrives anyway does not take the page down.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 62ff026aa507e5bda1daa5aa079f32fb5c21f16e)
2026-10-26 15:38:58 +01:00
leonarski_f 3d382ce0a0 Refinement: an exception in a trial slot or in RefineModel no longer ends the run
A slot builder thread that throws (gemmi's topology or hydrogen placement, bad_alloc) would
call std::terminate - rugnux or the broker gone. The slot is now left out as one the card
cannot take is, with a warning, and MakeOwnSlot's GPU engine catches std::exception, not only
JFJochException. Both callers of RefineModel (after the merge, and --mode refine) catch
std::exception: the refinement is reported NOT_PERFORMED with "refinement failed: <what>" and
the run writes everything else (cell e.s.d.s, unmerged MTZ, report), as the refinement is
processing-inert.

Test: --mode refine on a cell too large for the bulk-solvent grid comes back with
REFINE= NOT_PERFORMED and the reason, not an exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dc4e99fe4b3da2a9a3fa4290b5f70bfcc8c7bdb8)
2026-10-26 15:38:58 +01:00
leonarski_f 0143d57eb0 CellEsdTest: synthetic cells in place of cells that looked measured
The monoclinic cell of the .ins/mmCIF and fit tests is now 10/7/12, beta 100, and the
orthorhombic, hexagonal and tetragonal cells of the sampling tests are rounded synthetic ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 6f964bc0de0ca842f374e6aed83c8848e410173b)
2026-10-26 15:38:58 +01:00
leonarski_f 48bf9ebe3a CellEsd: PI from JFJochMath.h instead of M_PI, and the Herbstein (2000) credit at the block e.s.d.s
M_PI is not defined by MSVC without _USE_MATH_DEFINES, and CellEsd.cpp is reachable from the
viewer build. The citation, so far only in CellEsd.h, now also sits at the algorithm it credits.
The other files new in rc180 under common/, image_analysis/, reader/, writer/, rugnux/ and
viewer/ use no M_* constants or POSIX-only calls (grep of the rc180 diff).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit a019bc5fd2084907af9379068af3265ffe6c9ef5)
2026-10-26 15:38:58 +01:00
leonarski_f 1b48f3873f docs: front page and quick start - Rugnux goes on to a refined structure with a model
Landing page (title, hero, lede, Rugnux card), the documentation home's Rugnux card and the opening
of RUGNUX.md now say what the program does: from diffraction images to merged data and, with a
model, on to a refined structure and maps.

RUGNUX.md Quick start is rebuilt around how a run is started: rotation (master file, or the first
frame of a CBF/marCCD/SMV sweep), serial stills (several masters or all of a dataset's files by
glob), serial stills with -C/-S and --crystfel-geometry (wavelength/distance flags only where the
files state neither), and with a model (positional or --model): placement by rigid body or MR,
refinement by default (--no-refine), maps and blobs, and that the model changes no processing
decision except the indexing choice on serial stills with -C/-S. It says plainly that the defaults
are meant to be the answer, and lists the output files with and without a model. The tutorial's
stills section drops -X ffbidx, -z and the resolution limit from its first example and shows the
same three stills commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit bfca846332037688364b9d2d1c4250315f09410d)
2026-10-26 15:23:07 +01:00
leonarski_f cd0b994027 API: rugnux_live_refinement no longer says live refinement needs a GPU
Live refinement also runs on a broker without a GPU, on the CPU and slower, and the run's deadline
or cancel can cut it. Description only; regenerated broker/gen (Rugnux_live_refinement.h), the
frontend client (types.gen.ts, zod.gen.ts), broker/redoc-static.html and the python client doc
page - each differs from before only in this description.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 53d06c6e221fa1cf776822965560fb990cc69c73)
2026-10-26 15:23:07 +01:00
leonarski_fandClaude Opus 5.5 4b9809d9fa docs: BATTERY_REPORT.md removed - a stale battery report in no toctree
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-26 15:16:27 +01:00
leonarski_f 66cc0bd563 docs: acknowledge Herbstein (2000) for the unit-cell e.s.d.s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 795c98c9d6b5db0cde4d6e0b8c7b0dad9c99775f)
2026-10-26 14:50:42 +01:00
leonarski_f af28cf430b docs: technical pages caught up with the stills rocking curve, beam-shadow integration and refinement
INTEGRATION: the stills prediction window and partiality from the run's
measured rocking curve (a d* table, kept in _process.h5 for --mode scale),
the model-error term in the per-crystal fits, --min-partiality defaults,
the adaptive radius on stills (background-ring reach).
IMAGE: reflections reaching into the beam-stop shadow are not integrated;
no-data pixels masked as dead, not as shadow.
INDEXING: the rotation-scale walk's refusal and refit; the stills geometry
refinement on the given cell's crystal form only.
DECISIONS: the model is refined after validation by default.
CPU_DATA_ANALYSIS: titles in the references that lacked them.
GPU: the still prediction kernel follows the rocking-curve window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 5ae198b566b17de259fd0f0d6c8a0f8ea18eaa05)
2026-10-26 14:50:42 +01:00
leonarski_f 7b1bc3a3ea docs: report headline keys, model section for every way a model is given, still battery
RUGNUX_REPORT: a table of the headline keys (checked against
ResultReport.cpp); section 5 appears for --model, --mr or a model among
the inputs and holds the MR and REFINE_ keys; a --mode refine report has
one section; stills ambiguity resolved by the crystals' own cells first;
the developer-only CELL_ESD_OBSERVATIONS / CELL_DSPACING_FIT; the
"nothing here is refined" paragraph points at REFINE_*.
BENCHMARKING: the still battery, the refined R-free beside the placement
one, the Refinement and Doubled-cell steps in the timing table.
EXTERNAL_TEST_DATA_STILLS: the gain is measured, not passed; how the
battery runs these sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit e9f5e3b63fb6208d6bd5984c1ecd0eb0da7c29ab)
2026-10-26 14:50:42 +01:00
leonarski_f 07c8b2ddd4 docs: the Rugnux card and the naming table mention refinement and live collections
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit f8d07397288137dd5e3a0e5c101f2036d51228bd)
2026-10-26 14:50:42 +01:00
leonarski_f c3c5a32a0d docs: rugnux getting-started pages - refinement, the full output list, a troubleshooting note
RUGNUX: a model run also refines the model it placed; the model can be
given as an input; only the unmerged MTZ is written by default; the
_process.h5 is not a default output; a short "when a run fails" note.
TUTORIAL: the .ins, the refined model/maps/blobs, the log and _modules.h5
in the output list; the report is not written with -o ""; stills: -X Auto
picks ffbidx with a cell, a run over several files.
FORMATS: the XFEL stills formats and model/reference inputs told by content.
CALIBRATION: -C and -S as the calibrant. INSTALL: the CPU path runs every
indexer and the refinement. MODEL_AND_REFERENCE: the refined model is
written as .pdb too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 0062d72b95e996438ddc65dc343b526c84daa7fb)
2026-10-26 14:50:42 +01:00
leonarski_f 8edd0e5f3e docs: rugnux options, model and export pages brought in line with the usage text
RUGNUX_ADVANCED: --simulate-live/--fast-live rows, -e exclusive, stills
behaviour of --beam-center-check, -S, --min-partiality (0.5),
--capture-uncertainty, --adaptive-integration-radius (on for stills too),
--detector-distance/--rot1/--rot2/--no-refine-tilt; --background-radial
auto as it decides now; --refine-modules syntax; _cell.*_esd.
MODEL_AND_REFERENCE: the default refinement of a placed model in the
summary, the decision table and the step list; --mode scale does not refine.
INTEGRATION: the .ins written beside every .hkl (ZERR on rotation).
VIEWER: rugnux writes _process.h5 only with --no-merge/--write-process-h5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 51d2564356ba46bda8730c855ff8d3ee7c23d605)
2026-10-26 14:50:42 +01:00
leonarski_f d8c14046b2 rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report
Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated
observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed
centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal
metric, so the components of G* the point group leaves free (projected from the group's rotation
parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the
GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal
e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000),
so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second
estimate; the written e.s.d. is the larger of the two per parameter. Stills get none.

Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the
group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder
organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where
fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals;
report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS,
CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT).

On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets
written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated
.ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k
lysozyme observations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)
2026-10-26 14:33:38 +01:00
leonarski_f 22debb1be6 docs: the stills rocking-curve datasets in the _process.h5 master
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 1195b9be6e0ca66774774f6de4867180b7bd0a47)
2026-10-26 14:09:32 +01:00
leonarski_f e6b550fe42 Tests: the stills rocking width - the window at a reflection's resolution, _process.h5 round trip
StillsPartialityRefine_MeasureRunWidth_WindowAtReflectionResolution, JFJochReader_StillsRunWidth.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dafb8d1bf996d1000741eecf019aee780dd91ebe)
2026-10-26 14:09:32 +01:00
leonarski_f 8a123e794d Stills: the run's rocking curve goes to _process.h5 and --mode scale merges with it; MeasureRunWidth takes the window at each reflection's resolution
--mode scale re-measured the rocking width on the stored reflections, so on any file whose frames were
predicted through the run's curve it merged with a different curve than the mx run did. The curve the
post-refinement used (formula, table, radial scale) is now carried in the EndMessage (StillsRunWidth) and
written to the master: /entry/MX/stillsRockingWidthFormula [gamma0 1/A, gamma_e rad, bandwidth sigma],
/entry/MX/stillsRadialScale, /entry/MX/stillsRockingWidthTableDstar and /entry/MX/stillsRockingWidthTable
(1/A). The reader returns it, and --mode scale passes it to the post-refinement. A file without it is
re-measured as before.

MeasureRunWidth grouped each frame's reflections by the frame's window, taken as the largest excitation
error it predicted. With a window that follows sigma(d*) that is the high-resolution window, applied to the
frame's low-resolution reflections too. Given the curve the frames were predicted through, the window is now
that curve at the reflection's own resolution; without one, the frame's largest excitation error as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 992343d0206489991880bb927efc79131134a1d9)
2026-10-26 14:09:32 +01:00
leonarski_f 00662c4110 Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records
The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds:
- the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group,
  joined by the library's backbone link "p");
- the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out:
  its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A.
links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers),
CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications.

AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records
to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts
the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a
ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal
bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond
rmsd near 0.1 A.

Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the
ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P,
8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each
modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries
against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed,
ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)
2026-10-26 13:56:29 +01:00
leonarski_fandClaude Opus 5.5 d90bc68085 CI: the documentation also pushed to the gitea-pages branch of rugnux/gitea-pages, on every push for now (token test)
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 17m28s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 21m52s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m20s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 27m37s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m1s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m28s
Build Packages / Unit tests (push) Successful in 1h31m22s
Build Packages / Validation Battery (push) Successful in 38m39s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 26m41s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 28m19s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 22m55s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 24m33s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 28m7s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m57s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 30m2s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 30m18s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 24m56s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 28m29s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 28m41s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / Create release (push) Successful in 7s
Build Packages / Build documentation (push) Successful in 45s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-26 11:39:10 +01:00
leonarski_fandClaude Opus 5.5 771e669f75 rugnux usage: -S on serial stills also settles the indexing ambiguity from the data
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-26 11:16:34 +01:00
leonarski_f 6491cc5a7e docs: serial-stills indexing ambiguity described the same way everywhere
Two mechanisms, as the code has them: with -S and no reference or model, the crystals made
consistent from the data (Brehm & Diederichs objective, cell-metric anchors); with -z or --model
(-C/-S), per image against the reference, crystals their own cell decides keeping that indexing,
the data-only result as a check. Removes the statements that the ambiguity needs a reference or
model (tutorial, quick start, overview, §10.9), adds the anchoring to the reference/model rows,
the "no -S" case (mixed merge, can pass for higher symmetry, warning), and the report keys in the
reference case (AMBIGUITY_DATA_DISAGREES in place of AMBIGUITY_REINDEXED, AMBIGUITY_TIME_S,
none in --mode scale).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 8db1d3ecb97dba90471c4db238aff632d11bc3e8)
2026-10-26 11:15:54 +01:00
leonarski_f 0cc3129650 docs: refinement's GPU engine in GPU data analysis, refinement in the pipeline figures
GPU_DATA_ANALYSIS.md: the refinement section at the level of the other engines - density gather,
cuFFT r2c/c2r, compose + likelihood target, CSR adjoint, warp-per-atom gradient, restraints and
slot gather, fine mask and robust scale fit (on-device median), reductions and determinism,
stream priority and blocking-sync waits, trial slots over cards, DeviceBytes budget, CPU fallback,
no Metal; indicative RTX 5080 timings.

Pipeline figures: a dashed Refinement box after Structure & map on both, the model branch labelled
as deciding nothing; the stills row gains the indexing-ambiguity step before the first merge.
RUGNUX_OVERVIEW.md: captions, a refinement paragraph, refined outputs under Write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 8645fb2aca1abbbddec29cdfa99056471f8f38a5)
2026-10-26 11:15:54 +01:00
leonarski_f e406eb17ea docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits
- New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient,
  scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library
  subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps,
  blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and
  CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined.
- RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the
  --mode refine input keys and the _blobs.json fields.
- RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the
  refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the
  refinement's CUDA engines and the missing Metal path.
- ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980
  (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent
  mask), Kullback & Leibler 1951; all DOIs checked against Crossref.
- In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the
  R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at
  FindDensityBlobs, Read 1986 at the refined maps. Comments only.
- docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it
  lacked the CCP4 Monomer Library row).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10)
2026-10-26 11:07:43 +01:00
leonarski_f f960bd7a80 Docs landing page: a real diffraction frame in the hero, and the command with a model
The generated spot-and-ring drawing is replaced by one 0.2 deg rotation frame of the in-house
lysozyme standard (EIGER2 16M, SLS X10SA): Bragg spots on their lunes, the beam stop and its arm,
the module gaps. docs/landing/hero_frame.py makes the 920 px WebP (67 KB) from the master
file - crop about the beam centre, log tone map from the frame's own background, block mean+max
downsampling - and is run by hand; the WebP is committed and copied by the landing-page step.
The dark theme shows it inverted through a CSS filter.

The hero's command line drops its explanatory sentence and gains the second form,
rugnux <diffraction image file> <CIF/PDB model file>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit ff26000bfe7446cb4fda739c3b72596f8a8b7047)
2026-10-26 10:53:25 +01:00
leonarski_f 4dbcb35708 Stills partiality: the rocking width measured in shells of equal width in d*, used as a table
MeasureRunWidth measured the width in eight shells of equal count in d*^2 and fitted the three-term
formula gamma0^2 + (gamma_e d*)^2 + (bw |q_z|)^2 to them. On runs predicting ~1000 reflections a frame
the equal-count shells all sat at 3.8-1.8 A, so the low-resolution width was the formula's
extrapolation, and the fit's form flipped with the number of frames (to a pure bandwidth at 50k frames).

The same window-group moment is now also measured in eight shells of equal width in d*, made
non-decreasing (pool adjacent violators, weighted by each shell's summed signal) and kept in RunWidth as
a table that the partiality interpolates linearly in d*. The formula is still fitted on the equal-count
shells and still sets the prediction window. With fewer than three measurable shells the formula is
used as before.

Still battery, 12 non-float sets, together with the sigma-following prediction window (the previous
commit; run 20261026-1021_1b28d6_still-part-comb against rc180 20261026-0857_3882b0_still-part-base1,
0-cycle REFMAC R-free vs deposited, both --no-refine): 6 better, 3 unchanged, 3 worse (by 0.002), sum of
changes -0.027 (the window alone: -0.007); largest gains pal90 -0.014, alvra103_int16 -0.011, lcls222
-0.005; largest losses euxfel152 +0.005, alvra213_int16 +0.005 (+0.010 with the window alone), euxfel83
+0.003. Win/draw/loss against the depositions unchanged at 9/2/1. Rotation: p.hkl byte-identical on
5reo, lyso_x06da_ref and thau_x10sa_16keV.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit de779bb10b1813eeae625e3994b1ca73d6079d7a)
2026-10-26 10:50:52 +01:00
leonarski_f 0829b9a72d Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale
Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its
indexed spots): one width at every resolution, differing eightfold between frames. Against the run's
measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions
empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated).

Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way,
scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and
width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that
curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU
and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through
windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run.

Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited):
9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its
sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall
+13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three
rotation sets.

Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc)
2026-10-26 10:50:52 +01:00
leonarski_f 13da79b788 Refinement GPU: the atom gradient kernel walks its box without integer division
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m0s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m28s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 18m17s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m17s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 23m45s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 23m41s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 27m32s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 28m36s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 29m38s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 31m39s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 24m10s
Build Packages / Generate python client (push) Successful in 32s
Build Packages / Build documentation (push) Successful in 1m7s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 23m51s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 28m4s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 29m44s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 23m45s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 25m39s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 28m32s
Build Packages / Unit tests (push) Successful in 1h35m16s
Build Packages / Validation Battery (push) Successful in 41m34s
Bitwise. One warp per atom, each lane taking the box's points t = lane, lane + 32, ...: every point cost three
integer divisions/modulos to turn t into (u, v, w) and three more (imod) to wrap them into the grid. The lane
now steps (iu, iv, iw) on by 32 points with adds and carries, and wraps from the box's first point, wrapped once
per atom (a box is narrower than the cell, so one conditional subtraction per axis). The same points in the same
order per lane, the same arithmetic.

atom_gradient_kernel (nsys, side by side): 5reo 426 -> 237 us, 5f6m 887 -> 523, 9yl4 1436 -> 802, 7d98 692 ->
388 us per launch. Refined CIF md5 unchanged on the 5 GPU cases; side by side = one at a time. Wall: 5reo
1.43 -> 1.35, 5f6m 2.20 -> 2.00, 9yl4 6.88 -> 6.82, 7d98 3.58 -> 3.38, 5f6m MD start 4.16 -> 3.71 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 6554e9dcbe4b448eda582eb76d6a24c59fe6ab0c)
2026-10-26 09:16:34 +01:00
leonarski_f 4a8e13ddfa Refinement GPU engine: its waits on the stream asleep (blocking-sync event) rather than spinning
Bitwise. cudaStreamSynchronize spins a core for the whole wait; the weight trials' engines wait on a shared card
for most of a run, so about half of the user cycles of a refinement were the spin (PROFILE.md: 38-50%). A
StreamWaiter (CUDAMemHelpers.h: an event made with cudaEventBlockingSync, recorded and waited for) now takes the
refinement engine's waits - evaluation, Fcalc, Fmask, the uploads of positions, row model, blur and restraints.
The short waits of the host-driven loops stay spinning: the scale fit's Levenberg-Marquardt (one launch of
66-120 us per step) and the gather's pair count; making those block too cost 2-3% of wall (wake-up latency),
here the wall is unchanged. No process-wide cudaDeviceScheduleBlockingSync: that would change every GPU wait
of the broker.

CPU use (-N 8, time -v): 5reo 301-315% -> 190%, 5f6m 326% -> 181%, 9yl4 333% -> 222-228%, 7d98 307% -> 181%,
5f6m MD start 309-320% -> 151%. Wall, two interleaved runs each: 5reo 1.43-1.52 -> 1.43, 5f6m 2.19-2.20 ->
2.19-2.20, 9yl4 6.83-6.84 -> 6.88-6.89, 7d98 3.50-3.51 -> 3.58, 5f6m MD start 4.16-4.34 -> 4.16 s. Refined
CIF md5 unchanged on the 5 GPU cases; side by side = one at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit d23d2cf7bb99b30617a3f57c531b03db8bc03e05)
2026-10-26 09:16:34 +01:00
leonarski_f f34d2f4cca Refinement: the weight trials' own slots made while the placement runs
Bitwise. MakeOwnSlot copies the model as it was before the topology and reads only what the placement does not
change (cell, reflections, the engine's kind, the B ceiling), so the slots are now built on their threads right
after the start score, beside the rigid-body and group-B placement, and joined where they were built before
(also where the placement throws). The "slots" stage goes from 0.23 s to 0 on 9yl4; the placement is not
slowed (0.59 -> 0.60 s). Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h
(CPU); side by side = one at a time. Wall: 5reo 1.46-1.53 -> 1.54, 5f6m 2.26 -> 2.21, 9yl4 7.03 -> 6.83, 7d98
3.65 -> 3.56, 5f6m MD start 4.20 -> 4.15 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 0752a921a4293d4213894e3242c4ef36d8ce2a05)
2026-10-26 09:16:34 +01:00
leonarski_f 1f8b5b1986 Refinement: a Refit's solvent grid starts around where the context's last one ended
Logic change (deterministic), kept on evidence. Every Refit searched the whole k_sol/b_sol box - an 88-point
coarse pass of plain least squares, then the robust fine pass around its winner. ModelScaleBox (and
ModelScaleGPUPool::Fit / ModelScaleGPU::FitSolvent) take an optional start; with one, the coarse pass is
skipped and the fine pass is laid around it. Refinement passes the grid winner of the context's previous
Refit (before MoveOverallB moves the overall B into b_sol); it travels with a trial's fit, and a trial's first
Refit without one (Context::ForgetStarts, which also clears the likelihood's D start) searches the whole box,
so a trial still depends on nothing its slot ran before (side by side = one at a time, md5).

Evidence (GPU, -N 8): full refinement of the 14 parity sets from their placed models - 12 refined CIFs
identical (md5), 9ig7 and 6iu9 differ, final R-work/R-free equal to 4 digits but 6iu9 R-work 0.2911 -> 0.2910;
of the 5 timing cases 4 identical, 7d98 (cif start) R-free 0.3096 -> 0.3095; 5f6m MD starts 01-04 median R-free
0.1692 -> 0.1692; 6o2h on the CPU engine identical (md5). The score-only parity table is not touched (one Refit,
no start). refit.scale (TIMING sum): 5f6m 1.52 -> 1.10 s. Wall: 5reo 1.68 -> 1.53, 5f6m 2.43 -> 2.24, 9yl4 7.19 ->
7.02, 7d98 3.70 -> 3.65, 5f6m MD start 4.57 -> 4.22 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit ff285fb525ecf4b480698d4aafe212a34301a8fe)
2026-10-26 09:16:34 +01:00
leonarski_f 1f1aa18b67 ModelScaleGPU: the robust fit's median of the residuals on the device
Bitwise. The first robust round copied every fine-pass fit's residuals to the host (up to 25 x n floats,
9 MB at 92k reflections) and took std::nth_element of each there. The median is now a radix selection on the
card (one block per fit, four byte-wide passes of integer counts over the float bits - the residuals are not
negative, so the bits order as the values): the element nth_element puts at n / 2, the same value; only the
fits' c come back. FitSolvent and RobustBatch lose the host threads they took for it.

Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start. Wall (-N 8, quiet): 5reo 1.72 -> 1.68,
5f6m 2.57 -> 2.43, 9yl4 7.28 -> 7.19, 7d98 3.78 -> 3.70, 5f6m MD start 4.76 -> 4.57 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dc53896c335fdabdf4d417323038e05b8b63535d)
2026-10-26 09:16:34 +01:00
leonarski_f ca0a6dd343 Refinement: each slot's likelihood fit on all of the run's threads
Bitwise (the bins are fitted independently: the same bits at any thread count). The weight-trial slots Refit
in lockstep while the GPU idles, each with its 1/slots share of -N for the ML bins; the bins now go to the
shared worker pool with -N tasks per slot, so the threads of slots that are not fitting help the ones that are.
After the Newton searches (previous commit) the ML fit is 0.1-0.5 s summed over a run, so the wall gain is
within noise: 5reo 1.63-1.75 -> 1.72, 5f6m 2.60 -> 2.57, 9yl4 7.34 -> 7.28, 7d98 3.78 -> 3.78, 5f6m MD start
4.82 -> 4.76 s. Refined CIF md5 unchanged on 5reo, 5f6m, 9yl4, 7d98, 5f6m MD start (GPU) and 6o2h (CPU).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 56815f5390cc01e8da5d420fff73cdf4394c385c)
2026-10-26 09:16:34 +01:00
leonarski_f 2c1df0ebfa Refinement: the likelihood's D and beta searches by Newton's method; D on the working set from the last fit's
Deterministic, rounding changes. The per-bin ML fit took 4 rounds x (40 + 40) golden-section steps on the
free set and a 50-step golden section for D on the working set - each step a Bessel I0/I1 per row, ~4.6M per
Refit at 92k reflections, 27-37% of a trial's wall on the high-resolution sets. Each one-dimensional search is
now a safeguarded Newton iteration on the analytic first and second derivatives (bracketed on the sign of the
derivative, bisection where a step leaves the bracket or the curvature is not positive), within the same
brackets; the free-set alternation keeps its 4 rounds from the moments, so beta is the one it was (a converged
joint fit was tried: the 4 rounds are not converged in the low-resolution bins, and converging them moves
6iu9's FOM by -0.003, so it was not kept). D on the working set starts from the bin's D of the context's
previous Refit (XrayRowModel::bin_d), restored with a trial's fit and cleared at a trial's first Refit, so a
trial depends on nothing its slot ran before (side by side = one at a time, md5).

14 parity sets, score only: R-work/R-free identical to 4 digits, mean |FOM - REFMAC| 0.01186 -> 0.01186,
D within 1e-5 and var within 1e-4 (relative) per reflection. refit.ml (TIMING sum, -N 8): 5reo 1.47 -> 0.11 s,
5f6m 3.88 -> 0.26, 9yl4 1.37 -> 0.11, 5f6m MD start 4.97 -> 0.46. Wall: 5reo 2.03 -> 1.75, 5f6m 3.45 -> 2.60,
9yl4 7.99 -> 7.36, 7d98 4.04 -> 3.78, 5f6m MD start 6.10 -> 4.82 s. 5f6m MD starts 01-04 median R-free
0.1693 -> 0.1692.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit a91511945e0ccd7145a93877c0c231e87f86fb44)
2026-10-26 09:16:33 +01:00
leonarski_f a8d3a8dbc2 Refinement: the mask grid sized without allocating it; the L-BFGS pairs keep their s.y
BulkSolventMask::MaskGrid, documented as "sizes and spacing set, values not allocated", allocated and zeroed the
whole fine mask grid (up to 2^26 floats) - in BulkSolventMask's constructor and in every GPU engine's, i.e. once
more for each weight-trial slot, only for the grid's size. It now takes good_grid_size() as set_size_from_spacing()
does, and the CPU engine's Fmask() allocates its copy. rugnux_refine_test, RTX 5080, -N 8: 9yl4 8.6 -> 7.8 s,
7d98 4.85 -> 4.0 s (engine 0.66 -> 0.16 s, slots 0.60 -> 0.23 s on 9yl4); the high-resolution sets, whose mask
grid is small, are unchanged.

RefinementMinimiser keeps s.y of every pair it stores and Direction() takes rho and gamma from it instead of two
more dot products per pair per call (the same Dot() of the same vectors): Direction's CPU time -22% on 9yl4; no
wall change with the trials side by side, where other slots fill the card meanwhile.

Bitwise: refined CIF identical on 5reo, 5f6m, 9yl4, 7d98, a 5f6m MD start (GPU) and 6o2h (CPU engine).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 6f5caf5e1b1183a752acc5cca78bfb7ef002b12a)
2026-10-26 09:16:33 +01:00
leonarski_fandClaude Opus 5.5 6da51dc5d7 rugnux: no --refine flag - a placed model is refined by default, --no-refine skips it
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m22s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 10m49s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 18m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 22m42s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m12s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 23m21s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 27m29s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 28m15s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 29m18s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 31m25s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 23m32s
Build Packages / Generate python client (push) Successful in 39s
Build Packages / Build documentation (push) Successful in 1m12s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 24m34s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 27m41s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 23m41s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m2s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 28m20s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 25m53s
Build Packages / Unit tests (push) Successful in 1h35m43s
Build Packages / Validation Battery (push) Successful in 39m44s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-26 06:54:15 +01:00
leonarski_f 3882b0f062 Serial stills with a reference or model: crystals their own cells anchor keep that indexing
The per-image reference/model choice of indexing ignored the crystals' refined cells. On a
pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as
indexed than after a twin law, the cells decide that law - the rule the data-only resolution
already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver
now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills
merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets
the reference choose only among the indexings left open. Exactly merohedral lattices (no metric
signal) are unaffected.

Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets):
pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16
0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged;
lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against
cells that decide all three laws (96/96/100%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f)
2026-10-26 06:49:20 +01:00
leonarski_fandClaude Opus 5.5 ba52aadbce rugnux: the geometry refinement algorithm is -r, --geom-refine; --refine was declared twice and its no-argument form shadowed it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-26 06:49:20 +01:00
leonarski_f a78bfe951d rugnux: a model and a reference among the inputs, in any order, told by their contents
`rugnux <images> <model> <reflections>` in any order, without --model or -z. Each positional file
is classified by its contents (RugnuxInputKind.h, in JFJochRugnuxOutput - the library still reads
no files): an MTZ (magic) or an mmCIF with a _refln / _diffrn_refln loop is reference data (-z; the
input itself in --mode refine), a PDB file (a PDB record first, then ATOM/HETATM) or an mmCIF with
an _atom_site loop is a model (--model), anything else is images for the readers. CBF (###CBF or
_array_data) counts as images; a file whose first line is neither data_ nor a PDB record is decided
there, so image files are not read through. An mmCIF holding both atoms and reflections is refused
as ambiguous. At most one model and one reference, --model/--mr and -z included. Gzip read through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 8f844b0bf86f338d20e8aed426a387722dd8d7cc)
2026-10-26 01:38:25 +01:00
leonarski_f 794db0ca34 Refinement cut by the run's cancel; the hero shows the refined R-work / R-free
The refinement of a placed model gets the run's cancel flag (RefinementSettings::cancel, through
CancelFlag::RunFlag), so the broker's deadline or a cancel - and Ctrl+C in rugnux - cuts a running
refinement short with what its finished trials found. RefinementResult::cancelled goes to the live
status as rugnux_live_refinement.cancelled (API, regenerated clients) and to the step's note.

Hero: once refined, the model card's headline is "R-work / R-free, refined" (e.g. 0.32 / 0.35), its
green/red still the placement verdict, the placement and fit in the context line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 395016a30c2bc2aa26fdf2cec2ced9cebe6fad80)
2026-10-26 01:38:25 +01:00
leonarski_f baec34685d Rugnux live: the uploaded model is refined after placement; refined scores, model, maps and blobs in the status, the viewer and the hero
The live run refines a placed model as `rugnux` does (ProcessConfig::refine), after every decision
and before the end message; a cut before it starts leaves it out, as the model. The writer still
gets the placed model; the refined one stays with the broker.

API (broker/jfjoch_api.yaml):
- rugnux_live_model.refinement (rugnux_live_refinement): performed, reason, on_gpu, seconds, r_work,
  r_free, rmsd_bonds_A, rmsd_angles_deg, blob_count, blobs (at most 50, rugnux_live_blob: kind, x, y,
  z, peak_sigma, volume_A3, nearest_atom, nearest_atom_distance_A)
- rugnux_live_status.refined_model
- /rugnux/model.cif and /rugnux/map.ccp4 take stage=placed|refined, by default the refined model
  where there is one; the anomalous map stays on the placed model's phases (the refined
  coefficients carry none).
Regenerated broker/gen, frontend/src/client, redoc, and the python client docs (which also brings
the hero fields and rugnux_live_model it was missing).

Frontend: the model card keeps its green/red and gets "R-free 0.xx (refined)" as its context line,
"refining..." while the step runs; the model section adds refined R-free and blob tiles; Mol*
shows the refined model and maps, with the blobs as small spheres coloured by reading (switch, off
at first). The Refinement step is in the timeline like any other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 3b47fb9a53b7dfad1c1ed289e169094c8d3d9d3e)
2026-10-26 01:38:25 +01:00
leonarski_f 7bf906e4a2 rugnux: a placed model is refined by default - giving a model is the opt-in
Refinement (Refinement.h) now runs whenever --model or --mr places the model (rigid body fits, or
molecular replacement placed it), on the GPU or the CPU: a run that should not spend the time is
given no model. --no-refine skips it (for debugging); --refine is still accepted and ignored. A
model that was not placed (rejected by the rigid body, not placed by MR - e.g. the battery's
wrong-model null) is no longer refined.

Processing-inert as before: it runs after every decision; R_WORK/R_FREE and every key outside
REFINE_ (section 5), and <prefix>.mtz / _model.cif / _maps.mtz, are unchanged. The block moved
after the placement notes so they stay on the step that placed the model. The viewer's processing
jobs refine too. The equivalent command line gives --no-refine back where refinement was off.

Battery: score.py reads keys exactly, so REFINE_R_FREE cannot alias R_FREE; new report-only
fields rfree_refined, rwork_refined, refine_s (part of rugnux_wall_s), shown in the per-set table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit b0bb741831f517b553e59152b3b482a3f75d953a)
2026-10-26 01:38:24 +01:00
leonarski_f 3b06bcee78 Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win
Three changes to how the trajectory of commit 2bfb84bc4 moves and ends:
- A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the
  tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it
  was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking
  on while R-work falls.
- Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model
  improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's
  trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6).
- The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds,
  R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement
  that on deposited models cost R-free.

Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016
(e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time
halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this:
5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 ->
0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857)
2026-10-26 01:38:24 +01:00
leonarski_f bda90dbdd5 Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms
- B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two
  isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2.
  kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list
  sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these
  equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates
  included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU.
- The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on
  the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held
  and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the
  final ladder leave it.
- Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the
  solvent's B (F_model unchanged), so the two do not drift against each other.
Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the
median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265
vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time.

Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to
convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311),
11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the
model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)
2026-10-26 01:38:24 +01:00
leonarski_f 6369a263b1 Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start
Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so
~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now:

1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each
   with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only
   within the geometry bounds and by more than the R-free tie);
2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and
   the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers
   neither R-work nor R-free by more than the tie (at most 8 steps);
3. a final ladder from there at x1/2, x1, x2.

A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts
from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral),
and the slots that run share the threads.

On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's
scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles):
5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 ->
0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s).
Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder
can lower it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519)
2026-10-26 01:38:24 +01:00
leonarski_f 4424d8884d Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature
Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every
10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser
keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's
scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking
Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian.

The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution
shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference
probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints'
Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser,
since that stiffness is in directions that move bonded atoms together.

Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the
refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345,
7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393;
REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work
lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung
further). Time unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
(cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)
2026-10-26 01:38:24 +01:00