207 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	067f3a7d7b	docs landing: Rugnux card names synchrotron rotation and XFEL serial data and links the public test-data lists; footer motto removed ... The Rugnux card says in passing that it takes a synchrotron rotation sweep or XFEL serial stills, and ends with one sentence pointing at the two pages that list the public datasets it is validated on (External test data: over 200 rotation sets, 207 in the open battery manifest; External test data: serial crystallography: eleven CXIDB entries). The "two mountains, two railways" line at the foot of the page is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 03e4431836a2f4ba816e6d3cdd57cc6ada87b95d)	2026-10-26 16:59:43 +01:00
leonarski_f	2736f4c85e	Docs landing page: the hero is a live Rugnux run - a frame, the placed model in its density, the timeline ... The front page's hero image is now the cytochrome C collection at SLS X10SA as the broker's Rugnux live page ran it: a detector frame on the page's indigo beside the AlphaFold model placed in its electron density, over the two-lane timeline of the live analysis (the 18 s collection, the image processing alongside it, scaling, merging and molecular replacement 4.9 s after the last image). One WebP per theme (hero_live_light.webp, hero_live_dark.webp), shown by the same rules that pick the theme's colour tokens, so the toggle and the system preference both switch it; the hero's art column widens to 560 px on desktop (360 px on a phone). The lysozyme frame and its generator go; the Sphinx extension copies the two new files into the site. Made by docs/landing/hero_live/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5f2592390d9ec39428198df8cd8e18ad60167a79)	2026-10-26 16:54:39 +01:00
leonarski_f	2e296e6bed	jfjoch_writer: the live refinement's model, maps and blobs are written beside the master file ... In broker live mode the refined model, its map coefficients and the difference-map blobs stayed in the broker for the viewer routes and were lost at the next run. They now go to the writer in the END message as the placed model and maps already do - EndMessage::refined_model (PlacedModelMessage), refined_map_coefficients (MtzTable) and blobs (DensityBlobsMessage, one vector per field) - and the writer writes <prefix>_refined_model.cif (+ .pdb), <prefix>_refined_maps.mtz and <prefix>_blobs.json, the names the CLI gives them. The blobs' JSON is made by one function (DensityBlobsJson) that the CLI now writes through as well, so the two files are byte-identical. Optional CBOR fields, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c65220d96a96d2b1897f3c6632c94f9379b670a3)	2026-10-26 15:50:40 +01:00
leonarski_f	2461d67e14	NXmx: the unit cell's e.s.d.s are saved as /entry/sample/unit_cell_esd ... Rugnux measured the cell's e.s.d.s (CellEsd) but they only reached the mmCIF's _cell.*_esd, the .ins's ZERR and the report. They now travel in the END message (EndMessage::unit_cell_esd, CBOR key unit_cell_esd, same layout as unit_cell), are written beside the cell in NXsample as unit_cell_esd (float[6]: a b c alpha beta gamma, A and degrees) and are read back into JFJochReaderDataset::unit_cell_esd. rugnux's _process.h5 and, through AddToEndMessage, the broker's live master file get them. Rotation only; absent on stills and where nothing was measured. Optional CBOR field, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6902b5637aa92c27565f22bcf3fe11b5d2536038)	2026-10-26 15:50:40 +01:00
leonarski_f	825b88c19d	CHANGELOG: 1.0.0-rc.180 entry ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 339c6cbaa4177964e5eb06d0878efb83d1da2dbc)	2026-10-26 15:39:07 +01:00
leonarski_f	1b48f3873f	docs: front page and quick start - Rugnux goes on to a refined structure with a model ... Landing page (title, hero, lede, Rugnux card), the documentation home's Rugnux card and the opening of RUGNUX.md now say what the program does: from diffraction images to merged data and, with a model, on to a refined structure and maps. RUGNUX.md Quick start is rebuilt around how a run is started: rotation (master file, or the first frame of a CBF/marCCD/SMV sweep), serial stills (several masters or all of a dataset's files by glob), serial stills with -C/-S and --crystfel-geometry (wavelength/distance flags only where the files state neither), and with a model (positional or --model): placement by rigid body or MR, refinement by default (--no-refine), maps and blobs, and that the model changes no processing decision except the indexing choice on serial stills with -C/-S. It says plainly that the defaults are meant to be the answer, and lists the output files with and without a model. The tutorial's stills section drops -X ffbidx, -z and the resolution limit from its first example and shows the same three stills commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit bfca846332037688364b9d2d1c4250315f09410d)	2026-10-26 15:23:07 +01:00
leonarski_f	cd0b994027	API: rugnux_live_refinement no longer says live refinement needs a GPU ... Live refinement also runs on a broker without a GPU, on the CPU and slower, and the run's deadline or cancel can cut it. Description only; regenerated broker/gen (Rugnux_live_refinement.h), the frontend client (types.gen.ts, zod.gen.ts), broker/redoc-static.html and the python client doc page - each differs from before only in this description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 53d06c6e221fa1cf776822965560fb990cc69c73)	2026-10-26 15:23:07 +01:00
leonarski_fandClaude Opus 5.5	4b9809d9fa	docs: BATTERY_REPORT.md removed - a stale battery report in no toctree ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 15:16:27 +01:00
leonarski_f	66cc0bd563	docs: acknowledge Herbstein (2000) for the unit-cell e.s.d.s ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 795c98c9d6b5db0cde4d6e0b8c7b0dad9c99775f)	2026-10-26 14:50:42 +01:00
leonarski_f	af28cf430b	docs: technical pages caught up with the stills rocking curve, beam-shadow integration and refinement ... INTEGRATION: the stills prediction window and partiality from the run's measured rocking curve (a d* table, kept in _process.h5 for --mode scale), the model-error term in the per-crystal fits, --min-partiality defaults, the adaptive radius on stills (background-ring reach). IMAGE: reflections reaching into the beam-stop shadow are not integrated; no-data pixels masked as dead, not as shadow. INDEXING: the rotation-scale walk's refusal and refit; the stills geometry refinement on the given cell's crystal form only. DECISIONS: the model is refined after validation by default. CPU_DATA_ANALYSIS: titles in the references that lacked them. GPU: the still prediction kernel follows the rocking-curve window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5ae198b566b17de259fd0f0d6c8a0f8ea18eaa05)	2026-10-26 14:50:42 +01:00
leonarski_f	7b1bc3a3ea	docs: report headline keys, model section for every way a model is given, still battery ... RUGNUX_REPORT: a table of the headline keys (checked against ResultReport.cpp); section 5 appears for --model, --mr or a model among the inputs and holds the MR and REFINE_ keys; a --mode refine report has one section; stills ambiguity resolved by the crystals' own cells first; the developer-only CELL_ESD_OBSERVATIONS / CELL_DSPACING_FIT; the "nothing here is refined" paragraph points at REFINE_*. BENCHMARKING: the still battery, the refined R-free beside the placement one, the Refinement and Doubled-cell steps in the timing table. EXTERNAL_TEST_DATA_STILLS: the gain is measured, not passed; how the battery runs these sets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e9f5e3b63fb6208d6bd5984c1ecd0eb0da7c29ab)	2026-10-26 14:50:42 +01:00
leonarski_f	07c8b2ddd4	docs: the Rugnux card and the naming table mention refinement and live collections ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f8d07397288137dd5e3a0e5c101f2036d51228bd)	2026-10-26 14:50:42 +01:00
leonarski_f	c3c5a32a0d	docs: rugnux getting-started pages - refinement, the full output list, a troubleshooting note ... RUGNUX: a model run also refines the model it placed; the model can be given as an input; only the unmerged MTZ is written by default; the _process.h5 is not a default output; a short "when a run fails" note. TUTORIAL: the .ins, the refined model/maps/blobs, the log and _modules.h5 in the output list; the report is not written with -o ""; stills: -X Auto picks ffbidx with a cell, a run over several files. FORMATS: the XFEL stills formats and model/reference inputs told by content. CALIBRATION: -C and -S as the calibrant. INSTALL: the CPU path runs every indexer and the refinement. MODEL_AND_REFERENCE: the refined model is written as .pdb too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0062d72b95e996438ddc65dc343b526c84daa7fb)	2026-10-26 14:50:42 +01:00
leonarski_f	8edd0e5f3e	docs: rugnux options, model and export pages brought in line with the usage text ... RUGNUX_ADVANCED: --simulate-live/--fast-live rows, -e exclusive, stills behaviour of --beam-center-check, -S, --min-partiality (0.5), --capture-uncertainty, --adaptive-integration-radius (on for stills too), --detector-distance/--rot1/--rot2/--no-refine-tilt; --background-radial auto as it decides now; --refine-modules syntax; _cell.*_esd. MODEL_AND_REFERENCE: the default refinement of a placed model in the summary, the decision table and the step list; --mode scale does not refine. INTEGRATION: the .ins written beside every .hkl (ZERR on rotation). VIEWER: rugnux writes _process.h5 only with --no-merge/--write-process-h5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 51d2564356ba46bda8730c855ff8d3ee7c23d605)	2026-10-26 14:50:42 +01:00
leonarski_f	d8c14046b2	rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report ... Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal metric, so the components of G* the point group leaves free (projected from the group's rotation parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000), so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second estimate; the written e.s.d. is the larger of the two per parameter. Stills get none. Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals; report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS, CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT). On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated .ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k lysozyme observations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)	2026-10-26 14:33:38 +01:00
leonarski_f	22debb1be6	docs: the stills rocking-curve datasets in the _process.h5 master ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1195b9be6e0ca66774774f6de4867180b7bd0a47)	2026-10-26 14:09:32 +01:00
leonarski_f	00662c4110	Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records ... The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)	2026-10-26 13:56:29 +01:00
leonarski_f	6491cc5a7e	docs: serial-stills indexing ambiguity described the same way everywhere ... Two mechanisms, as the code has them: with -S and no reference or model, the crystals made consistent from the data (Brehm & Diederichs objective, cell-metric anchors); with -z or --model (-C/-S), per image against the reference, crystals their own cell decides keeping that indexing, the data-only result as a check. Removes the statements that the ambiguity needs a reference or model (tutorial, quick start, overview, §10.9), adds the anchoring to the reference/model rows, the "no -S" case (mixed merge, can pass for higher symmetry, warning), and the report keys in the reference case (AMBIGUITY_DATA_DISAGREES in place of AMBIGUITY_REINDEXED, AMBIGUITY_TIME_S, none in --mode scale). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8db1d3ecb97dba90471c4db238aff632d11bc3e8)	2026-10-26 11:15:54 +01:00
leonarski_f	0cc3129650	docs: refinement's GPU engine in GPU data analysis, refinement in the pipeline figures ... GPU_DATA_ANALYSIS.md: the refinement section at the level of the other engines - density gather, cuFFT r2c/c2r, compose + likelihood target, CSR adjoint, warp-per-atom gradient, restraints and slot gather, fine mask and robust scale fit (on-device median), reductions and determinism, stream priority and blocking-sync waits, trial slots over cards, DeviceBytes budget, CPU fallback, no Metal; indicative RTX 5080 timings. Pipeline figures: a dashed Refinement box after Structure & map on both, the model branch labelled as deciding nothing; the stills row gains the indexing-ambiguity step before the first merge. RUGNUX_OVERVIEW.md: captions, a refinement paragraph, refined outputs under Write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8645fb2aca1abbbddec29cdfa99056471f8f38a5)	2026-10-26 11:15:54 +01:00
leonarski_f	e406eb17ea	docs: refinement chapter (§15), REFINE_ report keys, acknowledgements and in-source credits ... - New docs/CPU_DATA_ANALYSIS_REFINEMENT.md (§15): the ML amplitude target and its FFT gradient, scale/robust fit/fine solvent mask, D and the variance (work/free), the embedded Monomer Library subset and topology, geometry/metal/KL-B restraints, the minimiser, the weight trials, refined maps, blobs, GPU/CPU, and a "Not yet" list; with its own References. Linked from TECHNICAL.md and CPU_DATA_ANALYSIS.md; §14 no longer says the structure is never refined. - RUGNUX_REPORT.md: a Refinement (REFINE_*) section - every REFINE_ and REFINE_SCORE_ key, the --mode refine input keys and the _blobs.json fields. - RUGNUX_MODEL_AND_REFERENCE.md: "What it does not do yet"; RUGNUX_ADVANCED.md: --model mentions the refinement, --no-refine as in the usage, a --mode refine options table; GPU_DATA_ANALYSIS.md: the refinement's CUDA engines and the missing Metal path. - ACKNOWLEDGEMENT.md: Brunger, Karplus & Petsko 1989 (gradient-balanced starting weight), Nocedal 1980 (two-loop recursion), Charbonnier et al. 1997 (pseudo-Huber), Jiang & Brunger 1994 (flat solvent mask), Kullback & Leibler 1951; all DOIs checked against Crossref. - In-source credit lines: the weight balancing (replacing a "Brunger (1992)-style" line that named the R-free paper), Nocedal 1980, Charbonnier 1997, Jiang & Brunger 1994, Emsley et al. 2010 at FindDensityBlobs, Read 1986 at the refined maps. Comments only. - docs/THIRD_PARTY_NOTICES.md regenerated from the root file with update_version.sh's own step (it lacked the CCP4 Monomer Library row). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 09375dba351bae0cc73e9a6f0f79455dc92b3a10)	2026-10-26 11:07:43 +01:00
leonarski_f	f960bd7a80	Docs landing page: a real diffraction frame in the hero, and the command with a model ... The generated spot-and-ring drawing is replaced by one 0.2 deg rotation frame of the in-house lysozyme standard (EIGER2 16M, SLS X10SA): Bragg spots on their lunes, the beam stop and its arm, the module gaps. docs/landing/hero_frame.py makes the 920 px WebP (67 KB) from the master file - crop about the beam centre, log tone map from the frame's own background, block mean+max downsampling - and is run by hand; the WebP is committed and copied by the landing-page step. The dark theme shows it inverted through a CSS filter. The hero's command line drops its explanatory sentence and gains the second form, rugnux <diffraction image file> <CIF/PDB model file>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ff26000bfe7446cb4fda739c3b72596f8a8b7047)	2026-10-26 10:53:25 +01:00
leonarski_fandClaude Opus 5.5	6da51dc5d7	rugnux: no --refine flag - a placed model is refined by default, --no-refine skips it ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 06:54:15 +01:00
leonarski_f	3882b0f062	Serial stills with a reference or model: crystals their own cells anchor keep that indexing ... The per-image reference/model choice of indexing ignored the crystals' refined cells. On a pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as indexed than after a twin law, the cells decide that law - the rule the data-only resolution already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets the reference choose only among the indexings left open. Exactly merohedral lattices (no metric signal) are unaffected. Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets): pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16 0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged; lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against cells that decide all three laws (96/96/100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f)	2026-10-26 06:49:20 +01:00
leonarski_fandClaude Opus 5.5	ba52aadbce	rugnux: the geometry refinement algorithm is -r, --geom-refine; --refine was declared twice and its no-argument form shadowed it ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-26 06:49:20 +01:00
leonarski_f	a78bfe951d	rugnux: a model and a reference among the inputs, in any order, told by their contents ... `rugnux <images> <model> <reflections>` in any order, without --model or -z. Each positional file is classified by its contents (RugnuxInputKind.h, in JFJochRugnuxOutput - the library still reads no files): an MTZ (magic) or an mmCIF with a _refln / _diffrn_refln loop is reference data (-z; the input itself in --mode refine), a PDB file (a PDB record first, then ATOM/HETATM) or an mmCIF with an _atom_site loop is a model (--model), anything else is images for the readers. CBF (###CBF or _array_data) counts as images; a file whose first line is neither data_ nor a PDB record is decided there, so image files are not read through. An mmCIF holding both atoms and reflections is refused as ambiguous. At most one model and one reference, --model/--mr and -z included. Gzip read through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8f844b0bf86f338d20e8aed426a387722dd8d7cc)	2026-10-26 01:38:25 +01:00
leonarski_f	794db0ca34	Refinement cut by the run's cancel; the hero shows the refined R-work / R-free ... The refinement of a placed model gets the run's cancel flag (RefinementSettings::cancel, through CancelFlag::RunFlag), so the broker's deadline or a cancel - and Ctrl+C in rugnux - cuts a running refinement short with what its finished trials found. RefinementResult::cancelled goes to the live status as rugnux_live_refinement.cancelled (API, regenerated clients) and to the step's note. Hero: once refined, the model card's headline is "R-work / R-free, refined" (e.g. 0.32 / 0.35), its green/red still the placement verdict, the placement and fit in the context line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 395016a30c2bc2aa26fdf2cec2ced9cebe6fad80)	2026-10-26 01:38:25 +01:00
leonarski_f	baec34685d	Rugnux live: the uploaded model is refined after placement; refined scores, model, maps and blobs in the status, the viewer and the hero ... The live run refines a placed model as `rugnux` does (ProcessConfig::refine), after every decision and before the end message; a cut before it starts leaves it out, as the model. The writer still gets the placed model; the refined one stays with the broker. API (broker/jfjoch_api.yaml): - rugnux_live_model.refinement (rugnux_live_refinement): performed, reason, on_gpu, seconds, r_work, r_free, rmsd_bonds_A, rmsd_angles_deg, blob_count, blobs (at most 50, rugnux_live_blob: kind, x, y, z, peak_sigma, volume_A3, nearest_atom, nearest_atom_distance_A) - rugnux_live_status.refined_model - /rugnux/model.cif and /rugnux/map.ccp4 take stage=placed|refined, by default the refined model where there is one; the anomalous map stays on the placed model's phases (the refined coefficients carry none). Regenerated broker/gen, frontend/src/client, redoc, and the python client docs (which also brings the hero fields and rugnux_live_model it was missing). Frontend: the model card keeps its green/red and gets "R-free 0.xx (refined)" as its context line, "refining..." while the step runs; the model section adds refined R-free and blob tiles; Mol* shows the refined model and maps, with the blobs as small spheres coloured by reading (switch, off at first). The Refinement step is in the timeline like any other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3b47fb9a53b7dfad1c1ed289e169094c8d3d9d3e)	2026-10-26 01:38:25 +01:00
leonarski_f	7bf906e4a2	rugnux: a placed model is refined by default - giving a model is the opt-in ... Refinement (Refinement.h) now runs whenever --model or --mr places the model (rigid body fits, or molecular replacement placed it), on the GPU or the CPU: a run that should not spend the time is given no model. --no-refine skips it (for debugging); --refine is still accepted and ignored. A model that was not placed (rejected by the rigid body, not placed by MR - e.g. the battery's wrong-model null) is no longer refined. Processing-inert as before: it runs after every decision; R_WORK/R_FREE and every key outside REFINE_ (section 5), and <prefix>.mtz / _model.cif / _maps.mtz, are unchanged. The block moved after the placement notes so they stay on the step that placed the model. The viewer's processing jobs refine too. The equivalent command line gives --no-refine back where refinement was off. Battery: score.py reads keys exactly, so REFINE_R_FREE cannot alias R_FREE; new report-only fields rfree_refined, rwork_refined, refine_s (part of rugnux_wall_s), shown in the per-set table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit b0bb741831f517b553e59152b3b482a3f75d953a)	2026-10-26 01:38:24 +01:00
leonarski_f	3b06bcee78	Refinement: the trajectory ends at its best step, steps try the positional weight halved too, the final ladder must win ... Three changes to how the trajectory of commit 2bfb84bc4 moves and ends: - A step is taken while its trial lowers R-free or R-work by more than the tie and its R-free stays within the tie of the best so far; the trajectory then ends at its best step. On a deposited model, whose free set it was fitted to, R-work falls while R-free rises, and the trajectory now ends where it began instead of walking on while R-work falls. - Positional steps try the weight halved as well as doubled: at a fixed weight the geometry loosens as the model improves (the X-ray gradient grows with D) until the trials leave the bounds, which stopped 8dqb's trajectories at 2 A (angles > 2.0 deg at w0/128 by step 6). - The final ladder's choice replaces the trajectory's end only where it displaces it (geometry within bounds, R-free lower by more than the tie); before, its first rung was taken by default, two more cycles of movement that on deposited models cost R-free. Deposited models (14 parity sets), R-free after refinement vs refine-combined: mean +0.0058 -> +0.0016 (e.g. 7n2s +0.012 -> +0.003, 9rcs +0.013 -> -0.008, 7d98 +0.015 -> +0.003, 9yl4 +0.010 -> +0.003), the time halved against refine-combined. MD starts, median R-free (REFMAC 0-cycle scorer), previous commit -> this: 5f6m 0.167 -> 0.168, 8dqb 0.294 -> 0.297, 7arr 0.278 -> 0.290 (REFMAC 0.311), 11if 0.244 -> 0.245, 5reo 0.258 -> 0.259, 7d98 0.335 -> 0.335. Determinism: run A == run B == --trials-at-once 1 on 5reo, 9rcs and a 5f6m MD start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c2b55aab1aaff637d9484dec0d25003b5c4a3857)	2026-10-26 01:38:24 +01:00
leonarski_f	bda90dbdd5	Refinement: B restraints in proportion to B (Kullback-Leibler form) with contacts, an own ADP weight, overall B into the atoms ... - B similarity: E = (B_i - B_j)^2 / (kappa^2 B_i B_j), the symmetric Kullback-Leibler divergence of two isotropic Gaussians (Murshudov et al. 2011), so sigma = kappa sqrt(B_i B_j) instead of a fixed 1.5-4.5 A^2. kappa 0.1 / 0.133 / 0.2 / 0.3 for main-chain bonded / angle-related and side-chain pairs (REFMAC's logs list sigma ~0.1 B for main-chain bonded pairs at B 11-140; the ratios are those of the old sigmas, which these equal at B = 15). New class: atoms within 4 A that no bond or angle relates, waters and symmetry mates included, kappa 0.5 (BuildBContacts, sharing the neighbour search with BuildRepulsions). CPU and GPU. - The B factors' X-ray weight is balanced on its own (BalancedADPWeight: B-restraint over X-ray B-gradient on the model with B shaken by 20%) instead of being a multiple of the positional weight; the ladder tries B held and refined at that weight, the trajectory's B steps divide and multiply it by four, positional steps and the final ladder leave it. - Before each B-refining macro-cycle the overall isotropic B of the scale moves into the atoms and the solvent's B (F_model unchanged), so the two do not drift against each other. Tried and not kept: xyz and B as one L-BFGS vector (the B block scaled by its curvature, the positions by the median X-ray curvature): no better than alternation with a memory per block on 12 starts (mean R-free 0.265 vs 0.262), worse on 7arr/7d98; a refit before each B step: no gain for +20-40% time. Refinement battery MD starts, median R-free by the REFMAC 0-cycle scorer, previous commit -> this (REFMAC to convergence): 5f6m x8 0.167 -> 0.167 (0.168), 8dqb x8 0.291 -> 0.294 (0.284), 7arr x4 0.270 -> 0.278 (0.311), 11if x4 0.245 -> 0.244 (0.251), 5reo x4 0.312 -> 0.258 (0.261: the ladder no longer keeps B held where B is the model's largest error), 7d98 x4 0.342 -> 0.335 (0.361). Deposited models: unchanged (B stays held there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ed292f35b482b696d81f71f1d97e0016ca05b144)	2026-10-26 01:38:24 +01:00
leonarski_f	4424d8884d	Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature ... Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)	2026-10-26 01:38:24 +01:00
leonarski_f	6075239be0	Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them ... The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec)	2026-10-26 01:38:24 +01:00
leonarski_f	bcb71136ab	Refinement restraints: metal ions bound to their donors by the model's LINK records ... Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68)	2026-10-26 01:38:24 +01:00
leonarski_f	a218c0bec8	rugnux --mode refine: refine a model against merged structure factors, nothing processed ... rugnux --mode refine --model model.cif -o out data.mtz The input is an MTZ or a structure-factor mmCIF, gzipped or not, by content (writer/RugnuxReflectionFile: parsing in the file layer, the library reads no file). Columns are found, not named: F/SIGF-like amplitudes (calculated amplitudes and map coefficients passed over), else intensities made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value), else rugnux's own 5 % set. RunRefineOnly (rugnux/RugnuxRefine) runs RefineModel; the outputs are --refine's (_refined_model.cif, _refined_maps.mtz, _blobs.json) and a report with the REFINE_ keys, the input's REFINE_FRENCH_WILSON / REFINE_FREE_FLAGS, and every term of the start score under REFINE_SCORE_ (also in --refine's section). --refine-score-only scores the model as given; --refine-d-min cuts the resolution. tools/battery/refine_parity.py runs REFMAC 5.8 at 0 cycles and rugnux --refine-score-only on the same files and prints the terms side by side, with R by resolution from both programs' model amplitudes on the same reflections. Tests: [refine_only] - column choice (observed over map coefficients, intensities, 1-marks-free flags, no flags, no usable column), an MTZ read from its bytes, and the mode end to end on a synthetic cell (French-Wilson, generated flags, score-only, the report keys). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 63ee830c39c988588cffb8c9d4638a678815ec2c) (cherry picked from commit d7397311e6ce0d3bb37ac91e8973b6052e38589f)	2026-10-26 01:38:24 +01:00
leonarski_f	c4d2d419ce	Refinement score: the model's structure factors as REFMAC 5.8 scores them at 0 cycles ... Three terms of the score of a model - the same F_model at every stage, the as-given start and the refined end alike - and a breakdown of it: - Riding hydrogens from the start: the dictionary's hydrogens are added before the model is first scored (REFMAC's 0-cycle restores them; they move R by 0.010-0.013 at 1.1 A). The engine is made once, with them. - The bulk-solvent mask on a grid of its own, min(d_min/3, 0.4 A), at most 2^26 points. On the density grid (1.1 A at 3.3 A) gemmi's 0.8 A shrink found no grid offset and was skipped, which put R 0.015-0.03 high at >= 2.5 A. 0.3 and 0.4 A agree to 0.001. - The overall scale and anisotropic B by robust least squares (pseudo-Huber, iteratively reweighted, c = median |residual|) in the fine pass of the k_sol/b_sol grid (ModelScaleBox::robust, refinement only; MR and validation keep the plain fit). Plain least squares on |F| is the exact LS optimum but lets a few badly-fitted strong reflections set the scale (1.1 A set: R 0.1706 against REFMAC's 0.1600). The scale fit of refinement now runs on the CPU (no ModelScaleGPUPool). RefinementSettings::score_only scores the model as given and stops; RefinementResult::score_start carries every term (k, B, anisotropic B, k_sol, b_sol, mask spacing, R in 20 shells, -lnL work/free, the likelihood FOM, hydrogens; per-reflection terms with score_only). GeometryStatistics gains rmsZ. rugnux_refine_test: --score-only, --dump. Measured against REFMAC 5.8.0431 ncyc 0 (refi bref iso) on the same model, data and flags, 14 open-arm sets 1.1-3.6 A: R-work/R-free within 0.002 on 12 (was +0.003..+0.03); 7n2s and 9yl4 score 0.003/0.005 below REFMAC, which a 0.6-0.8 A mask grid reproduces. REFMAC with ANISOU stripped scores identically, so isotropic B is not a parity term. Open: the likelihood's D/FOM (ours 0.01-0.11 higher at high resolution); geometry differs only by REFMAC's metal LINK restraints and residues outside the embedded dictionary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 4f94b21611874ad9dcf89e6771fe311a0e10bb06) (cherry picked from commit 00db6a6f539ac2da515c0a649474335f4c4b07b8)	2026-10-26 01:38:23 +01:00
leonarski_f	07ae590e54	Serial stills: the indexing ambiguity resolved from the data when no reference or model is given ... Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without -z or --model used to merge the crystals' random indexings together (and warned). The crystals are now made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities, weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent, four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest crystals. Deterministic (fixed summation order, sequential passes). A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals' own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as indexed, those crystals are anchors and only the others are decided by the intensities. Without this, two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350). With a reference or model the per-image choice stands; the same resolution runs as a check, reported as AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS, AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev). --mode scale is not changed. Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with --model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213 0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t 0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes 0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36)	2026-10-25 23:26:32 +01:00
leonarski_fandClaude Opus 5.5	2daa173859	docs: --model on serial stills - breaking the indexing ambiguity is the one thing taken from it before the merge, and why ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-25 20:10:38 +01:00
leonarski_f	cf7676f5d5	Stills merge: outlier rejection on by default (10 sigma), the band linear in the error model's sigma ... The stills merge had no working outlier rejection. --reject-outliers was 0 for stills, and when set its band took the systematic part as a factor exp(+-n s), s the spread of ln(I / median) - on stills a reflection's observations are partials and per-crystal scales, many near zero, so s is wide and the band let through observations hundreds of sigma out. On a high-multiplicity broad-bandpass run with a spotty powder contaminant, single grains in a weak reflection's box were that reflection's whole merged intensity (half-set means -2 vs 2284 on 141 observations); the shells holding the powder rings and the lowest shell merged at CC1/2 0.6-0.75 beside 0.99 elsewhere, at multiplicity ~400-1000. Stills now test the plain linear band, n sqrt(a sigma^2 + (b <I>)^2) about the median - the error model's b is the spread as it is - at n = 10 by default (rotation keeps 6 and its multiplicative band). The stills merge statistics now report the rejected count (OBSERVATIONS_REJECTED was always 0 there). Measured (REFMAC 0-cycle R-free of an isomorphous room-temperature model, depositor's free set, scale mode on stored integrations, vs the same code with --reject-outliers 0): - broad-bandpass in-house stills, 50k frames: 0.318 -> 0.265; 20k: 0.274 -> 0.244; R_split 0.29 -> 0.06, CC1/2 0.79 -> 0.98. 0.1% of the observations are rejected. - monochromatic in-house stills, 30k frames: 0.216 -> 0.212; R_split 0.126 -> 0.087. - n = 6 gains ~0.003 more there but costs the public XFEL sets 0.003-0.006 on three of four; 20 keeps most of the gain. Still battery, 12 non-float sets, against rc180: R_split better on all 12; R-free change mean -0.0005 (range -0.006..+0.003, plus pal86 +0.009 that is within that set's run-to-run spread of ~0.007 - a rerun of rc180's behaviour gave 0.192 against the baseline's 0.185); one draw becomes a win (lcls242_s), nothing else changes class. Rotation is untouched (the stills band and default are gated on the experiment not being a rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3a7142801056f21c720956255f1207e0bf1a8f43)	2026-10-25 19:58:38 +01:00
leonarski_f	7759643d4c	rugnux --refine: refine the placed model, list its difference-map blobs, report under REFINE_ keys ... Opt-in. After the model's placement and everything it decides (setting, indexing, enantiomorph, MR, twin choice), the placed model is refined against the merge it was placed on (RefineModel) and the blobs of the refined mFo-DFc map are listed (FindDensityBlobs). Nothing reads the outcome: the reflection files, the space group and R_WORK/R_FREE of the model as given are those of a run without --refine. Writes <prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz (FP FC PHIC FWT PHWT DELFWT PHDELWT FOM FREE) and <prefix>_blobs.json; the report's model section gains REFINE_* keys (start, placement and restrained R-factors, rmsd bonds/angles, clashes, fixed residues, weights, blob counts by kind) and the five strongest blobs. Documented in RUGNUX_MODEL_AND_REFERENCE.md and CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit aaee92711a205bf6a575a47c4db4555d34e7e8fe)	2026-10-25 07:20:32 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00
leonarski_f	8ff2abf3aa	docs: a standalone front page with the three programs and the downloads, the Sphinx tree one page below it ... docs/landing/ is a self-contained page (HTML, CSS, a little JS, no framework): a hero, a card per program, the download section with the archives of the newest release and the reader's own platform marked, the licence in one paragraph, light and dark. The jfjoch_downloads extension renders it into the HTML output root as index.html at the end of every Sphinx build, so it is served wherever the documentation is - Read the Docs, the gitea-pages branch and the broker's frontend bundle - and its download buttons are the same ones as on the Sphinx download pages, built from the configured release and repointed at the newest published one by the shared script. The Sphinx root page moves to documentation.html (root_doc), and the frontend's documentation frame opens that page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 64cde61f0b941b3e8c99eaa9b74afeaa75aefc78)	2026-10-25 07:20:29 +01:00
leonarski_f	9c543f60b6	docs: home page with the three programs, download pages for rugnux and the viewer, technical reference grouped below ... The home page introduces Rugnux, the viewer and the acquisition system in three cards, with the licence stated once, and the sidebar is regrouped by program: what a user of Rugnux or the viewer needs comes first, and the algorithms, formats, interfaces, hardware and FPGA pages sit under one collapsed "Technical reference" hub (TECHNICAL.md). No page moves, so every existing URL stays. Download pages (RUGNUX_DOWNLOAD.md, VIEWER_DOWNLOAD.md) are rendered by a small Sphinx extension (docs/_ext/jfjoch_downloads.py) from the `release` in conf.py, which update_version.sh rewrites, so no version string on them is maintained by hand; a script then asks the Gitea API for the newest published release and repoints the buttons at its assets, naming any archive that release does not carry. Without network access the build-time links stand. JUNGFRAUJOCH.md is the landing page of the acquisition side with its install pointers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 050442592504ce879dc51d78b6d3d1c58268a800)	2026-10-25 07:20:28 +01:00
leonarski_f	c62a67975e	Polarization plane: measured on the pooled background, refused beyond what a source can give, logged, and kept across passes ... The pre-scan's plane test read per-pixel sector medians of the mean projection. On a large photon-counting detector with a background of a fraction of a count per pixel over the 60 frames read, those medians were zero and the test returned no harmonic at all: on 21 of 255 battery sweeps (EIGER2 16M and JUNGFRAU 4M sweeps, the laboratory EIGER sweeps, the small-molecule sweeps at 20-25 keV) the correlation was NaN or within 0.01 of zero, so the plane could not be decided there. The background is now pooled over 9 x 9 pixels first, as the mask itself already is; 216 of 244 synchrotron sweeps then show their (horizontal) plane past both gates, against 159 of 229 before. Pooling also lifted one long-distance sweep (500 mm, 2theta under 21 deg, polarization can move its background by at most 4 %) over the correlation gate: its background carries a 10 % cos 2phi harmonic along image y, 2.9 times what a fully polarized source would give, on a horizontally polarized beamline. No polarized source exceeds a scale of 1, so a harmonic of twice that or more now decides nothing (PLANE_MODULATION_MAX). The pre-scan logs the correlation and the scale on every run that detects the beam stop (GetPlaneEvidence). A turned plane is kept on the run and re-applied where a pass starts again from an experiment taken before the decision while the shadow is already in the mask (the rotation measured-centre arm, the no-lattice pass), which previously went on with the stated plane. No battery sweep turns; where the plane is not turned the applied coefficient is unchanged. The degree is still never fitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 50a145a4f1ddf6d58332a8ab745627dc36a2dc3d)	2026-10-25 07:20:25 +01:00
leonarski_f	fca2a77b0c	Scaling: the low-resolution limit bounds the scale fit, not the merge ... Reflections coarser than --scaling-low-resolution (default 50 A) are now merged and written - what maps and refinement want of the lowest resolutions - but stay out of every fit of the scale model, by the same path an ice-ring observation takes (excluded from scaling, kept for merging), under a name of its own: RotationScaleMerge's per-observation flag is now `unscaled` (UNSCALED_ICE | UNSCALED_LOW_RES; on the device OBS_UNSCALED = OBS_ON_ICE_FLAG | OBS_LOW_RES_FLAG), set from the reflection on the host and in the device ingest, and only its ice bit marks a merged reflection as on an ice ring. The stills merge drops its low-resolution cut; ScaleOnTheFly, StillsPartialityRefine and the reindexing correlation keep it. The merge statistics run over the whole sphere unless --report-resolution declares a range, so the reflections the beam stop hides count as missing. Which coarse reflections exist at all is the beam-stop shadow test's decision (a7610d847). Why not simply lift the limit: on 5nw5, 35 reflections beyond 50 A bent the scale model enough to move the merged 20-50 A intensities by +61 % and 10-20 A by +19 % (Wilson B 319 -> 382, REFMAC R-free +0.006). Kept out of the fit, the merged intensities at 10-50 A are unchanged to 0.2 % and Wilson B is 307. CPU subset battery against the previous commit (lp-cpu-v2 / lp-cpu-v1): REFMAC 0-cycle R-free 5nw5 +0.0004, 6oel +0.0005, others within 0.0001; rugnux's own R-free/R_model rise where the new >50 A terms enter it (lyso R_model 0.2714 -> 0.2739; the model is poorest there). Kept beyond 50 A: 7qij 61, 5nw5 35, 6oel 14 unique. In-house photon-counting sets unchanged, except insu_I_x06da_ref, where the delta-CC1/2 frame exclusion now removes 449 instead of 78 frames: refres R_meas 0.246 -> 0.167, ISa 33.5 -> 35.4, R_free 0.2443 -> 0.2427, anomalous signal 3.31 -> 3.38 (XDS: R_meas 0.349, ISa 25.1). Stills (lcls242_s/t, lcls222, euxfel83, sacla236) within 0.0002 of the previous commit. GPU (device ingest) and CPU give the same 5nw5 merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 32a9fbcb3ca67879ceb051c13f58d61fe85530f4)	2026-10-25 07:20:21 +01:00
leonarski_f	e97b902603	Beam stop: a reflection whose signal disk reaches the shadow is not integrated; no-data pixels are dead, not shadow ... The profile fit keeps a signal disk that lost pixels to the mask and fits the pixels that remain. That holds for a gap or a dead pixel, but not for the beam-stop shadow: the mask is drawn where the background is dimmed, and the Bragg half-shadow does not end there, so the rest of such a disk is attenuated too. Measured on summed rocking events against their symmetry mates (stratified by resolution and zeta, the detector edge excluded), observations whose disk touches the shadow read low by 12-60 % on 5uth, 7qij, 5nw5, 3r6o, 3meb (CCD and PILATUS alike) and level on the in-house photon-counting sweeps; observations with a clean disk are unbiased except on 5nw5, whose half-shadow reaches ~20 px past the mask. The test sits in MXAnalysisWithoutFPGA::IntegrateOnImage, so rotation, stills, the _process.h5 and --mode scale see the same reflections; with no shadow in the mask (the broker) it does nothing. The shadow finder also reports pixels holding a detector's no-data mark (PILATUS CBF module gaps at -1): they are the darkest pixels there are. The pre-scan now masks those as dead (bit 10) instead of beam-stop shadow (bit 9), so that the shadow test does not drop every reflection clipped by a module gap and the reported shadow share is the shadow's. A mean of zero is not taken as dead: a deep shadow on a still counts nothing in the projected frames. CPU subset battery vs rc180 (REFMAC 0-cycle R-free): 3meb -0.0004, 5uth -0.0004, 6h2p -0.0003, 6oel -0.0002, 3r6o +0.0001, 7qij/7rji 0, 5nw5 +0.002 (its lowest-shell CC1/2 0.78 -> 0.89, ISa 8.4 -> 9.3); in-house refres ISa up 0.1-0.3 on lyso/thau/insu/cytc/myob, half_image refres R_meas 75 -> 64 %, no set lost quality. Stills (lcls242_s/t, lcls222, euxfel83, sacla236): R-free within +-0.002, the noise of a perturbed run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a7610d847694b5540dacc9ef609344ea7c7ae8fd)	2026-10-25 07:20:21 +01:00
leonarski_fandClaude Opus 5.5	672a375b31	v1.0.0-rc.180: VERSION and update_version.sh ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu	2026-10-24 16:57:12 +02:00
leonarski_f	b00c668e3a	v1.0.0-rc.179 (#90) ... * Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer. Reviewed-on: #90 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-24 14:52:29 +02:00
leonarski_f	afbe844b20	v1.0.0-rc.178 (#88) ... * Rugnux: Processes serial crystallography data from XFELs - SACLA (MPCCD), PAL-XFEL and European XFEL (CXI), LCLS (XTC) and SwissFEL Alvra (JUNGFRAU with a CrystFEL module geometry) - with several files or Jungfraujoch master files merged in one run (`rugnux *_master.h5`). * Rugnux: Stills processing takes a known unit cell and space group as a starting point, finds the beam centre (and, with a cell, the detector distance) from the images, detects the saturation level in other facilities' files and refines the placement of individual detector panels. * Rugnux: The stills report gives R_split and CC1/2 from frame half-sets, comparable with CrystFEL. * jfjoch_viewer: Opens the same XFEL file formats. * Validation: A stills battery checks Rugnux against deposited structures from SACLA, PAL-XFEL, LCLS, European XFEL and SwissFEL. * Rugnux: More reliable convergence of rotation scaling, and live molecular replacement starts as soon as the model is uploaded. * jfjoch_broker: Initialize clears the previous dataset's bearer tokens, image buffer and live Rugnux status, and a slow CUDA JIT cache is reported at start. Reviewed-on: #88 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-21 11:20:01 +02:00
leonarski_f	7ed32b899f	v1.0.0-rc.177 (#87) ... * jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes `.ccp4` maps (the map coefficients are in `<prefix>_maps.mtz`). Reviewed-on: #87 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 16:07:31 +02:00
leonarski_f	8261b7354e	v1.0.0-rc.176 (#86) ... * jfjoch_broker: Optional live Rugnux analysis of DECTRIS rotation collections inside the broker, with merged data, maps and the placed model written beside the master file, the rotation lattice in the scan result, and a "Rugnux live" tab and dashboard row in the web frontend. * Rugnux: Faster processing, with molecular replacement and model validation spread over several GPUs and run on the Apple GPU on macOS. * Rugnux: The polarization stated in miniCBF and d*TREK headers is used, twinned crystals' space groups are decided more reliably, and a run that finds no lattice still reports every image. Reviewed-on: #86 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 07:03:28 +02:00