417 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	2e296e6bed	jfjoch_writer: the live refinement's model, maps and blobs are written beside the master file ... In broker live mode the refined model, its map coefficients and the difference-map blobs stayed in the broker for the viewer routes and were lost at the next run. They now go to the writer in the END message as the placed model and maps already do - EndMessage::refined_model (PlacedModelMessage), refined_map_coefficients (MtzTable) and blobs (DensityBlobsMessage, one vector per field) - and the writer writes <prefix>_refined_model.cif (+ .pdb), <prefix>_refined_maps.mtz and <prefix>_blobs.json, the names the CLI gives them. The blobs' JSON is made by one function (DensityBlobsJson) that the CLI now writes through as well, so the two files are byte-identical. Optional CBOR fields, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit c65220d96a96d2b1897f3c6632c94f9379b670a3)	2026-10-26 15:50:40 +01:00
leonarski_f	2461d67e14	NXmx: the unit cell's e.s.d.s are saved as /entry/sample/unit_cell_esd ... Rugnux measured the cell's e.s.d.s (CellEsd) but they only reached the mmCIF's _cell.*_esd, the .ins's ZERR and the report. They now travel in the END message (EndMessage::unit_cell_esd, CBOR key unit_cell_esd, same layout as unit_cell), are written beside the cell in NXsample as unit_cell_esd (float[6]: a b c alpha beta gamma, A and degrees) and are read back into JFJochReaderDataset::unit_cell_esd. rugnux's _process.h5 and, through AddToEndMessage, the broker's live master file get them. Rotation only; absent on stills and where nothing was measured. Optional CBOR field, no version bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6902b5637aa92c27565f22bcf3fe11b5d2536038)	2026-10-26 15:50:40 +01:00
leonarski_f	2be88e11e3	Refinement restraints: contacts with lattice translates are kept, and an atom's own symmetry contact once ... Contacts() dropped every pair found in image 0 with a nonzero lattice shift ("the identity term covers the contact"), but that pair is a contact with a translated copy of j, not with j itself: in P1 no intermolecular repulsion or B-contact restraint was made at all, and in every group the contacts along a, b and c were missing (MeasureGeometry's clash count with them). Such a pair is now kept like a symmetry mate's, from the lower atom of the two. An atom against its own image x -> T x is also met through T^-1 at the same distance (another operator, or the same one at another lattice shift for a screw), and both were kept: atoms near a 3-, 4- or 6-fold or a screw axis were repelled twice. For i == j the image first in fractional coordinates of T x and T^-1 x is kept; an involution is kept once as before. Test: in P1 two waters 2.4 A apart across a cell face get one repulsion and one B contact (none before); a water 1.4 A from a 3-fold axis gets one repulsion with its images (two before). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d6e4d0491e655a49d59b709e1398843506d162a)	2026-10-26 15:38:59 +01:00
leonarski_f	ac58921445	rugnux live: a score that is not finite is left out of the status, not sent as null ... tf_z, rf_z, mr_d_min, the placed model's R-factors, CC, map and anomalous sigmas, the rigid body's move, the refinement's R-factors and rmsds, and the headline d_min are set only where finite (llg_z and null_sigma already were). nlohmann writes NaN/inf as null, which the page's "!== undefined" checks let through to .toFixed(), and which the zod schema rejects for the required blob distance - a blob with no atom within 12 A has an infinite one. That distance is now optional (jfjoch_api.yaml; C++ model and frontend client regenerated), absent then. The model cards test for != null, so a null that arrives anyway does not take the page down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 62ff026aa507e5bda1daa5aa079f32fb5c21f16e)	2026-10-26 15:38:58 +01:00
leonarski_f	3d382ce0a0	Refinement: an exception in a trial slot or in RefineModel no longer ends the run ... A slot builder thread that throws (gemmi's topology or hydrogen placement, bad_alloc) would call std::terminate - rugnux or the broker gone. The slot is now left out as one the card cannot take is, with a warning, and MakeOwnSlot's GPU engine catches std::exception, not only JFJochException. Both callers of RefineModel (after the merge, and --mode refine) catch std::exception: the refinement is reported NOT_PERFORMED with "refinement failed: <what>" and the run writes everything else (cell e.s.d.s, unmerged MTZ, report), as the refinement is processing-inert. Test: --mode refine on a cell too large for the bulk-solvent grid comes back with REFINE= NOT_PERFORMED and the reason, not an exception. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dc4e99fe4b3da2a9a3fa4290b5f70bfcc8c7bdb8)	2026-10-26 15:38:58 +01:00
leonarski_f	0143d57eb0	CellEsdTest: synthetic cells in place of cells that looked measured ... The monoclinic cell of the .ins/mmCIF and fit tests is now 10/7/12, beta 100, and the orthorhombic, hexagonal and tetragonal cells of the sampling tests are rounded synthetic ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 6f964bc0de0ca842f374e6aed83c8848e410173b)	2026-10-26 15:38:58 +01:00
leonarski_f	d8c14046b2	rugnux: unit-cell e.s.d.s - ZERR in a new SHELX .ins beside the .hkl, _cell.*_esd in the mmCIF, CELL_ESD_* in the report ... Rotation runs now estimate the standard uncertainties of the cell. Every strong integrated observation (I > 3 sigma, unclipped, unsaturated, off ice rings) gives |q| = 1/d at its observed centroid, which depends on the detector geometry only; q^2 = h^T G* h is linear in the reciprocal metric, so the components of G* the point group leaves free (projected from the group's rotation parts, any setting) are fitted by weighted linear least squares with 4-sigma rejection, and the GoF-scaled covariance is carried analytically onto a, b, c, alpha, beta, gamma (ties give equal e.s.d.s, fixed angles zero). Least-squares cell e.s.d.s are known to be too small (Herbstein 2000), so the same fit is made on 8 blocks of consecutive frames and their scatter / sqrt(8) is the second estimate; the written e.s.d. is the larger of the two per parameter. Stills get none. Outputs: <prefix>.ins next to every .hkl (TITL, CELL at the wavelength, ZERR with Z = order of the group, LATT/SYMM from the space group - positive LATT only with -1 at the origin - a placeholder organic SFAC/UNIT at 18 A^3 per non-H atom, HKLF 4, END); the mmCIF gains _cell.*_esd ("." where fixed) and _cell_measurement.reflns_used/theta_min/theta_max, and writes the cell to 4/3 decimals; report keys CELL_ESD, CELL_ESD_LSQ, CELL_ESD_BLOCKS, CELL_ESD_BLOCK_COUNT, CELL_ESD_REFLECTIONS, CELL_ESD_THETA_RANGE_DEG (+ dev CELL_ESD_OBSERVATIONS, CELL_DSPACING_FIT). On 13 small-molecule sets the block scatter is 10-40x the least-squares e.s.d. and is what gets written; it lands within ~0.3-5x of XDS's CORRECT.LP e.s.d.s. SHELXT solves from the generated .ins and SHELXL carries ZERR into its CIF su. Costs 12 ms on a small molecule, 0.16 s on 690k lysozyme observations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit e74a3be76261fe1b6c0ff7b80c21afc4768c46cd)	2026-10-26 14:33:38 +01:00
leonarski_f	e6b550fe42	Tests: the stills rocking width - the window at a reflection's resolution, _process.h5 round trip ... StillsPartialityRefine_MeasureRunWidth_WindowAtReflectionResolution, JFJochReader_StillsRunWidth. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dafb8d1bf996d1000741eecf019aee780dd91ebe)	2026-10-26 14:09:32 +01:00
leonarski_f	00662c4110	Refinement: common cofactors and modified RNA nucleotides in the embedded dictionary; heme iron restrained from LINK records ... The embedded CCP4 Monomer Library subset grows from 62 to 84 compounds: - the modified tRNA/rRNA nucleotides PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, M2G, 7MG, 2MG, 5MU (RNA group, joined by the library's backbone link "p"); - the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM, SAH, and dioxygen (OXY). CMO is left out: its dictionary has C-O at 1.25 A, where bound CO is near 1.15 A. links_and_mods.cif gains the links these compounds use: HEC-CYS1/HEC-CYS2 (heme C thioethers), CYS-HEMB/CYS-HEMC, CYS-FAD and HIS-FAD1/2, with their modifications. AddMetalLinks now takes a metal atom of any residue, not only a one-atom ion, so a heme iron's LINK records to His NE2 / Met SD / water become bonds at the library's metal-site distances; the site's coordination counts the metal's bonds in its own dictionary (a heme's four pyrrole N) as well as the records. A metal within a ligand is also a metal for the repulsions (1.9 A to N, O, S) and its dictionary bonds are kept with the metal bonds. Note HEC.cif's own Fe-N(pyrrole) ideal is 1.90 A (HEM: 2.04 A), so a cytochrome c reports a metal-bond rmsd near 0.1 A. Test RefinementRestraints_deposited_ligands builds the topology of eleven deposited models cut down to the ligand and its surroundings (tests/test_data/ligands, 66 KB, from PDB 1EHZ, 8BKH, 6K9I, 2W3V, 5VML, 6F3P, 8CON, 3TK0, 7TCD, 6W4H, 5EPE): every ligand restrained, bond rmsd < 0.03 A, the RNA backbone bonded to each modified nucleotide, the heme Fe-His bonds, and the HEC thioethers. rugnux --mode refine on the full entries against their deposited structure factors: R-free within +-0.003 of the build where these were held fixed, ligand bond rmsd vs the dictionary from 0.004-0.058 A down to 0.002-0.042 A. rugnux grows by 425 KiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f4d407c8de6cb462320edb21a70665350e436d28)	2026-10-26 13:56:29 +01:00
leonarski_f	0829b9a72d	Stills prediction: every frame predicts through two rocking widths of the run, measured first and centred on its radial scale ... Each still predicted through |dist_ewald| <= 2x its own profile radius (the RMS excitation error of its indexed spots): one width at every resolution, differing eightfold between frames. Against the run's measured rocking width that was 5-8 sigma at low resolution on one SACLA run (59 % of the 40-8 A predictions empty) and below one sigma at high resolution on a crowded LCLS run (half of the curve never integrated). Now the stills image pass first integrates a sample (a tenth of the run, at least 2000 frames) the old way, scales it against its own merge and measures the run's rocking curve on it - centre (MeasureRadialScale) and width (MeasureRunWidth), as the merge measures them - then predicts every frame through 2 sigma(d) of that curve, centred on the reciprocal lattice scaled by the radial scale (BraggPredictionSettings::window_*, CPU and CUDA). The post-refinement uses the same curve instead of measuring it again: re-measured through windows that follow it, the constant term falls to ~0 and the result swung by up to +-0.015 per run. Still battery, 12 non-float sets, against the st-partial baseline (0-cycle REFMAC R-free vs deposited): 9 better, 3 worse, sum of changes -0.013; the off-centre run -0.012, worst +0.008 on one JUNGFRAU run (its sample measures a different width form than the full run). Integrated reflections 0.8-1.7x, battery wall +13 % on a loaded box. Rotation untouched (window fields zero unless set); p.hkl byte-identical on three rotation sets. Tests: BraggPrediction_WindowFollowsResolution, BraggPrediction_CPU_GPU_consistency_window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 1b28d671d780bc88525ceac5120017963d937bbc)	2026-10-26 10:50:52 +01:00
leonarski_f	3882b0f062	Serial stills with a reference or model: crystals their own cells anchor keep that indexing ... The per-image reference/model choice of indexing ignored the crystals' refined cells. On a pseudo-merohedral cell, where at least 90% of the crystals' cells fit the run's cell better as indexed than after a twin law, the cells decide that law - the rule the data-only resolution already uses (AnchorOnCells, factored out of ResolveReindexReferenceFree and shared). The resolver now keeps each image's CC per indexing (IntegrationOutcome::reference_cc); before the stills merge, AnchorReferenceReindex puts every image back as indexed, applies the cell rule, and lets the reference choose only among the indexings left open. Exactly merohedral lattices (no metric signal) are unaffected. Still battery (REFMAC 0-cycle R-free, model before -> anchored; data-only in brackets): pal90 0.1883 -> 0.1834 (0.1832), alvra213 0.2073 -> 0.2011 (0.2013), alvra213_int16 0.2081 -> 0.2021 (0.2024); euxfel83 0.2037, euxfel152 0.2452, sacla237, lcls222 unchanged; lcls242_t 0.2847 -> 0.2952 (0.2942): there the model had moved about half the crystals against cells that decide all three laws (96/96/100%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 85c804676ffd15ffd66e2ba05928000d6f4e765f)	2026-10-26 06:49:20 +01:00
leonarski_f	a78bfe951d	rugnux: a model and a reference among the inputs, in any order, told by their contents ... `rugnux <images> <model> <reflections>` in any order, without --model or -z. Each positional file is classified by its contents (RugnuxInputKind.h, in JFJochRugnuxOutput - the library still reads no files): an MTZ (magic) or an mmCIF with a _refln / _diffrn_refln loop is reference data (-z; the input itself in --mode refine), a PDB file (a PDB record first, then ATOM/HETATM) or an mmCIF with an _atom_site loop is a model (--model), anything else is images for the readers. CBF (###CBF or _array_data) counts as images; a file whose first line is neither data_ nor a PDB record is decided there, so image files are not read through. An mmCIF holding both atoms and reflections is refused as ambiguous. At most one model and one reference, --model/--mr and -z included. Gzip read through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8f844b0bf86f338d20e8aed426a387722dd8d7cc)	2026-10-26 01:38:25 +01:00
leonarski_f	794db0ca34	Refinement cut by the run's cancel; the hero shows the refined R-work / R-free ... The refinement of a placed model gets the run's cancel flag (RefinementSettings::cancel, through CancelFlag::RunFlag), so the broker's deadline or a cancel - and Ctrl+C in rugnux - cuts a running refinement short with what its finished trials found. RefinementResult::cancelled goes to the live status as rugnux_live_refinement.cancelled (API, regenerated clients) and to the step's note. Hero: once refined, the model card's headline is "R-work / R-free, refined" (e.g. 0.32 / 0.35), its green/red still the placement verdict, the placement and fit in the context line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 395016a30c2bc2aa26fdf2cec2ced9cebe6fad80)	2026-10-26 01:38:25 +01:00
leonarski_f	baec34685d	Rugnux live: the uploaded model is refined after placement; refined scores, model, maps and blobs in the status, the viewer and the hero ... The live run refines a placed model as `rugnux` does (ProcessConfig::refine), after every decision and before the end message; a cut before it starts leaves it out, as the model. The writer still gets the placed model; the refined one stays with the broker. API (broker/jfjoch_api.yaml): - rugnux_live_model.refinement (rugnux_live_refinement): performed, reason, on_gpu, seconds, r_work, r_free, rmsd_bonds_A, rmsd_angles_deg, blob_count, blobs (at most 50, rugnux_live_blob: kind, x, y, z, peak_sigma, volume_A3, nearest_atom, nearest_atom_distance_A) - rugnux_live_status.refined_model - /rugnux/model.cif and /rugnux/map.ccp4 take stage=placed|refined, by default the refined model where there is one; the anomalous map stays on the placed model's phases (the refined coefficients carry none). Regenerated broker/gen, frontend/src/client, redoc, and the python client docs (which also brings the hero fields and rugnux_live_model it was missing). Frontend: the model card keeps its green/red and gets "R-free 0.xx (refined)" as its context line, "refining..." while the step runs; the model section adds refined R-free and blob tiles; Mol* shows the refined model and maps, with the blobs as small spheres coloured by reading (switch, off at first). The Refinement step is in the timeline like any other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3b47fb9a53b7dfad1c1ed289e169094c8d3d9d3e)	2026-10-26 01:38:25 +01:00
leonarski_f	7bf906e4a2	rugnux: a placed model is refined by default - giving a model is the opt-in ... Refinement (Refinement.h) now runs whenever --model or --mr places the model (rigid body fits, or molecular replacement placed it), on the GPU or the CPU: a run that should not spend the time is given no model. --no-refine skips it (for debugging); --refine is still accepted and ignored. A model that was not placed (rejected by the rigid body, not placed by MR - e.g. the battery's wrong-model null) is no longer refined. Processing-inert as before: it runs after every decision; R_WORK/R_FREE and every key outside REFINE_ (section 5), and <prefix>.mtz / _model.cif / _maps.mtz, are unchanged. The block moved after the placement notes so they stay on the step that placed the model. The viewer's processing jobs refine too. The equivalent command line gives --no-refine back where refinement was off. Battery: score.py reads keys exactly, so REFINE_R_FREE cannot alias R_FREE; new report-only fields rfree_refined, rwork_refined, refine_s (part of rugnux_wall_s), shown in the per-set table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit b0bb741831f517b553e59152b3b482a3f75d953a)	2026-10-26 01:38:24 +01:00
leonarski_f	6369a263b1	Refinement: one trajectory of short weight trials instead of a ladder of full refinements from the start ... Every rung of the old ladders (6 positional, then 3 B) was a full 10-cycle refinement from the placed model, so ~90% of the evaluations were thrown away and the kept trajectory had 100 iterations. The run is now: 1. a ladder from the placed model, 2 macro-cycles per trial: three positional weights (w0/1024, /128, /16), each with the B factors held and refined (adp = 4 w), chosen by the old rule (tighter first, a later trial wins only within the geometry bounds and by more than the R-free tie); 2. a trajectory of steps from where the last one ended, each two trials side by side - the weights as they are and the X-ray term raised, in turn the positions' (x2, at most w0/8) and the B factors' (x4) - until a step lowers neither R-work nor R-free by more than the tie (at most 8 steps); 3. a final ladder from there at x1/2, x1, x2. A macro-cycle is 30 positional and 20 B iterations (the minimiser's memory kept within a trial). A trial starts from the scale, solvent and likelihood fit its predecessor ended with (the same computation, so result-neutral), and the slots that run share the threads. On 5 battery MD starts (start 1 of 5f6m 1.09 A, 8dqb 2.04, 7arr 0.92, 11if 1.37, 7d98 3.32; R-free by rugnux's scorer, REFMAC 0-cycle within 0.002), previous commit -> this, against REFMAC run to convergence (40-100 cycles): 5f6m 0.215 -> 0.167 (REFMAC 0.168), 8dqb 0.341 -> 0.290 (0.284), 7arr 0.356 -> 0.289 (0.288), 11if 0.295 -> 0.236 (0.244), 7d98 0.393 -> 0.337 (0.360). Time 5.4 / 8.6 / 7.3 / 3.9 / 10.0 s on the RTX 5080 (base 6-7 s). Halving a weight during the trajectory never won in the logs, so steps only try raising it; the final ladder can lower it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 2bfb84bc4f3c72e8da37e54b728839b65a636519)	2026-10-26 01:38:24 +01:00
leonarski_f	4424d8884d	Refinement: an own L-BFGS whose memory outlives the macro-cycle, B steps scaled by their curvature ... Each macro-cycle of a weight trial used to start a fresh Ceres solve: the L-BFGS memory was dropped every 10 (positions) or 5 (B) iterations and every cycle began with a steepest-descent step. RefinementMinimiser keeps one memory for the positions and one for the B factors across a trial's macro-cycles (the target's scale and likelihood parameters are refitted in between, which the memory tolerates), with a backtracking Armijo line search and Oren-Shanno scaling of a diagonal initial inverse Hessian. The diagonal: for B, the X-ray curvature per atom - the likelihood's curvature in |F_model| per resolution shell against the atom's (s^2/4)^2 f^2 exp(-B s^2/2), calibrated once per trial by a central-difference probe along a random direction - plus the B restraints'. For the positions none: tried with the restraints' Gauss-Newton diagonal (bonds and angles, ~1e4 per axis against the X-ray's ~1e2) it slowed the minimiser, since that stiffness is in directions that move bonded atoms together. Same budget (10 cycles x 10 + 5 iterations, same ladder), median R-free (REFMAC 0-cycle scorer) from the refinement battery's MD starts, base -> this: 5f6m (8 starts) 0.250 -> 0.210, 8dqb (8) 0.387 -> 0.345, 7arr 0.383 -> 0.356, 11if 0.325 -> 0.295, 5reo 0.316 -> 0.326 (CA RMSD 0.48 -> 0.43), 7d98 0.416 -> 0.393; REFMAC converged: 0.168 / 0.284 / 0.311 / 0.251 / 0.261 / 0.361. Deposited models (14 parity sets): R-work lower, R-free +0.0035 on average (their free sets were fitted to; the minimiser now moves the bottom rung further). Time unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit dfba4c1bd3c9345f31b64fd1cadc805810960825)	2026-10-26 01:38:24 +01:00
leonarski_f	6075239be0	Refinement likelihood: D on the working set, the measurement error per component, as REFMAC 5.8 estimates them ... The likelihood's variance per bin (beta) is fitted by maximum likelihood on the free reflections as the whole spread of Fo about D Fc, and D is then fitted on the working reflections with the variance eps beta + 2 sigma^2 (acentric; + sigma^2 centric), which is also the target's variance. Murshudov et al. (2011) Acta D67 355: D from the working set, the variance from the free set. The measurement error is thereby counted twice, so D and the figure of merit fall where sigma is a large part of the spread - at high resolution. Measured on REFMAC 5.8.0431 0-cycle output (black box) of the 14 parity sets: its FOM inverted per reflection gives var = eps a + 2 sigma^2 (acentric, fitted coefficient 2.00) and eps a + sigma^2 (centric); its D (FC_ALL/FC_ALL_LS = the log's SigmaA_Fc1) and a are reproduced by this estimate (on REFMAC's own Fc, one set to rms 0.009 in D over all 20 bins). The previous per-bin free-set ML put D at 0.39-0.48 where REFMAC has 0.06-0.13 in the outermost bins. rugnux score vs REFMAC 0-cycle, 14 sets: mean |FOM difference| 0.056 -> 0.012 (max 0.112 -> 0.042), per-bin D rms 0.107 -> 0.068; R-work/R-free unchanged (D does not enter them). The score's FOM is now the mean over the working reflections, as REFMAC reports it, and the refined model's maps are weighted by the refinement's own m and D rather than a separate sigma_A estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d00d635e2874bff6cd216b7f24367362d0cc2e56) (cherry picked from commit b93e0ea2fc1ffcc6bb40dae9b77816a321f4d6ec)	2026-10-26 01:38:24 +01:00
leonarski_f	bcb71136ab	Refinement restraints: metal ions bound to their donors by the model's LINK records ... Metal sites were held only by the 1.9 A metal contact of the repulsions; REFMAC restrains the metal LINKs of the coordinate file (its documentation), which was the one geometry term the 0-cycle comparison did not share (one 2.8 A set: bond rmsd 0.018 against REFMAC's 0.042). - monomers/metals.cif: the Monomer Library's own statistics of metal sites (metals.json, LGPL data as the rest of monomers/), cut by update_monomers.py to the metal ions of its compound list and O, N, S donors: mean distance, spread and number of sites per metal, donor element and coordination number. - AddMetalLinks (before prepare_topology): every LINK / struct_conn record between a metal ion and an O, N or S that no link of the dictionary describes (its ZN-CYS, ZN-HIS, FE-CYS, FE-HIS are taken as they are) gets a link of one bond at EmbeddedMetalDistance for the two elements and the ion's coordination - the number of its donor records, symmetry mates included - from the rows resting on at least 20 sites (the coordination's own row, else the neighbours interpolated), sigma their spread (>= 0.02 A); a His or Cys donor loses the proton the bond replaces (DEL-HD1 / DEL-HE2 / CYSmod1, as the library's links do). Records only, no distance search; records to a symmetry mate count for the coordination but are not restrained (bonds to symmetry mates are not, anywhere). - The metal bonds are a restraint set of their own (RefinementTopology::metal_bonds), evaluated with the geometry and reported apart (GeometryStatistics::rmsd_metal, REFINE_RMSD_METAL_BONDS, REFINE_SCORE_RMSD_METAL_BONDS / _BONDS_ALL): their sigma is the spread of real sites, so they sit further from ideal than covalent bonds, and the weight ladder's 0.02 A bound stays a bound on the covalent ones. refine_parity.py compares REFMAC's bond rmsd (links included) with the combined one, and takes --gpu. Score-only geometry against REFMAC 5.8 0-cycle, all bonds (count): 2.8 A Zn/Fe set 0.0402 (4934) vs 0.042 (4935), was 0.0178; a Zn-Cys set 0.0186 (7102) vs 0.019 (7102); a Ca set 0.0297 vs 0.026 (REFMAC also refines the sugars and uses one Ca-O distance); a Mg set 0.0114 vs 0.017 (REFMAC restrains 8 symmetry links and puts Mg-O at 1.99 A, the library's statistics at 2.07). R of the scored model unchanged (+0.0001 on the first from the removed His/Cys protons). Refined R-free with the links vs without: +0.0016, -0.0002, +0.0005, +0.0004. Test: RefinementRestraints_metal_links (Zn-Cys4 by the dictionary's link, Mg(H2O)6 by the new ones, both back at their distances). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0fcffc5c71e03e184cff18f07524473462a57b68)	2026-10-26 01:38:24 +01:00
leonarski_f	9da3851844	Refinement: at most n weight trials at a time, and a cancel flag the caller can set ... - RefinementSettings::max_trials_at_once (rugnux_refine_test --trials-at-once): 0 as many as the ladders and the cards take, 1 one after another - the check that side by side changes nothing. Verified: refined model and report identical, side by side vs one after another, GPU (5reo, 9yl4, 6o2h, 9rcs) and CPU engine (6o2h, 9rcs, 5reo), and run to run. - RefinementSettings::cancel, an atomic flag for the broker's live deadline: checked at every iteration of the L-BFGS (positions, B, rigid bodies, group B: the minimiser stops where it stands), at every macro-cycle and before every weight trial. The trials that ran are chosen among as usual; cut before the first, the model is the placement's. RefinementResult::cancelled says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f708d3d227bb42215ffe3d9a15c766dbd73b783c)	2026-10-26 01:38:24 +01:00
leonarski_f	46ad2c10ed	Refinement GPU: the robust scale fit on the card, the pool back in every trial slot ... The score's robust fine pass (pseudo-Huber IRLS of ModelScaleBox::robust) on the device, so that refinement on a GPU no longer fits its scale on the host every cycle - with the trials side by side that was a robust grid search on one thread per slot (13-133 s per refinement on the seven timing sets, against 1.7-8 s before the score commit). - ModelScaleGPU: the weights multiply the WSSR and J^T J / J^T r in scale_sums (a weight row per fit, reserved at the first robust fit); robust_residuals / robust_weights kernels; the median of each fit's first residuals taken on the host as FitModelScale takes it (nth_element at n/2, on the fits' threads); LevMarBatch split out of FitBatch so that the rounds restart from where a fit stands. FitSolvent(..., robust) refits the coarse winner in the fine pass, as FitModelScale's robust fit does; the plain fit is unchanged. - ModelScaleGPUPool::Fit(..., robust); its streams at the highest priority: a fit is a chain of small launches the host waits on, and on a card shared with the trials' long kernels each wait queued behind them (refit.scale 58-87 ms -> 14-45 ms per cycle, the result the same to the bit). - Refinement: the main context and every trial slot fit on their pool (robust); the CPU engine keeps FitModelScale's robust fit. The card's fit is FitModelScale's to rounding (float residuals and weights, the steps' anisotropic factor in float), deterministic run to run and whichever engine a fit takes: R-work/R-free of the 14 parity sets on the GPU equal the CPU engine's to 4 digits (largest difference 1.4e-6 on the 3.6 A set). Test: ModelScaleGPUPool_RobustMatchesFitModelScale (one reflection in fifty a 4x outlier, five cells). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit d15f048cbd7ed2d6b430f9f9cb5018ca284da8f0)	2026-10-26 01:38:24 +01:00
leonarski_f	a218c0bec8	rugnux --mode refine: refine a model against merged structure factors, nothing processed ... rugnux --mode refine --model model.cif -o out data.mtz The input is an MTZ or a structure-factor mmCIF, gzipped or not, by content (writer/RugnuxReflectionFile: parsing in the file layer, the library reads no file). Columns are found, not named: F/SIGF-like amplitudes (calculated amplitudes and map coefficients passed over), else intensities made into amplitudes by French-Wilson, else the mean of the anomalous pairs; the free set from a FreeR_flag-like column (0 the test set, or the minority value), else rugnux's own 5 % set. RunRefineOnly (rugnux/RugnuxRefine) runs RefineModel; the outputs are --refine's (_refined_model.cif, _refined_maps.mtz, _blobs.json) and a report with the REFINE_ keys, the input's REFINE_FRENCH_WILSON / REFINE_FREE_FLAGS, and every term of the start score under REFINE_SCORE_ (also in --refine's section). --refine-score-only scores the model as given; --refine-d-min cuts the resolution. tools/battery/refine_parity.py runs REFMAC 5.8 at 0 cycles and rugnux --refine-score-only on the same files and prints the terms side by side, with R by resolution from both programs' model amplitudes on the same reflections. Tests: [refine_only] - column choice (observed over map coefficients, intensities, 1-marks-free flags, no flags, no usable column), an MTZ read from its bytes, and the mode end to end on a synthetic cell (French-Wilson, generated flags, score-only, the report keys). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 63ee830c39c988588cffb8c9d4638a678815ec2c) (cherry picked from commit d7397311e6ce0d3bb37ac91e8973b6052e38589f)	2026-10-26 01:38:24 +01:00
leonarski_f	07ae590e54	Serial stills: the indexing ambiguity resolved from the data when no reference or model is given ... Where the space group is given (-S) and its lattice admits alternative indexings, a stills run without -z or --model used to merge the crystals' random indexings together (and warned). The crystals are now made consistent with each other before anything is scaled (ReferenceFreeReindex): each takes the indexing in which it agrees best with all the others (Brehm & Diederichs 2014), agreement being the sum over shared reflections of the products of shell-normalised, per-crystal standardised intensities, weighted by 1/(1+(sigma/sd)^2). The sum over all pairs is (|M|^2 - const)/2 for M the running sum, so a pass over the crystals is one pass over the reflections rather than over all pairs: coordinate ascent, four seeded starts, stop when <= 1 crystal in 1000 moves, the overall relabel that moves the fewest crystals. Deterministic (fixed summation order, sequential passes). A twin law of a pseudo-merohedral cell whose axes differ by a few per cent is decided by the crystals' own refined cells, which the indexer has already used: where >= 90% of the crystals' cells fit better as indexed, those crystals are anchors and only the others are decided by the intensities. Without this, two public sets got worse (pal90 R-free 0.183 -> 0.207, lcls242_t 0.296 -> 0.350). With a reference or model the per-image choice stands; the same resolution runs as a check, reported as AMBIGUITY_DATA_DISAGREES. New report keys (section 2): INDEXING_AMBIGUITY_RESOLVED, AMBIGUITY_CRYSTALS, AMBIGUITY_REINDEXED, AMBIGUITY_SEPARATION_SIGMA, AMBIGUITY_DECIDED_FRACTION (+ AMBIGUITY_TIME_S, dev). --mode scale is not changed. Still battery, no --model (R-free, REFMAC 0-cycle against the deposited model; before -> after; with --model for comparison): euxfel83 0.321 -> 0.207 (0.204), euxfel152 0.346 -> 0.257 (0.245), alvra213 0.209 -> 0.201 (0.207), alvra213_int16 0.210 -> 0.202 (0.208), pal90 0.183 -> 0.183 (0.188), lcls242_t 0.296 -> 0.294 (0.285); controls sacla237 / lcls222 unchanged (no ambiguity, no-op). The step takes 0.2-1.6 s on 2.7k-54k crystals; synthetic 64k crystals x 200 reflections 0.5 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 80d378a1fe92364fd5b99a17f5c9ee8498412d36)	2026-10-25 23:26:32 +01:00
leonarski_f	64bbc44e11	fast-feedback-indexer vendored in image_analysis/indexing/fast-feedback-indexer instead of fetched ... The GPU known-cell indexer behind FFBIDX is now built from an in-tree copy of upstream commit a94fd4f5c55289e70860feea358182d2072e02b5 (the archive the FetchContent block pinned), so it can carry local fixes. No behaviour change. Copied: indexer/src/{indexer.cpp, indexer_gpu.cu, log.cpp} and the ffbidx/ headers the build uses (indexer.h, indexer_gpu.h, refine.h, log.h, exception.h, envvar.h), each with a header line naming the upstream URL, commit and licence; LICENSE.md beside them; and the library's test spot lists (data/simple/files/image{0..8}_{local,peakfinder8,radial}.txt) that FFBIDXIndexerCPU_MatchesGPU reads, now found in-tree. ffbidx/version.h is written once ("1.1.3-a94fd4f") instead of generated - the generator ran git in the parent build, so it used to stamp Jungfraujoch's own hash. The in-tree target keeps the name fast_indexer_static and what upstream's indexer/src/CMakeLists.txt gives the static library: the source list, CUDA_RUNTIME_LIBRARY Static, cxx_std_17 (PUBLIC), the source directory as PUBLIC include, CUDA::cudart_static and Threads::Threads (PRIVATE), Eigen3::Eigen (PUBLIC, the project's), the CUDA-architecture default only where none is set, and -Xcompiler=/Zc:preprocessor for the CUDA sources under MSVC (CUB needs the conforming preprocessor). Left out: the shared library, the C API (c_api.cpp, unused), install/pkg-config rules, the version_txt generator target, the top-level project's data/tests/python/examples subdirectories, and -DVECTOR_CANDIDATE_REFINEMENT=1 (no source reads it). Licences: COLLECT.sh takes the text from the vendored LICENSE.md; the THIRD_PARTY_NOTICES row moves from the fetched to the vendored table. CLAUDE.md's list of FetchContent downloads no longer names it. Verified: clean configure and build of rugnux and jfjoch_test; FFBIDXIndexerCPU_MatchesGPU and the Portable_FFBIDXIndexerCPU tests pass; a 3000-image stills run gives the md5-identical merged .hkl with the fetched and the vendored library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0e4fcb87de0fbb7c0022750ff4d62d8c5a5c3c18)	2026-10-25 19:58:38 +01:00
leonarski_f	cf7676f5d5	Stills merge: outlier rejection on by default (10 sigma), the band linear in the error model's sigma ... The stills merge had no working outlier rejection. --reject-outliers was 0 for stills, and when set its band took the systematic part as a factor exp(+-n s), s the spread of ln(I / median) - on stills a reflection's observations are partials and per-crystal scales, many near zero, so s is wide and the band let through observations hundreds of sigma out. On a high-multiplicity broad-bandpass run with a spotty powder contaminant, single grains in a weak reflection's box were that reflection's whole merged intensity (half-set means -2 vs 2284 on 141 observations); the shells holding the powder rings and the lowest shell merged at CC1/2 0.6-0.75 beside 0.99 elsewhere, at multiplicity ~400-1000. Stills now test the plain linear band, n sqrt(a sigma^2 + (b <I>)^2) about the median - the error model's b is the spread as it is - at n = 10 by default (rotation keeps 6 and its multiplicative band). The stills merge statistics now report the rejected count (OBSERVATIONS_REJECTED was always 0 there). Measured (REFMAC 0-cycle R-free of an isomorphous room-temperature model, depositor's free set, scale mode on stored integrations, vs the same code with --reject-outliers 0): - broad-bandpass in-house stills, 50k frames: 0.318 -> 0.265; 20k: 0.274 -> 0.244; R_split 0.29 -> 0.06, CC1/2 0.79 -> 0.98. 0.1% of the observations are rejected. - monochromatic in-house stills, 30k frames: 0.216 -> 0.212; R_split 0.126 -> 0.087. - n = 6 gains ~0.003 more there but costs the public XFEL sets 0.003-0.006 on three of four; 20 keeps most of the gain. Still battery, 12 non-float sets, against rc180: R_split better on all 12; R-free change mean -0.0005 (range -0.006..+0.003, plus pal86 +0.009 that is within that set's run-to-run spread of ~0.007 - a rerun of rc180's behaviour gave 0.192 against the baseline's 0.185); one draw becomes a win (lcls242_s), nothing else changes class. Rotation is untouched (the stills band and default are gated on the experiment not being a rotation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 3a7142801056f21c720956255f1207e0bf1a8f43)	2026-10-25 19:58:38 +01:00
leonarski_f	d3673d17fc	RefinementTest: the GPU restraint check on an engine made for a cell its gather takes ... The restraints do not read the cell; the 14 A cell that makes the symmetry contacts is too narrow for the gather's boxes, so the engine was refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a6e00e10ce1798a97d1f054c7553cc9eb6b5cf4b)	2026-10-25 07:20:32 +01:00
leonarski_f	b4b68d368f	Restrained refinement on the GPU: ML amplitude target, FFT gradient, dictionary restraints, R1 + R2 driver, blobs ... RefineModel (Refinement.h) refines a placed model against the merge, reading and writing nothing: - placement stage (R1): every chain of 50+ atoms as a rigid body at once (L-BFGS on six parameters per chain, their gradient from the per-atom one through Ceres jets), then one B shift per residue; - restrained stage (R2): positions and isotropic B of every atom of a residue the embedded dictionary describes, the rest held fixed but kept in Fcalc and the mask; riding hydrogens added by gemmi's topology, moved with their parents, re-placed every macro-cycle. Macro-cycles refit the mask, the scale and bulk solvent (FitModelScale / ModelScaleGPUPool) and the likelihood's D and beta (ML on the free set in resolution bins), rebuild the repulsion pairs (symmetry mates included), then L-BFGS on positions and, where chosen, on B. - weights: a ladder of full refinements at multiples of the gradient-balanced X-ray weight, chosen by R-free within bond/angle bounds (ties to the tighter geometry), then the B refinement tried at its own weights and kept only where it beats B held by more than the tie. The target and its gradient (RefinementXray, RefinementGPU): the Rice/centric amplitude likelihood (RefinementMath.h, shared host/device code), the gradient by Agarwal's FFT method - the adjoint of the reciprocal-space symmetry composition into a half-l (CPU) or half-u (cuFFT) spectrum, one inverse real transform, then each atom's density derivatives (xyz and B) summed over its box. On the GPU everything per evaluation stays on the device: ModelDensityGPU's gather, r2c, a fused compose + likelihood kernel, a CSR gather for the adjoint (no atomics), c2r, one warp per atom for the gradient; the restraints are one kernel per restraint plus a per-atom gather of gradient slots; cub reductions. Deterministic. Restraints (RefinementRestraints): bonds, angles, torsions (with esd, no phi/psi, no alternative sugar puckers), chiral volumes, least-squares planes from gemmi's Topo; repulsions beyond 1-4 (vdW radii from the dictionary's energy library, 2.6 A for H-bond pairs, 1.9 A metal-N/O/S); B similarity of bonded and angle-related atoms. DensityBlobs (D1): gemmi's flood-fill blobs of mFo-DFc above 3 sigma peaking above 4 sigma, placed beside the model and labelled WATER / ION / LIGAND / MODEL_ERROR by distance, contacts, volume and the anomalous map. tools/rugnux_refine_test refines a model against a finished run's p.mtz (developer utility, not installed). Tests: likelihood derivative, the embedded library, restraint values on ideal residues and every restraint kind's gradient against finite differences (CPU, and the GPU against the CPU), the X-ray gradient against finite differences in P1, P212121, C2 and P6122 (CPU and GPU), and a synthetic refinement that recovers a shaken model. Credits in docs/ACKNOWLEDGEMENT.md and at the algorithms. Measured on the open arm's battery outputs against REFMAC 5.8, 10 cycles, same p.mtz, free set and start model (R-free): 5reo 0.2228 vs 0.2236, 8dyz 0.1733 vs 0.1651, 9chw 0.1995 vs 0.1969 (CPU engine). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 8efe6f2322b3bfdae580bb34f256e67730dfc3c2)	2026-10-25 07:20:32 +01:00
leonarski_f	2fb4b1d015	Rugnux live page: a hero panel with resolution, ISa, space group and whether the model was placed ... Under the header of the Rugnux live page, four big-number cards in the manner of jfjoch_viewer's merge statistics window: Resolution (d_min of the merge), ISa, Space group (short symbol, screw axes subscripted) and the model. The model card shows the fit over random orientations - R-work of the placed model below its random-orientation null, in sd - which is the test rugnux places a model by, both for molecular replacement (after the TFZ/LLG-Z gate) and for a model that fits where it is. The card is green where rugnux placed the model (placement MR or RIGID_BODY), red where it did not (with MR's reason), grey where no model was uploaded; no threshold of its own. "—" while the run is going, "searching…" during the MR step. /rugnux/status gains `space_group`, `d_min` and `isa` (set with the result; absent until then, isa also where the error model did not determine it), filled from ProcessResult in RugnuxThread with the values the summary's Space group, Resolution and Signal rows print. Generated code regenerated from jfjoch_api.yaml: broker/gen (cpp-pistache-server 7.20.0), the TypeScript client (npm run openapi) and broker/redoc-static.html. Python client not regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit fcc019707751aab9d8c9f349309eeadcf364b9c2)	2026-10-25 07:20:28 +01:00
leonarski_f	78d3b7ef75	Stills geometry: fit only the given cell's crystal form; a stray frame is no cell population ... RefineStillsGeometry fitted distance, beam and cell on the strongest sampled frames whatever their cell, so a sample holding two crystal forms put the cell, and with it the distance, between them - while the merge later keeps only the given cell's population (GivenCellPopulation). The sampled frames are now filtered by the same population before the bundle and the module placement are formed, the population told from the strong frames (>= 20 indexed spots), whose cells show the valley between two forms; a weak frame's cell at the start geometry spreads across it. A run of one form has no valley and keeps every frame. GivenCellPopulation: a stray frame or two far out in a population's tail is a local maximum of the histogram with no valley of its own, so its range was everything, both forms included, and its mean - the mean of all frames - could sit nearer the given cell than the given form's own and win. A population is now represented only by its highest peak. Test CellPopulation_StrayFramesBeyondTheTail (fails before, passes after). Still battery, against the same code without the change: the two runs of a two-form sample -0.0019 and -0.0015 R-free (0-cycle REFMAC; the second repeated three times, -0.0013..-0.0021, run noise ~0.001), R_split 10.6 -> 9.8 % on the first; single-form controls fit the identical geometry (R-free within 0.0003). Rotation does not call either path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ada1b08003fa9748175cce381fc8f28569f28296)	2026-10-25 07:20:25 +01:00
leonarski_f	c62a67975e	Polarization plane: measured on the pooled background, refused beyond what a source can give, logged, and kept across passes ... The pre-scan's plane test read per-pixel sector medians of the mean projection. On a large photon-counting detector with a background of a fraction of a count per pixel over the 60 frames read, those medians were zero and the test returned no harmonic at all: on 21 of 255 battery sweeps (EIGER2 16M and JUNGFRAU 4M sweeps, the laboratory EIGER sweeps, the small-molecule sweeps at 20-25 keV) the correlation was NaN or within 0.01 of zero, so the plane could not be decided there. The background is now pooled over 9 x 9 pixels first, as the mask itself already is; 216 of 244 synchrotron sweeps then show their (horizontal) plane past both gates, against 159 of 229 before. Pooling also lifted one long-distance sweep (500 mm, 2theta under 21 deg, polarization can move its background by at most 4 %) over the correlation gate: its background carries a 10 % cos 2phi harmonic along image y, 2.9 times what a fully polarized source would give, on a horizontally polarized beamline. No polarized source exceeds a scale of 1, so a harmonic of twice that or more now decides nothing (PLANE_MODULATION_MAX). The pre-scan logs the correlation and the scale on every run that detects the beam stop (GetPlaneEvidence). A turned plane is kept on the run and re-applied where a pass starts again from an experiment taken before the decision while the shadow is already in the mask (the rotation measured-centre arm, the no-lattice pass), which previously went on with the stated plane. No battery sweep turns; where the plane is not turned the applied coefficient is unchanged. The degree is still never fitted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 50a145a4f1ddf6d58332a8ab745627dc36a2dc3d)	2026-10-25 07:20:25 +01:00
leonarski_f	6ee1a5759e	rugnux: a rotation scale is integrated with pass 1's lattice refit at it, and walked to the refit's fixed point ... Two defects let a stage turning 1.9% slower than its stored angles cost 9qw8 half its signal (R-free on the depositor's flags 0.346 against their 0.251, ISa 5.2, c 100.24 against the deposited 100.92): - Pass 1's lattice, integrated wherever the refined-geometry re-index finds less (Pass1Fallback), kept the cell and orientation pass 1 fitted at the stored angles. The rate error sits partly in that cell, so the adopted scale was integrated with a cell that undoes it. The lattice is now refit (cell + orientation, detector and axis held, no re-index) on the validation frames of the whole sweep at the angles the pass runs at, and kept where it indexes no fewer of them. - The walk took a refit step only where the validation frames significantly preferred it. A first pass indexes through a degree of drift late in the sweep, so the frames score 0.98772 and 0.98198 alike while integration at the first loses half the signal. Off the stored angles a refit step is now refused only where the frames score it significantly worse; leaving the stored angles still needs significant evidence. 9qw8 (CPU, battery protocol): walk 0.98772 -> 0.98087, c 100.94, ISa 5.2 -> 11.2, R_meas 0.270 -> 0.157, CC1/2 0.975 -> 0.993, R-free on the depositor's flags 0.346 -> 0.267 (ratio 1.381 -> 1.066; forcing --rotation-scale 0.9816 gives 0.255). The other eight sets that adopt a scale (2wnq, 2wnn, 6ttn, 8r5r, 8pqd, 8xte, 9i80, 9gdj) move by at most 0.0006 in R-free; the photon-counting controls lyso_x06da_ref, thau_x10sa_16keV, cytc_x10sa, 5reo and 3mc4/3r6o are unchanged (lyso p.hkl byte-identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 5daa4ed635b22e29b25e3e02f8ad8c058f9176a2)	2026-10-25 07:20:25 +01:00
leonarski_f	fca2a77b0c	Scaling: the low-resolution limit bounds the scale fit, not the merge ... Reflections coarser than --scaling-low-resolution (default 50 A) are now merged and written - what maps and refinement want of the lowest resolutions - but stay out of every fit of the scale model, by the same path an ice-ring observation takes (excluded from scaling, kept for merging), under a name of its own: RotationScaleMerge's per-observation flag is now `unscaled` (UNSCALED_ICE | UNSCALED_LOW_RES; on the device OBS_UNSCALED = OBS_ON_ICE_FLAG | OBS_LOW_RES_FLAG), set from the reflection on the host and in the device ingest, and only its ice bit marks a merged reflection as on an ice ring. The stills merge drops its low-resolution cut; ScaleOnTheFly, StillsPartialityRefine and the reindexing correlation keep it. The merge statistics run over the whole sphere unless --report-resolution declares a range, so the reflections the beam stop hides count as missing. Which coarse reflections exist at all is the beam-stop shadow test's decision (a7610d847). Why not simply lift the limit: on 5nw5, 35 reflections beyond 50 A bent the scale model enough to move the merged 20-50 A intensities by +61 % and 10-20 A by +19 % (Wilson B 319 -> 382, REFMAC R-free +0.006). Kept out of the fit, the merged intensities at 10-50 A are unchanged to 0.2 % and Wilson B is 307. CPU subset battery against the previous commit (lp-cpu-v2 / lp-cpu-v1): REFMAC 0-cycle R-free 5nw5 +0.0004, 6oel +0.0005, others within 0.0001; rugnux's own R-free/R_model rise where the new >50 A terms enter it (lyso R_model 0.2714 -> 0.2739; the model is poorest there). Kept beyond 50 A: 7qij 61, 5nw5 35, 6oel 14 unique. In-house photon-counting sets unchanged, except insu_I_x06da_ref, where the delta-CC1/2 frame exclusion now removes 449 instead of 78 frames: refres R_meas 0.246 -> 0.167, ISa 33.5 -> 35.4, R_free 0.2443 -> 0.2427, anomalous signal 3.31 -> 3.38 (XDS: R_meas 0.349, ISa 25.1). Stills (lcls242_s/t, lcls222, euxfel83, sacla236) within 0.0002 of the previous commit. GPU (device ingest) and CPU give the same 5nw5 merge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 32a9fbcb3ca67879ceb051c13f58d61fe85530f4)	2026-10-25 07:20:21 +01:00
leonarski_f	e97b902603	Beam stop: a reflection whose signal disk reaches the shadow is not integrated; no-data pixels are dead, not shadow ... The profile fit keeps a signal disk that lost pixels to the mask and fits the pixels that remain. That holds for a gap or a dead pixel, but not for the beam-stop shadow: the mask is drawn where the background is dimmed, and the Bragg half-shadow does not end there, so the rest of such a disk is attenuated too. Measured on summed rocking events against their symmetry mates (stratified by resolution and zeta, the detector edge excluded), observations whose disk touches the shadow read low by 12-60 % on 5uth, 7qij, 5nw5, 3r6o, 3meb (CCD and PILATUS alike) and level on the in-house photon-counting sweeps; observations with a clean disk are unbiased except on 5nw5, whose half-shadow reaches ~20 px past the mask. The test sits in MXAnalysisWithoutFPGA::IntegrateOnImage, so rotation, stills, the _process.h5 and --mode scale see the same reflections; with no shadow in the mask (the broker) it does nothing. The shadow finder also reports pixels holding a detector's no-data mark (PILATUS CBF module gaps at -1): they are the darkest pixels there are. The pre-scan now masks those as dead (bit 10) instead of beam-stop shadow (bit 9), so that the shadow test does not drop every reflection clipped by a module gap and the reported shadow share is the shadow's. A mean of zero is not taken as dead: a deep shadow on a still counts nothing in the projected frames. CPU subset battery vs rc180 (REFMAC 0-cycle R-free): 3meb -0.0004, 5uth -0.0004, 6h2p -0.0003, 6oel -0.0002, 3r6o +0.0001, 7qij/7rji 0, 5nw5 +0.002 (its lowest-shell CC1/2 0.78 -> 0.89, ISa 8.4 -> 9.3); in-house refres ISa up 0.1-0.3 on lyso/thau/insu/cytc/myob, half_image refres R_meas 75 -> 64 %, no set lost quality. Stills (lcls242_s/t, lcls222, euxfel83, sacla236): R-free within +-0.002, the noise of a perturbed run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit a7610d847694b5540dacc9ef609344ea7c7ae8fd)	2026-10-25 07:20:21 +01:00
leonarski_f	925701a1ce	ScalingSettings::MinPartiality: validate the argument, not the value it replaces ... Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit df5f2ae22f979a5569f4d569fe2a8cdd5999d242)	2026-10-25 07:20:21 +01:00
leonarski_f	2dbb9d5951	Stills partiality: merge only partials of p >= 0.5, adopt the refined model on every crystal, centre the rocking curve on the run's radial scale ... Three changes to the stills post-refinement and merge, decided on 0-cycle REFMAC R-free against the deposited models (diff = ours - deposited data, noise ~0.002): - Stills merge only observations whose model partiality is at least 0.5 (RugnuxDefaultScalingSettings; rotation keeps 0.02, and --min-partiality still overrides). Diagnosis on per-observation dumps: the empirical rocking curve has a floor of 5-15 % of its peak out on the tail (mis-oriented crystals), and dividing such observations by a small model p makes them biased, not noisy. Variance terms in (1-p) or (1-p)/p recovered a third of what the cut does; a narrower (correct-looking angular) width or a Lorentzian form was worse. On one set in scale mode the optimum is flat over 0.3-0.6. - The CC adoption gate in RefineOne is gone: it left 25-33 % of crystals at p = 1 next to the refined ones, and collapsed to 3 % on a run whose curve is off-centre. - The run's radial scale of the reciprocal lattice (a wavelength or cell a little off, invisible to spot positions since they fix only lambda d*) is measured from where the curve is centred against d*^2 (MeasureRadialScale, iterated three times) and applied to q in the width measurement and the post-refinement. One run had its curve centred at +2-3e-3 d*^2 (scale -1.6e-3 measured); the others measure below 7e-4. --mode scale on stills now starts where the full pipeline starts: p = 1, the excitation error recomputed from the stored lattice (the window was read as zero before, so no run width was ever measured), and collapsed scales rejected after the refinement. Still battery, the 12 non-float sets, change of diff vs the previous build: -0.0035 mean, 9 better, three within +-0.0006 except one +0.0053 (an edge win becoming a draw); the off-centre run -0.005 here and -0.017 in an earlier run of the same algorithm (that set varies by ~0.01 between runs). Rotation p.hkl byte-identical on three sets. Test StillsPartialityRefine_MeasureRadialScale added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0d85e5b975ba863828bd17ebc3b5308e9aa294ce)	2026-10-25 07:20:21 +01:00
leonarski_f	128830d15e	rugnux live status: the model's scores, and whether the data carry anomalous signal ... /rugnux/status gains `model` (rugnux_live_model) - placement (RIGID_BODY/MR/NOT_PLACED), R-work and R-free of the placed, unrefined model, CC(model, data), R-work over its random-orientation null, mean 2mFo-DFc at the atoms, the rigid body's move, molecular replacement's decision, group, copies, TFZ/RFZ/LLG-Z and search resolution, and the strongest anomalous site - taken from ProcessResult::model_validation and ::model_prior (LiveModel). `anomalous_signal` says the summary's "Anomalous" row reports usable signal (CC_anom > CC_ANOM_SIGNAL, now a named constant shared with the report), so the viewer can show the anomalous map by default only then. The step that placed the model (Molecular replacement, or Model validation) now carries notes: what MR placed and its TFZ/RFZ, R-work/R-free (unrefined) and CC(model, data), the rigid-body move. Generated code regenerated from jfjoch_api.yaml: broker/gen (cpp-pistache-server), the TypeScript client (npm run openapi) and broker/redoc-static.html. Python client not regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit f602e931dc6afcccb3767b8c6df87650d9496033)	2026-10-24 19:06:04 +02:00
leonarski_f	789e064e6a	Stills scaling: a model error in the per-crystal scale fit, and the rocking width measured per window group ... Two changes to the stills scale model; rotation is untouched (p.hkl md5 identical on two rotation sets). The per-crystal scale fits (ScaleOnTheFly and the partiality post-refinement's G and tilt) measured residuals in counting sigmas. The strong reflections of a still scatter about the model by tens of percent (robust sd of ln(I/<I>) 0.45-0.8 by shell on one run), so with a Cauchy loss at 3 counting sigma every strong reflection was an outlier and the scale was set by the weak ones: re-fitting each frame's scale against the final merge moved it by a factor 0.45-1.5 (5-95%). The fit's variance is now the expected one at the model, background + counting + (0.3 * model)^2. The coefficient is not delicate: 0.1-1.0 fit alike in an offline replay; what matters is that the term is there. The run's rocking width came from the intensity-weighted moment of the excitation error over all frames pooled. Frames predict through windows of their own that differ by an order of magnitude, and the widest windows dominated both moments: on a run whose windows are mostly narrower than the curve at high resolution the measurement read the window at every resolution, 2.8e-4 1/A with no angular term, several times too wide at low resolution (low-partiality observations there merged at 0.1-0.3 of their reflection's mean). The moment is now taken per group of frames of similar window (4 per shell), each against its own window, and the shell takes the median of the groups whose curve is narrower than their window. On that run the width becomes 6.7e-5 1/A + 0.018 deg angular; where the pooled measurement was already right (a SASE-bandwidth run), it stays where it was. 0-cycle REFMAC R-free against the deposited models, 9 still battery sets (6 used while developing, 3 held out), change from rc179: -0.0100, -0.0095, -0.0057, -0.0055, -0.0051, -0.0045, -0.0030, -0.0024, +0.0009 (held out: -0.0055, -0.0051, +0.0009). The width change alone accounts for most of the largest gain; the scale-fit variance for 0.000-0.003 per set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit ccc0e01f56827bbe0648ed7c5b8049912d80650f)	2026-10-24 16:57:12 +02:00
leonarski_f	b00c668e3a	v1.0.0-rc.179 (#90) ... * Rugnux: Stills processing needs only the unit cell and space group beyond what the file records - detector distance, beam centre, tilt, panel placement, saturation level, gain and polarisation plane are found from the images - and is up to twice as fast. * Rugnux: Stills can be indexed without a GPU, with a CPU implementation of the fast-feedback indexer (macOS, computers without a GPU). * Rugnux: Rotation and stills data are integrated in the detector's own units with one gain model, improving data from CCD detectors. * Web frontend: The Rugnux live page shows the placed model with its electron density and anomalous difference map in a Mol* viewer. Reviewed-on: #90 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-24 14:52:29 +02:00
leonarski_f	afbe844b20	v1.0.0-rc.178 (#88) ... * Rugnux: Processes serial crystallography data from XFELs - SACLA (MPCCD), PAL-XFEL and European XFEL (CXI), LCLS (XTC) and SwissFEL Alvra (JUNGFRAU with a CrystFEL module geometry) - with several files or Jungfraujoch master files merged in one run (`rugnux *_master.h5`). * Rugnux: Stills processing takes a known unit cell and space group as a starting point, finds the beam centre (and, with a cell, the detector distance) from the images, detects the saturation level in other facilities' files and refines the placement of individual detector panels. * Rugnux: The stills report gives R_split and CC1/2 from frame half-sets, comparable with CrystFEL. * jfjoch_viewer: Opens the same XFEL file formats. * Validation: A stills battery checks Rugnux against deposited structures from SACLA, PAL-XFEL, LCLS, European XFEL and SwissFEL. * Rugnux: More reliable convergence of rotation scaling, and live molecular replacement starts as soon as the model is uploaded. * jfjoch_broker: Initialize clears the previous dataset's bearer tokens, image buffer and live Rugnux status, and a slow CUDA JIT cache is reported at start. Reviewed-on: #88 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-21 11:20:01 +02:00
leonarski_f	7ed32b899f	v1.0.0-rc.177 (#87) ... * jfjoch_broker: Faster live Rugnux analysis, with the GPU indexers prepared at the start of a collection and kept between collections; Rugnux no longer writes `.ccp4` maps (the map coefficients are in `<prefix>_maps.mtz`). Reviewed-on: #87 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 16:07:31 +02:00
leonarski_f	8261b7354e	v1.0.0-rc.176 (#86) ... * jfjoch_broker: Optional live Rugnux analysis of DECTRIS rotation collections inside the broker, with merged data, maps and the placed model written beside the master file, the rotation lattice in the scan result, and a "Rugnux live" tab and dashboard row in the web frontend. * Rugnux: Faster processing, with molecular replacement and model validation spread over several GPUs and run on the Apple GPU on macOS. * Rugnux: The polarization stated in miniCBF and d*TREK headers is used, twinned crystals' space groups are decided more reliably, and a run that finds no lattice still reports every image. Reviewed-on: #86 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-19 07:03:28 +02:00
leonarski_fandFilip Leonarski	19b135186d	v1.0.0-rc.175 (#85) ... * Rugnux: Processes a collection live while jfjoch_broker writes it (`rugnux <master.h5> <broker URL>`). * Rugnux: Processing starts on the first images and confirms its choices once all are in, so analysis can begin during collection. * Rugnux: Faster processing, especially on servers with several GPUs and CPU sockets and on computers without a GPU (`--no-gpu`, macOS). * Rugnux: Uses much less GPU memory, so large data sets stay on the GPU of smaller cards, and a data set too large for the card is processed on the CPU instead. * Rugnux: On macOS, Bragg integration runs on the Apple GPU. * Rugnux: Better handling of twinning and pseudo-symmetry, with the twin hypothesis written as `<prefix>_twin_<SG>.mtz`, and a wrong `--model` no longer confirms the data or decides the hand. * Rugnux: `--model` accepts predicted (AlphaFold) models and falls back to molecular replacement when the model is not placed as a rigid body (experimental `--mr` forces it), settling space group, hand and twinning and writing a different answer as `<prefix>_mr.mtz`. * Rugnux: Molecular replacement is faster and settles the space group in more cases with several copies in the asymmetric unit. * Rugnux: Deposition-ready mmCIF, a richer report, and numbered steps on the terminal with the full log in `<prefix>_log.txt`; `_P1.mtz`, `--estimate-beam-center` and `--beam-center-search` are removed. * Rugnux: The beam-stop shadow is found correctly on tilted detectors. * Rugnux: Stills files that record a stationary goniometer axis get the default stills geometry refinement. * jfjoch_viewer: "Analyze dataset" uses the same defaults as a bare `rugnux <file>`, and shows the results of processed CBF, marCCD and SMV sweeps. * jfjoch_viewer: The Files panel refreshes on F5 and every few seconds, and the plot type and other selections stay put during live collection. * jfjoch_writer: Memory stays bounded and old data files are closed efficiently on long runs, any failure ends the run in an error state the broker is told about, and the master file appears only after its data files; the ZeroMQ pusher logs when a stalled writer makes it drop images. * Fixes: The TCP image pusher no longer blocks the broker's receiver, GPU out-of-memory failures stop with a clear message, and the XDS plugin masks module and chip edges. --------- Co-authored-by: Filip Leonarski <jungfrau@filips-mac-mini.home> Reviewed-on: #85 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-16 15:34:43 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00