35 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	30c7bee15f	rugnux: --af-model becomes --mr, for predictions and experimental structures alike ... - The option is --mr <file>. What kind of model it is is read from the file: a prediction where it is ModelCIF (_ma_qa_metric) or has no cell and one B value per residue in 0-100 (AlphaFold PDB/mmCIF); an experimental structure otherwise. A prediction is handled as before (pLDDT >= 70 trim, B from pLDDT, PAE domains where the AlphaFold DB's PAE file sits beside it). An experimental structure is taken as its polymer - no waters, ligands, hydrogens or alternative conformers, occupancy 1, isotropic B as refined - with a coordinate error of 0.67 A (a prediction's at pLDDT 90) for sigma_A; a B column above 100 no longer stops the search. - The report section is MOLECULAR REPLACEMENT (EXPERIMENTAL), its keys MR_* (MR, MR_MODEL_FILE, MR_MODEL_KIND, MR_DECISION, MR_SPACE_GROUP, MR_COPIES, MR_HAND_CHECK, MR_SECONDS, ...); the <prefix>_mr outputs are unchanged. - The twin hypothesis' merge that is made only for the model to test (a metric re-ask promoted the indexed lattice's group) is written and reported only where the model chooses the subgroup. Raw deposited coordinates of nine control sets, unconverted: every one placed and confirmed against the deposition (all chains as one body), the groups the same as with the prediction or the prepared model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 19:30:22 +02:00
leonarski_fandClaude Opus 5.5	0d33d7a40a	rugnux --af-model: likelihood judging, PAE domains, twin comparison, tNCS carry test, faster multi-copy search ... - Likelihood (Read 2001; Z-score over random positions as McCoy et al. 2005): log-likelihood gain of Rice/Woolfson E distributions, sigma_A from the pLDDT coordinate error with its scale refined, to 2.5 A (or the 20000 lowest-resolution reflections). Further-copy TF peaks ranked by the gain on the search's reflections (peaks that lower it dropped); a copy whose TF Z is below 8 also stands out where its refined likelihood Z-score reaches 8 - only where the copies placed are as many as Matthews makes most likely. The likelihood is computed only where a translation function does not stand out. Copy structure factors exact from one FFT of the cell's grid, or read off the molecular transform where it reaches the likelihood's resolution. - PAE domains: the AlphaFold DB's predicted_aligned_error JSON beside the model (same name pattern) is read automatically; average linkage on the symmetrised PAE (10 A, domains >= 40 residues). Where the whole model is placed nowhere, each domain is searched by itself side by side; placed domains must agree, and a domain alone may only confirm the answer's group. - Twin comparison: where the answer's merge chose and the twin hypothesis' merge exists, the chosen placement read as a crystal in the subgroup (its symmetry images as copies, the subgroup's setting found from the two cells) against the model searched for afresh in the subgroup, both against the data twinned by the law (fraction refined on the working set, Yeates 1997); the subgroup is taken by a 0.03 R-free lead plus its own null/margin -> TWIN_SUBGROUP, written to p_mr with law and fraction. With --af-model rugnux makes the subgroup merge wherever a metric re-ask promoted the indexed group. - tNCS rule: a group with fewer copies beside tNCS (screws bridged by the vector, or the enantiomorph) no longer leaves the choice undecided where the chosen copies, carried into it and refined, fit worse by 0.03 in R-free. - Speed: the molecular transform's FFT on the GPU (cuFFT, serialised), no rigid body for trials that lower the likelihood, re-search only with tNCS, PAE domains searched in parallel. Survey (193 sets, MR replay on stored merges): single GREEN 58 -> 68, multi 48 -> 58, partial 8 -> 9; 4bwl (and, in full runs, 2wnq and a withdrawn entry of the same protein) TWIN_SUBGROUP P21 instead of a wrong AGREES; a P21212 set with tNCS now SPACE_GROUP P21212 instead of UNDECIDED; no wrong call. CPU and GPU builds make the same decisions on 18 demo/multi-copy/twin sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 13:21:51 +02:00
leonarski_fandClaude Opus 5.5	fcbe92042b	Merge branch 'af-mr3' into rc175: --af-model searches several copies per asymmetric unit (Matthews probability, sequential translation function, fixed-copy rigid body, tNCS) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 01:36:26 +02:00
leonarski_fandClaude Opus 5.5	63e0a85467	rugnux --af-model: several copies in the asymmetric unit, tNCS, a search-matched null, faster ... Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies are searched one after another up to the largest copy number with a Matthews probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode and mean, no resolution dependence); COPIES_LIKELY reports the most probable. Sequential search. Copy n's translation function carries copies 1..n-1 as a known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation). Its peaks, past those that clash, are ranked by the correlation of the whole assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed copy onto an origin shift), the first that stands out (TF Z >= 8), packs and lowers R-free by 0.01 is kept; each copy is refined beside the others held fixed (RefineRigidBody's fixed copies), and the first copy again beside the second, the earlier ones once more at the end. Packing: ClashFraction counts C-alpha clashes with other copies and all their mates, binned (7mzt: 51 s of clash checks -> 1.9 s). Search quality: rotation-function peaks hill-climbed off the 6 deg sampling (6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the first copy goes down the TF list until three placements that pack are refined, so a group is CLASH only where no offered orientation packs; groups are searched side by side, and every group the best one leads is searched again with the best one's copy orientations tried first, the better search kept. tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies' orientations join every further copy's search; and where the tNCS vector turns one group's screw axes into another's (TncsRelatesGroups), a group whose search stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three copies to within 0.01 in R-free, and where one hand's search failed the other was taken as ENANTIOMORPH; now UNDECIDED. Null: the replicates are placed by the translation function in their own orientation, as the search placed the real one (a null at the real centroid reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw null, the gate --model's MODEL_FIT is now read at. Speed: placements scored on the GPU structure-factor engines; further copies' trials score with the first copy's bulk solvent held, the assembly refitted at the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080 shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s. Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 00:27:17 +02:00
leonarski_fandClaude Opus 5.5	ee350f1f74	rugnux --model: a wrong model confirms no shell, fits no hand, and fits the GPU reserve ... Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins): CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections, but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24 wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells). The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL 0.05-0.13 in every shell). The report no longer says a model cannot agree by accident. MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and the best random placement fits as well as it does, and drawing the null about the cell axes instead changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up. The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and a deposited model settles one at +4.15. Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones count them by the d of the index in the model's ASU, and a model's group need not keep the data's metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died with "zone over the engine's reserve". Counted the zones' way, under every indexing probed. Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric (fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	50e3b59fb7	rugnux --af-model: hand check reads only the anomalous map, on MR's placement and solvent ... The ENANTIOMORPH hand check ran a whole ValidateAgainstModel on the placed model: model read and frame probe, a rigid body, a 9-replicate null (~0.5 s of the ~0.6-0.9 s), sigma_A, three maps written. The decision needs only --model's rule: the model fits its null (the MR choice has already required its own null of the same construction) and the anomalous map at the model's atoms is not inverted. ReadModelAnomalousMap (ModelValidation) computes F_calc and F_mask to the data's resolution (GPU engine where there is one), carries them through the bulk solvent MR scored the placement with (the overall scale and anisotropic B do not move phases), and reads the anomalous map with the same code as the validation: its coefficient construction and its reading at the atoms are factored out of Validate (anomalous_coefficients, read_anomalous_map) and used by both, so plain --model is unchanged. Hand check 0.6/0.6/0.9 s -> 0.027/0.028/0.040 s on 11if/7ris/9zmu; same ADOPTED/VETOED on all seven cases tried (the three demos, two anomalous lysozyme sets and their mirrored-detector runs), with the scatterer means and deepest troughs within 0.1 sigma of the full validation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 18:11:01 +02:00
leonarski_fandClaude Opus 5.5	2a3d60fb82	rugnux --af-model: anomalous veto on the enantiomorph, search on the GPU, cheaper rescoring ... - An MR ENANTIOMORPH decision is taken into the answer's label only as --model takes a model's enantiomorph: ValidateAgainstModel runs on the placed model (in a scratch directory, removed after) and its adopted_model_enantiomorph - fit over its null, anomalous map at the atoms not inverted - decides; otherwise ENANTIOMORPH_VETOED. The hand-check numbers go to the MODEL PRIOR section. - Rotation and translation functions on the GPU (MolecularReplacementGPU.cu): one block per orientation for the rotation function; per orientation the copies, the bin weights, a gather of the Crowther-Blow coefficients (CSR lists, fixed order, no atomics) and a cuFFT C2R, block-reduced statistics finished on the host in a fixed order. The CPU path (FFTW) is unchanged in what it computes; the orientation-independent part of the translation function (TranslationSetup) is now built once per group on either path. - Rescoring: a placement clashing with its mates over twice the limit is not refined, one rigid-body engine per group, and the null is built only for the candidate the choice falls on. Measured MR step on the RTX 5080 (before -> after, seconds; hand check extra where it runs): 3r6o 1.77 -> 0.67, neg 2.14 -> 0.37, 11if 1.20 -> 0.51 (+0.6 hand check), 7ris 1.46 -> 0.56 (+0.6), 9zmu 4.71 -> 1.07 (+0.9). Rotation 0.50 -> 0.02 s and translation 1.17 -> 0.13 s on 9zmu. Mirrored-detector runs of two anomalous lysozyme sets: the hand check vetoes (map inverted at the model's atoms); the unmirrored runs adopt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 17:10:48 +02:00
leonarski_fandClaude Opus 5.5	60a8bf3981	rugnux --af-model (experimental): single-copy molecular replacement of a predicted model ... A predicted model (AlphaFold mmCIF, pLDDT in the B column) is trimmed at pLDDT 70, given B-factors from the pLDDT (Oeffner et al. 2022) and placed by molecular replacement, one copy in the ASU: - rotation function on a uniform 6 deg sampling of the point group's rotation ASU, each orientation scored by the correlation of E^2 with the symmetry-summed |F|^2 read off the model's molecular transform, computed once by FFT and interpolated (Navaza 1994); - full-symmetry Crowther & Blow translation function (one FFT per orientation and group, through MapFromFPhi/FFTW) for the 50 best orientations in every group with the merge's rotations; - the best three placements per group through RefineRigidBody (GPU where there is one), bulk solvent and R-free to 3 A, a C-alpha packing check against symmetry mates, Matthews, and the random-orientation null for a placed candidate. Decision: PLACED = TF Z >= 8, clash <= 5%, Vm in [1.5, 4), null >= 3 sd. Within the answer's merge the placed group with the lowest R-free is chosen where it leads every other packing group by 0.03; the twin hypothesis' merge is searched, and chosen, only where no group of the answer's merge can hold the model at all (Matthews, packing). An enantiomorph relabels the answer as --model does; another group goes to <prefix>_mr.mtz/.cif plus <prefix>_mr_model.pdb; the MODEL PRIOR report section carries the table. The search runs on the CPU in both builds; without --af-model nothing changes. Measured (RTX 5080, MR step): 3r6o twin I41 chosen over I4122 (Matthews + clash) in 1.8 s, solution 0.09 A CA rmsd from Phaser's; 11if P41 -> P43 1.2 s; 7ris P3121 -> P3221 1.7 s; 9zmu P6122 -> P6522 4.7 s; 11if's model against 3r6o's data NOT_PLACED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 13:35:05 +02:00
leonarski_fandClaude Opus 5.5	40fcaa985c	Global Wilson B: minimum fit window, no upper cap ... CalcGlobalWilsonB fits ln<I> against 1/d^2 from 4 A to d_min. With d_min just under 4 A that window is a sliver, and after the shells past the signal limit are dropped it held 0-2 shells, so the estimate was NaN (or a fit over almost no range); and any B above 200 A^2 was discarded, which is every crystal diffracting to 4-7 A. Now the window is never narrower in 1/d^2 than the 4-3 A window a 3 A data set gets: a narrower one is widened towards low resolution (d_min 3.15 A -> from 4.39 A, 3.61 A -> 5.96 A, 3.87 A -> 7.40 A; data coarser than ~4.4 A use everything, as before). The 200 A^2 cap is dropped for the dataset-wide estimate (kept for the per-image one). The window is logged, printed with the statistics and reported (WILSON_B_RANGE_A, developer key); the correlation is reported as before. Data with d_min <= 3 A are unchanged. Report only: the value feeds the statistics text, the log, the report and the mmCIF _reflns.B_iso_Wilson_estimate; nothing in scaling, merging or the MTZ reads it. Measured on the open arm (replica on the a897b08 merged data, confirmed by rugnux runs): 12 sets change, 7 previously NaN now finite - 5nw5 315, 9z44 493, 6r72 302 A^2 (were above the cap; model mean B 328/310/114, deposited Wilson 388/322/179), 9yzk 210 (model 383, ctruncate 243, xtriage 104), 7qij 104 (model 194, ctruncate 150), 9yl4 78 (model 146), 8qq7 35 at 4 shells (model 197; its signal ends well inside the written 3.15 A). 6yqf 132 (corr 0.51) -> 96 (corr 0.69; model 57, ctruncate 145, xtriage 114); 9rcs 69 -> 148 (model 74, ctruncate 131). At 3-4 A the estimators themselves disagree by 1.5-3x, so these are better-defined numbers, not exact ones. p.hkl and MTZ data identical to the base (e34a5fb88) on every set compared (ci tier, private arm, the Wilson sets). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 02:21:32 +02:00
leonarski_fandClaude Opus 5.5	fe181ca042	Anisotropy verdict: the directional limits are a second hypothesis beside the tensor ... The verdict gated only deltaB_linear, the Debye-Waller part of the tensor, against the data set's own systematic floor. A deficit that does not follow s^2 (a FLAT signature) can stop the signal far earlier in one direction and still read NOT_DETECTED: one open-arm set reported directional limits of 3.18/3.18/2.14 A beside NOT_DETECTED. The cone limits (<I/sigma> = 2 in 20 deg cones) were computed but never consulted. Now each statement is decided on its own evidence and either establishes anisotropy: - tensor: unchanged gate; - directional limits: at the best direction's limit the weakest cone's <I/sigma> is below the best cone's by more than 3.5 of their joint standard error (both read linearly in s^2 between shell centres, as the limits are), the weakest limit is measured (not censored at the data edge), and the limits are at least 0.25 A apart - the spread the report already calls "noticeable". Why the magnitude term: with the standard errors alone (the counting statistics of the cones) the weakest cone is "significantly" below the best on 172 of 216 battery sets, down to limits 0.5% apart (measured with a replica of ConeLimits on the a897b08 battery's merged data). The cones differ systematically (multiplicity and coverage depend on direction), so significance alone made DETECTED near-universal. A refusal (CANNOT_DETERMINE: dose ramp, short sweep, triclinic, no unmerged data) is not overruled - its reasons bias the cones too - but now reports the measured directional limits. Report: ANISOTROPY_DETECTED_BY (TENSOR/DIRECTIONAL/BOTH/NONE); developer keys for the two cones' <I/sigma> and errors; the summary row names the test that carried the verdict, and CANNOT_DETERMINE rows give the directional limits; the mmCIF verdict comment says so. Report only: no intensity, flag or decision downstream reads the verdict (the French-Wilson prior uses the tensor regardless). Measured (battery replica, 225 sets with limits): directional statement established on 95; 23 NOT_DETECTED -> DETECTED (directional ratio 1.14-1.59); of the 17 sets with ratio > 1.2 and an uncensored weakest limit, 11 flip, 5 stay CANNOT_DETERMINE with limits reported, 1 (sparse, noisy cones) stays NOT_DETECTED. ANISOTROPY warnings (spread > 0.5 A) appear on 6 more sets. Confirmed with rugnux runs on the ci tier, those 17 and the private arm: p.hkl and MTZ data identical to the base (e34a5fb88) on every set compared. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 02:21:32 +02:00
leonarski_fandClaude Opus 5.5	55dd0d4f6b	rugnux: doubled-cell hypothesis - the supercell probe reads its class to the limit, and a class that stands clear is merged in the doubled cell as well ... Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice reflection predicted on the same frame lies within r3, and each resolution shell is read against the median of the six other parity classes - which a single doubling leaves empty - as an excess in units of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z beside it ("Supercell probe to the limit" in the log). Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its noise and at more than twice the strongest of them, in at least three shells), the run integrates and merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both (DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT (the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the class reaches a quarter of the lattice's intensity in the finest shell both are measured in). The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed. Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell (odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at 8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the neighbour correlation, the separation from the nearest lattice reflection after the r3 filter - separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call. Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no longer advises -C with the doubled cell, which the indexer reduces back to the half cell. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 10:47:55 +02:00
leonarski_fandClaude Opus 5.5	8442d55f6f	rugnux --model: score the twin hypothesis too, and take the maps from the merge the model prefers ... Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also validated against that merge: as merged, and as twinned by its law - |F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled model, one overall scale refitted per a, a scanned 0..0.5 on the working set and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU structure factors as before). The maps and the placed model are then written from whichever merge has the lower R-free; the report gives MAPS_FROM and the subgroup's plain and twinned R-work/R-free and fraction. The subgroup file follows its own validation's indexing, as the answer follows its own. The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	475df9e5cb	rugnux: write the twin hypothesis beside the answer as <prefix>_twin_<SG>.mtz/.cif ... Near-perfect twins of a subgroup are undecidable from intensities, and a merge in the higher group cannot be twin-refined. Where the adopted group may have absorbed a twin law - a twin gate refused it and was overruled, or a group above P1 was adopted on a merge that read twin-like before the search - the same observations are also merged in a subgroup and written beside the answer, with the twin law and its fraction in _pdbx_reflns_twin. The subgroup is the one a decision named (the group a refused promotion was adopted over, the indexed lattice's group a metric re-ask promoted from), else the index-2 subgroup the search confirmed as a finalist whose lost two-fold agrees worst with the intensities. It is merged from the same engine, to the answer's resolution, without writing per-frame scales back; it carries the answer's free set (flags read off the answer's merge by the adopted group's asymmetric unit) and is written in the subgroup's reference and standard setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is Yeates' H on the pairs the law relates (1/2 - median H, a lower bound). At most one file; other qualifying subgroups are named in the report (TWIN_SUBGROUP_* keys). The answer's files are byte-identical. IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the point-group ledger now carries each hypothesis's representative group. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	8f05793b67	Twinning report: decide whether a twin law is possible from the lattice ... Whether a twin law can exist was read off a hard-coded list of Laue classes (4/mmm, 6/mmm, m-3m and rhombohedral -3m "holohedral", everything else eligible). That is a property of the lattice, not of the class: a C-centred orthorhombic or a monoclinic lattice with no special metric admits no twin law for mmm or 2/m either, and a pseudo-merohedral one admits a law the class says nothing about. On five pseudo-merohedral sets the report said "Twinning INDICATED - refinement needs a twin law" where the lattice admits none. TwinLawPossible asks gemmi's find_twin_laws (Le Page two-folds within 3 deg, as RfreeFlags and ReindexAmbiguity already do), and AnalyzeTwinning takes the cell to decide it; in P1 or without a cell a law stays possible, so the pre-promotion analysis and the space-group search are unchanged. Twin-like statistics (<|L|> below 0.44) above the false-operator bar where no law is possible are now reported as NO_TWIN_LAW: not a twin in this group, so the symmetry may be too high or something else narrowed the distribution. Report-only: no reflection file changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:33:51 +02:00
leonarski_fandClaude Opus 5.5	e9d4f22210	ResultReport: RUGNUX_GIT has no trailing space when the build knows no commit date ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:33:01 +02:00
leonarski_fandClaude Opus 5.5	5eba68432e	Merge branch 'mmcif-provenance' into rc175: deposition-grade mmCIF, provenance in the reflection files, git hash for builds without git ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:32:50 +02:00
leonarski_fandClaude Opus 5.5	c0bcf1a7a9	rugnux report: XDS geometry keywords as one report section, not p_XDS.INP ... The refined geometry is no longer written as a whole p_XDS.INP (and the report's XDS_INPUT= key is gone). Instead the report carries one section, XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated, as verified with XDS IDXREF). No cell or space group: XDS determines those. The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste. The same keywords were logged by LogXDSGeometry at the start of every pass, with the axis in rugnux's sense; that printing is removed, so the report section is the one place they are stated. Docs updated (RUGNUX_REPORT, RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL). Output only; p.mtz md5-identical on 3 battery sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:09:27 +02:00
leonarski_fandClaude Opus 5.5	7ed1544136	rugnux report: BEAM_CENTRE_OFF only beyond three times the tolerance ... At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery sets - most headers are a few pixels off. It now fires beyond three times the tolerance; between one and three times the summary's Beam centre line states the offset and recommends correcting the beamline value, without a warning. The background-only branch (runs that refined no centre) uses the same bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:57:06 +02:00
leonarski_fandClaude Opus 5.5	244f3c0f9d	rugnux: write the refined geometry as <prefix>_XDS.INP ... A rotation run now writes the geometry it integrated at as an XDS input beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the detector axes carrying the tilt, wavelength, the rotation axis the run integrated at (ProcessResult::used_rotation_axis, after any sign rescue), oscillation (with an adopted rotation scale), starting angle/frame, data range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left commented where the group is not in its number's reference setting), FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the master file as an x_??????.h5 template and a LIB= line to point at the reader plugin; a CBF series gets its ???? template and its own frame numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch (2010) Acta Cryst. D66, 125-132. The rotation axis goes out negated: rugnux turns the crystal by -phi about its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged (apart from the job list): 99.8% complete, ISa 23.3, direct beam within 0.15 px and cell within 0.07 A of rugnux's. Output only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	2632b77eb5	rugnux report: count overloads and overlapping spots, warn with advice ... Overloads: the integrated reflections whose signal disk held a saturated pixel are counted on the integration outcome before the merge, over the whole range and over the reflections coarser than 4 A, where they concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution reflections, and recommends attenuation / a shorter exposure. Measured: a strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%. Overlaps: both integration engines count the signal disks a nearer neighbour's prediction takes pixels from (the overlap ownership the default --overlap exclude already builds), and those left under --overlap-minpk of their own - BraggIntegrationCounts::overlapped / overlapped_major (log line "Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning fires from 1% in the second and recommends a longer distance, finer slicing or a smaller beam; on 10 battery sets it fires on the two long-axis ones (8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both; documented in RUGNUX_REPORT.md. Counters only; nothing reads them. p.mtz md5-identical to the rc175 battery on 10 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	fc6b91555f	Merged mmCIF: deposition keys, R_merge/R_pim/CC*, experiment metadata, scaled unmerged block ... The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on _reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/ _diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a loop naming Rugnux for 'data reduction' and for 'data scaling'. The experiment as the input states it: _diffrn.ambient_temp from the sample temperature, the source's general class (from NXmx /entry/source/type, which the HDF5 reader now reads), facility and beamline, the detector make/model in _diffrn_detector.type (where .detector used to carry it) and its class in .detector where the name settles it, and the collection date from /entry/start_time. R_merge and R_pim are formed beside R_meas from the same per-reflection sums (stills and rotation merges) and written with CC* overall and per shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the _reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as three new columns of the shell tables. R_pim following Weiss (2001), CC* following Karplus & Diederichs (2012). A second data block, data_sample_unmerged, carries the scaled unmerged observations of a rotation merge as _diffrn_refln (measured index, frame, scaled I and sigma, pdbx_scale_value) - the same records as the .hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now carries the scale and the frame. The unmerged MTZ stays unscaled. Reflection data are unchanged: the merged _refln rows, the .mtz columns and the .hkl are written as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:50:11 +02:00
leonarski_fandClaude Opus 5.5	f462047038	rugnux report: BEAM_CENTRE_OFF warning when the file's beam centre is wrong ... The pre-scan's beam-centre check and the post-refinement both measure where the direct beam lands, but the report only carried the centre the run used. Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the log's check prints, here on the run's own cell). A shift above the tolerance is a BEAM_CENTRE_OFF warning naming the file's, refined and background centres, the difference in px and mm and the value to correct the beamline to; a run that refined no centre is warned on the background alone where its fit resolves the difference (3 sigma). The summary gains a Beam centre line. The refined centre, not the background, is the trigger wherever one exists: on a battery set where XDS's refined direct beam and rugnux's agree to 1 px, the background fit is 12 px away from both. Report-only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:07:22 +02:00
leonarski_fandClaude Opus 5.5	1227d4168f	rugnux: drop the P1 cross-check; --finalist-ledger makes its own P1 merge ... The P1 cross-check (<prefix>_P1.mtz) and --no-p1-crosscheck are gone. The tail waited for that merge after the reflection files were written: 12-18 s on a large rotation set (7200 frames), ~2 s on a small protein. A user who wants a P1 merge re-runs with -S 1. --finalist-ledger keeps working: it now starts a P1 merge of its own beside the analyses (made only when the flag is given) and folds the ledger from it. The twin-immune zone evidence, which was read on the cross-check merge, moves with it and is reported only with --finalist-ledger. The unmerged MTZ was built in the shadow of the cross-check merge; it is now built beside the reflection files, from the point where the model validation has settled the indexing. WriteMtzReflections loses its amplitudes flag (only the cross-check wrote an intensity-only MTZ). The merged outputs (p.mtz, .hkl, _unmerged.mtz) are byte-identical to the base on the gate matrix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 19:04:40 +02:00
leonarski_fandClaude Opus 5.5	37b7fdd38d	rugnux: the terminal shows numbered steps with timing; the full log goes to <prefix>_log.txt ... Standard mode prints only steps, one-line results with the time since the line above, a rule closing each step with its total, warnings (!!) and errors (** ERROR:), and a summary at the end. Every log line of every module goes to <prefix>_log.txt instead, at debug level and timestamped, opened with version, build, GPU, threads, command line and the header geometry. -v puts the whole log back on the terminal. - RugnuxObserver gains OnStep / OnNote. RunPipeline posts them only on a pass that makes the result (the observer is held in steps_, nullptr on probe passes and on copies run beside the main pass); RunAllPasses opens steps for the checks between passes (rotation scale, metric symmetry, geometry walk) and notes its re-runs. OnPhase is written to the log as a marker. - RugnuxConsole (CLI only): on a terminal a progress bar and a spinner status line drawn in place with \r, durations only; not a terminal, plain ASCII with the time of day on every line and quarter milestones for long passes. isatty/_isatty, no environment variables. A warning repeated by later merges is shown once more on one line and then counted per step. - The CLI's former stdout blocks (space-group search, merging statistics, per-image cost, processing time, tNCS/twinning/anisotropy in --mode scale) go to the log; the calibration result block stays on the terminal as that mode's summary. The MX summary takes its rows from the report's own SUMMARY (ReportDocument::summary_rows); WriteResultReport returns the document it rendered. HDF5's automatic error-stack printing is switched off. - The banner keeps the "Version X (git abcdef" form battery.py reads off a bare invocation; battery.py's failure note now reads p_log.txt, whose lines are not wrapped. Output only: p.mtz is bit-identical on the three in-house reference sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:11:21 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00
leonarski_f	680c36c20d	v1.0.0-rc.166 (#76) ... * `rugnux --mode calibration` writes `<prefix>.json` beside the `.poni`, whose `dataset_settings` member is a `jfjoch_broker` `dataset_settings` body as it stands. * `rugnux` and `jfjoch_viewer` read PILATUS miniCBF sweeps natively, without conversion. * Masters written by other facilities open, including Eiger 1.x and third-party NXmx variants. * `rugnux` measures the beam centre on every run, and indexes with it when the file's value indexes nothing. * A detector swung out on a 2theta arm is placed where the file says it stands, and the calibration can hold the tilt fixed. * `rugnux` writes the unmerged MTZ by default, and a P1 merge beside it, so a wrong space group can be re-merged without reprocessing. * Significant improvements to symmetry handling in `rugnux`: the lattice, the point group, the setting and the systematic absences. * The `rugnux` report gives the resolution the CC1/2 fit reached, beside the range the reflections were written to. * The `rugnux` report gives the twinning statistics measured before the space group was decided, beside the ones measured after. * The `rugnux` report gives the strong-direction diffraction limit, and warns when CC1/2 is not monotone with resolution. * `rugnux` ranks screw axes on the evidence their absences carry, rather than on how many control reflections a candidate happens to have. * Twinning is no longer reported when the L-test contradicts it. * The `rugnux` report gives the detector tilt, the measured tilt and the direct beam beside the beam centre, and a post-refined beam centre is judged against the run's own measurement rather than the file's. * `--no-refine-tilt` holds the detector tilt at the value in the file, instead of zeroing it, when the calibration starts from the spots. * The `jfjoch_viewer` grid scan view draws the cells in the proportion of the scan steps, so the map has the shape of the scanned area. Reviewed-on: #76 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-02 21:17:31 +02:00
leonarski_f	511be0c366	v1.0.0-rc.165 (#75) ... * `rugnux --model` adopts the model's space group as a label where the data were merged in its enantiomorph, instead of reindexing the reflections - which swapped I(+) with I(-). * `rugnux --model` warns, naming the atom, when the anomalous density at the model's atoms comes out inverted, which means the data and the model are in opposite hands. * `rugnux --model` writes an anomalous difference map (`<prefix>_anom.ccp4`) when the merge kept the Bijvoet split, and names the ten model atoms it peaks highest on as `ANOMALOUS_SITE_01`..`_10`. * `MEAN_ATOM_DENSITY_SIGMA` is read from the map by cubic rather than linear interpolation and comes out around a tenth higher; it is no longer comparable with the figure earlier versions printed. * `rugnux --model` reads an mmCIF coordinate file as well as a PDB one, gzipped or not, taking the format from the file's content rather than its name. * A model `rugnux --model` cannot use is reported as a `WARNING:` line in the results report instead of only in the log. * The rugnux results report has a `10. MODEL VALIDATION` section when `--model` was given; `REPORT_VERSION` is 4, `WARNINGS` moves to section 11 and no existing key changed. * The rugnux results report records how the run was invoked, what it cost and what it ran on: `COMMAND_LINE=`, `WALL_TIME=` and `GPU_COUNT=` / `GPU=`. * rugnux says which GPUs it can see before it starts processing. * `rugnux --export-unmerged` also writes `<prefix>_unmerged.mtz` on a `--no-merge` run, and is ignored on a run with no output prefix instead of writing a file called `_unmerged.mtz`. * `/start` asks the writer whether the run can be written before the detector is armed, so a run whose master file already exists, or whose output directory cannot be created, is refused up front with the writer's own message. This needs the TCP image stream or the built-in HDF5 writer; the ZeroMQ stream is unchanged. * A calibration that fails goes to `Error` carrying the reason instead of `Inactive`, so `/wait_till_done` and `/wait_until_running` report it; a cancelled calibration still goes to `Inactive`. * `/wait_till_done` answers 500 with the message when a collection ended in an error. A cancelled collection and a collection that only triggered a warning still answer 200. * A pending start failure is discarded by `/cancel` and `/deactivate`, as it already was by `/start` and `/initialize`. * `/scan_result` no longer reports the previous run's images after a collection that failed to start, or after `/deactivate`. * The TCP image stream protocol version is 4. `jfjoch_writer` and `jfjoch_broker` have to be of the same release, as before. Reviewed-on: #75	2026-08-27 22:16:54 +02:00
leonarski_f	749db470ca	v1.0.0-rc.164 (#74) ... * rugnux now tells you whether a crystal diffracts anisotropically and how far it reaches in each direction, without a second program: a new `9. DIFFRACTION ANISOTROPY` section in `<prefix>_report.txt` and matching `_reflns.pdbx_aniso_B_tensor_*` / `_reflns.jfjoch_aniso_*` items in the merged mmCIF report the anisotropic deltaB, the diffraction limit along each principal direction, and a `NOT DETECTED` / `DETECTED` / `CANNOT DETERMINE` verdict measured against the data set's own systematic error. It is a description only - no intensity is corrected, no reflection is removed, and the merged data do not depend on direction. * rugnux can hand its integrated observations to another scaling program: `--export-unmerged` writes `<prefix>_unmerged.mtz`, an unmerged MTZ readable by aimless, pointless, careless and `iotbx.merging_statistics`, in `--mode mx` and `--mode scale` alike. Each rotation reflection's partials are summed into one full; `--export-unmerged-partials` writes one row per image instead. Intensities carry the Lorentz-polarization factor and nothing else, since those programs scale the data themselves. Lattice-centring absences are not written; screw and glide absences are. * rugnux integrates crystals with broad spots better - where it changes anything, per-shell mean I/sigma improves by up to 31% and R_meas by up to 24% - because on rotation data the integration signal radius is now taken from the crystal's own measured spot width instead of a fixed 4 px. `--adaptive-integration-radius=off` restores the fixed radius and an explicit `--integration-radius` still overrides both. The widened radius applies to the final integration pass only, and a pattern too dense for it is re-integrated at 4 px with a note in the log. * rugnux discards fewer stills reflections for want of a background ring, improving per-shell R_meas over most of the signal-bearing range: the stills background ring now runs to 14 px instead of 12. The gain reverses in shells below a mean I/sigma of about 4. * rugnux determines the space group with thresholds that mean the same thing on a weak crystal as on a strong one: symmetry operators are scored on resolution-normalised intensities (E squared) instead of raw merged intensities, and a reflection counts as genuinely present on its counting significance instead of on the merged I/sigma, which saturates at the merge's own ISa. The search resolution cut is no longer able to move the answer, and the twin-law H bound moves from 1.70 to 1.85, which stops one class of correct high-symmetry assignment being refused as twinning. * rugnux says what the space-group search tested and what it could not: the twin-law disagreement H is printed for every operator together with the adopted point group's H ratio and its bound; alternatives that are not on the reported lattice are named with how their cell differs; and a lattice centring the data could not test - the crystal having been integrated on the primitive sub-cell, so the reflections it extinguishes were never measured - is marked `UNTESTED` and warned about where it is adopted, as coming from the lattice metric rather than from the intensities. * rugnux `--mode scale` re-merges a `_process.h5` in the right symmetry without being told it: the file now records the space group on every run - a two-pass rotation run wrote none before, so re-merging defaulted to P1 - together with the change of basis under `/entry/MX/reindexMatrix` where the lattice was re-seated, and `--mode scale` also reports the Wilson B-factor estimate instead of `WILSON_B= nan`. A file written before this stops with a message naming the two cells and the override to use, instead of failing inside the merge. A third-party reader of a `_process.h5` must apply `reindexMatrix` where it is present. * rugnux installs on its own, as a package called `rugnux` - `dnf install rugnux` or `apt install rugnux` - instead of arriving inside `jfjoch-viewer`. It pulls in none of the acquisition stack, so a machine that only processes data no longer has to carry the broker, the detector libraries or Qt to get it. Installing it over a `jfjoch-viewer` from rc.163 or earlier, which still owns `/usr/bin/rugnux`, upgrades cleanly rather than failing on the duplicate file. * rugnux is also a standalone download, built for arm64 as well as x86_64: `rugnux-<version>-linux-{x86_64|aarch64}-cuda<major>.tgz` and `rugnux-<version>-win64-cuda<major>.zip` on the release page, for machines that are not managed by a package manager. The aarch64 build targets GH200 and DGX Spark, and is untested on hardware. * Every portable Linux binary is now a single self-contained file: cuFFT is linked statically instead of being shipped beside the executable and found through an rpath, so `rugnux` and `jfjoch_viewer` need nothing but an NVIDIA driver, and only to use the GPU. The `.rpm`/`.deb` continue to take cuFFT from the distribution. The developer utilities `jfjoch_extract_hkl` and `jfjoch_recompress` are no longer packaged anywhere. * Jungfraujoch needs six fewer shared libraries on the machine - libopenblas and libmetis, and libgfortran, libquadmath, libgomp and libz behind them - because the Ceres LAPACK, METIS and SuiteSparse back-ends are no longer built. Nothing in the code ever selected them, and results are unchanged. * The PCIe driver DKMS package builds for the kernel it is being installed for instead of the running one, so a module built while a kernel update is being applied loads after the reboot. * The PCIe driver builds on RHEL 9.5 and later, and on their CentOS Stream, Rocky and AlmaLinux equivalents, where the `vm_flags` kernel interface was backported into the 5.14 kernel. * A data collection started with `async_start` that fails to start - a writer refusing to overwrite an existing file, for instance - is reported as an error by `/wait_until_running` and `/wait_till_done` instead of as a timeout and a successful collection respectively. The error message is the one the writer gave. * A calibration that is cancelled or that fails to collect its pedestals is no longer reported as a successful one. The broker goes to `Inactive` with an error message and has to be initialized again, instead of sitting in `Idle` looking ready to measure while holding partial pedestals - data collected in that state was silently mis-converted. * A failed `/initialize` is reported to `/wait_until_running` and `/wait_till_done` as soon as it happens, instead of when their timeout expires. * `space_group_number` accepts space groups up to 230 in the API schema, so cubic space groups can be recorded. The broker always accepted them; the generated clients rejected them before the request was sent. * The results report's `REPORT_VERSION` is 3, two sections having been added. Existing key names and table columns are unchanged. * The merged statistics table has **9** resolution shells instead of 10, which is what XDS reports. The bins were already XDS's - equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept - so at the same resolution limits the two tables now have the same shell boundaries and can be read row for row. `--resolution-shells` sets a different count. * `rugnux --model` now settles the frame the merged reflections are written in, not only the frame the R-factors and the maps are computed in: the `.mtz`/`.cif`/`.hkl` come out in the model's indexing, and where the data were merged in the model's enantiomorph they take the model's hand and space group - which on anomalous data puts I(+) and I(-) the right way round. The indexing choice is logged with the winning R-free and the runner-up, so a decision made within noise is visible. * `rugnux --model` can resolve the indexing ambiguity of a **serial stills** run, which a model could not do before: structure factors computed from the model become the per-image reference, the same role a reference MTZ plays. It needs the cell and space group up front (`-C` / `-S`). Without one or the other, a merohedral serial run still merges both hands together and says so. * The rugnux documentation opens with a quick start - the default run, and runs with a reference MTZ, with a model, or with the space group and cell pinned - and explains the indexing ambiguity: what it costs on rotation and on serial data, and which of `-z` / `--model` resolves it in each case. The long reference pages now carry a table of contents. Reviewed-on: #74 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-26 22:47:00 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00