215 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	f567771d56	CHANGELOG: rc.175 names --mr, which takes predicted and experimental models ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 20:08:50 +02:00
leonarski_fandClaude Opus 5.5	30c7bee15f	rugnux: --af-model becomes --mr, for predictions and experimental structures alike ... - The option is --mr <file>. What kind of model it is is read from the file: a prediction where it is ModelCIF (_ma_qa_metric) or has no cell and one B value per residue in 0-100 (AlphaFold PDB/mmCIF); an experimental structure otherwise. A prediction is handled as before (pLDDT >= 70 trim, B from pLDDT, PAE domains where the AlphaFold DB's PAE file sits beside it). An experimental structure is taken as its polymer - no waters, ligands, hydrogens or alternative conformers, occupancy 1, isotropic B as refined - with a coordinate error of 0.67 A (a prediction's at pLDDT 90) for sigma_A; a B column above 100 no longer stops the search. - The report section is MOLECULAR REPLACEMENT (EXPERIMENTAL), its keys MR_* (MR, MR_MODEL_FILE, MR_MODEL_KIND, MR_DECISION, MR_SPACE_GROUP, MR_COPIES, MR_HAND_CHECK, MR_SECONDS, ...); the <prefix>_mr outputs are unchanged. - The twin hypothesis' merge that is made only for the model to test (a metric re-ask promoted the indexed lattice's group) is written and reported only where the model chooses the subgroup. Raw deposited coordinates of nine control sets, unconverted: every one placed and confirmed against the deposition (all chains as one body), the groups the same as with the prediction or the prepared model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 19:30:22 +02:00
leonarski_fandClaude Opus 5.5	6ee167df9a	Merge rc175 (the null turns each copy about its own centroid) into mr-7qij ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:49:20 +02:00
leonarski_fandClaude Opus 5.5	c6912e2a06	rugnux --af-model: tNCS half-vector moves checked, the merge's own group not ruled out beside tNCS, P1's first copy judged by its orientation (WIP, validation pending) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 16:49:02 +02:00
leonarski_fandClaude Opus 5.5	aabee7423f	CHANGELOG: rc.175 release notes ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 14:10:51 +02:00
leonarski_fandClaude Opus 5.5	0d33d7a40a	rugnux --af-model: likelihood judging, PAE domains, twin comparison, tNCS carry test, faster multi-copy search ... - Likelihood (Read 2001; Z-score over random positions as McCoy et al. 2005): log-likelihood gain of Rice/Woolfson E distributions, sigma_A from the pLDDT coordinate error with its scale refined, to 2.5 A (or the 20000 lowest-resolution reflections). Further-copy TF peaks ranked by the gain on the search's reflections (peaks that lower it dropped); a copy whose TF Z is below 8 also stands out where its refined likelihood Z-score reaches 8 - only where the copies placed are as many as Matthews makes most likely. The likelihood is computed only where a translation function does not stand out. Copy structure factors exact from one FFT of the cell's grid, or read off the molecular transform where it reaches the likelihood's resolution. - PAE domains: the AlphaFold DB's predicted_aligned_error JSON beside the model (same name pattern) is read automatically; average linkage on the symmetrised PAE (10 A, domains >= 40 residues). Where the whole model is placed nowhere, each domain is searched by itself side by side; placed domains must agree, and a domain alone may only confirm the answer's group. - Twin comparison: where the answer's merge chose and the twin hypothesis' merge exists, the chosen placement read as a crystal in the subgroup (its symmetry images as copies, the subgroup's setting found from the two cells) against the model searched for afresh in the subgroup, both against the data twinned by the law (fraction refined on the working set, Yeates 1997); the subgroup is taken by a 0.03 R-free lead plus its own null/margin -> TWIN_SUBGROUP, written to p_mr with law and fraction. With --af-model rugnux makes the subgroup merge wherever a metric re-ask promoted the indexed group. - tNCS rule: a group with fewer copies beside tNCS (screws bridged by the vector, or the enantiomorph) no longer leaves the choice undecided where the chosen copies, carried into it and refined, fit worse by 0.03 in R-free. - Speed: the molecular transform's FFT on the GPU (cuFFT, serialised), no rigid body for trials that lower the likelihood, re-search only with tNCS, PAE domains searched in parallel. Survey (193 sets, MR replay on stored merges): single GREEN 58 -> 68, multi 48 -> 58, partial 8 -> 9; 4bwl (and, in full runs, 2wnq and a withdrawn entry of the same protein) TWIN_SUBGROUP P21 instead of a wrong AGREES; a P21212 set with tNCS now SPACE_GROUP P21212 instead of UNDECIDED; no wrong call. CPU and GPU builds make the same decisions on 18 demo/multi-copy/twin sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 13:21:51 +02:00
leonarski_fandClaude Opus 5.5	fcbe92042b	Merge branch 'af-mr3' into rc175: --af-model searches several copies per asymmetric unit (Matthews probability, sequential translation function, fixed-copy rigid body, tNCS) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-12 01:36:26 +02:00
leonarski_fandClaude Opus 5.5	63e0a85467	rugnux --af-model: several copies in the asymmetric unit, tNCS, a search-matched null, faster ... Copy number. The hard "Vm >= 4: MULTI_COPY, no decision" rule is gone. Copies are searched one after another up to the largest copy number with a Matthews probability of 1% (a Gumbel distribution with Kantardjieff & Rupp's 2003 mode and mean, no resolution dependence); COPIES_LIKELY reports the most probable. Sequential search. Copy n's translation function carries copies 1..n-1 as a known partial structure: the cross term 2 Re(conj(F_known) F(t)) goes on the same FFT grid (CPU FFTW and GPU cuFFT; KnownStructureFactors, AssemblyCorrelation). Its peaks, past those that clash, are ranked by the correlation of the whole assembly with E^2 (the TF itself, linear in |F|^2, rewards copying a placed copy onto an origin shift), the first that stands out (TF Z >= 8), packs and lowers R-free by 0.01 is kept; each copy is refined beside the others held fixed (RefineRigidBody's fixed copies), and the first copy again beside the second, the earlier ones once more at the end. Packing: ClashFraction counts C-alpha clashes with other copies and all their mates, binned (7mzt: 51 s of clash checks -> 1.9 s). Search quality: rotation-function peaks hill-climbed off the 6 deg sampling (6ze4: RF Z 10.4 -> 16.0, 3.3 deg off the deposited orientation -> 0.2); the first copy goes down the TF list until three placements that pack are refined, so a group is CLASH only where no offered orientation packs; groups are searched side by side, and every group the best one leads is searched again with the best one's copy orientations tried first, the better search kept. tNCS: the native Patterson of E^2 (5-10 A) off the lattice; placed copies' orientations join every further copy's search; and where the tNCS vector turns one group's screw axes into another's (TncsRelatesGroups), a group whose search stopped at fewer copies does not lose to the other - UNDECIDED. Measured on an in-house tNCS crystal (vector c/3 along the 3-fold): both hands fit with three copies to within 0.01 in R-free, and where one hand's search failed the other was taken as ENANTIOMORPH; now UNDECIDED. Null: the replicates are placed by the translation function in their own orientation, as the search placed the real one (a null at the real centroid reads high), and the gate is 4.51 = t(8) sqrt(1 + 1/9), 3 sd of a nine-draw null, the gate --model's MODEL_FIT is now read at. Speed: placements scored on the GPU structure-factor engines; further copies' trials score with the first copy's bulk solvent held, the assembly refitted at the end. Survey of 193 sets (jfjoch_mr_replay on stored merges, RTX 5080 shared with other jobs): MR median 1.15 s, max 24 s; demos 0.8-1.1 s. Report: COPIES and COPIES_LIKELY columns, MODEL_PRIOR_COPIES. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-12 00:27:17 +02:00
leonarski_fandClaude Opus 5.5	ee350f1f74	rugnux --model: a wrong model confirms no shell, fits no hand, and fits the GPU reserve ... Three defects the battery's wrong-model null found (5 sets x 5 deposited models of unrelated proteins): CC(model, data) significance. The per-shell sigma was Fisher's transform on independent reflections, but the data and a scaled model share the falloff across a shell, the anisotropic B fitted to these very data, the bulk solvent and the symmetry enhancement epsilon - whatever the model is. 21 of 24 wrong models read CC_MODEL_CONFIRMED_TO_D_MIN to the data's d_min (10-15 sigma in most shells). The correlation is now formed on I/epsilon and |F_model|^2/epsilon, and its null is empirical: the data at h against the model at h + d for the 26 nearest lattice offsets, which keeps everything smooth in reciprocal space and loses what the model knows of h. SIGMA = (z - mean) / max(sd, 1/sqrt(N-3)) in Fisher z over those 26, a new CC_NULL column shows the null, and a shell confirms only beside a coarser shell that also clears 3 sigma (one shell alone did so once in 216 wrong-model shells). Prototyped on the p_maps.mtz of a full open-arm run and the null runs: wrong models confirm 0/24 (was 21/24), the deposited models confirm 197/200 (was 198/200 in the prototype's reading; the one lost has CC_MODEL 0.05-0.13 in every shell). The report no longer says a model cannot agree by accident. MODEL_FIT gate. Accepted at 3.0 sample sds of a nine-draw null, as if the sd were known; an unrelated P1 helical model on a P3(1) helical crystal was ACCEPTED at +3.05. Against 48 draws it reads +2.09 and the best random placement fits as well as it does, and drawing the null about the cell axes instead changes nothing (+2.15): the cause is the nine-draw sd, not the orientation. The gate is now 3 sigma allowing for that: t(8) * sqrt(1 + 1/9) = 4.51. The deposited models of the open arm read +7.0 and up. The indexing lead stays at 3 sigma (INDEXING_MARGIN_SIGMA): it is read only behind the fit gate, and a deposited model settles one at +4.15. Rigid-body GPU reserve. The finest zone's observations were counted by the data's d, but the zones count them by the d of the index in the model's ASU, and a model's group need not keep the data's metric (an I2(1)3 model on I222 data: 2320 in the zone against 1712 reserved), so the validation died with "zone over the engine's reserve". Counted the zones' way, under every indexing probed. Tests: ModelValidation_CCModelUnrelatedModelConfirmsNothing (Fisher alone reads +4.7 on an unrelated model with an anisotropic falloff; the new sigma stays under 1), ModelValidation_WrongModelInAGroupOffTheDataMetric (fails with the old count), ResultReport_ModelValidationSection (CC_NULL column, lone-shell rule). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 21:30:34 +02:00
leonarski_fandClaude Opus 5.5	50e3b59fb7	rugnux --af-model: hand check reads only the anomalous map, on MR's placement and solvent ... The ENANTIOMORPH hand check ran a whole ValidateAgainstModel on the placed model: model read and frame probe, a rigid body, a 9-replicate null (~0.5 s of the ~0.6-0.9 s), sigma_A, three maps written. The decision needs only --model's rule: the model fits its null (the MR choice has already required its own null of the same construction) and the anomalous map at the model's atoms is not inverted. ReadModelAnomalousMap (ModelValidation) computes F_calc and F_mask to the data's resolution (GPU engine where there is one), carries them through the bulk solvent MR scored the placement with (the overall scale and anisotropic B do not move phases), and reads the anomalous map with the same code as the validation: its coefficient construction and its reading at the atoms are factored out of Validate (anomalous_coefficients, read_anomalous_map) and used by both, so plain --model is unchanged. Hand check 0.6/0.6/0.9 s -> 0.027/0.028/0.040 s on 11if/7ris/9zmu; same ADOPTED/VETOED on all seven cases tried (the three demos, two anomalous lysozyme sets and their mirrored-detector runs), with the scatterer means and deepest troughs within 0.1 sigma of the full validation's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 18:11:01 +02:00
leonarski_fandClaude Opus 5.5	2a3d60fb82	rugnux --af-model: anomalous veto on the enantiomorph, search on the GPU, cheaper rescoring ... - An MR ENANTIOMORPH decision is taken into the answer's label only as --model takes a model's enantiomorph: ValidateAgainstModel runs on the placed model (in a scratch directory, removed after) and its adopted_model_enantiomorph - fit over its null, anomalous map at the atoms not inverted - decides; otherwise ENANTIOMORPH_VETOED. The hand-check numbers go to the MODEL PRIOR section. - Rotation and translation functions on the GPU (MolecularReplacementGPU.cu): one block per orientation for the rotation function; per orientation the copies, the bin weights, a gather of the Crowther-Blow coefficients (CSR lists, fixed order, no atomics) and a cuFFT C2R, block-reduced statistics finished on the host in a fixed order. The CPU path (FFTW) is unchanged in what it computes; the orientation-independent part of the translation function (TranslationSetup) is now built once per group on either path. - Rescoring: a placement clashing with its mates over twice the limit is not refined, one rigid-body engine per group, and the null is built only for the candidate the choice falls on. Measured MR step on the RTX 5080 (before -> after, seconds; hand check extra where it runs): 3r6o 1.77 -> 0.67, neg 2.14 -> 0.37, 11if 1.20 -> 0.51 (+0.6 hand check), 7ris 1.46 -> 0.56 (+0.6), 9zmu 4.71 -> 1.07 (+0.9). Rotation 0.50 -> 0.02 s and translation 1.17 -> 0.13 s on 9zmu. Mirrored-detector runs of two anomalous lysozyme sets: the hand check vetoes (map inverted at the model's atoms); the unmirrored runs adopt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-11 17:10:48 +02:00
leonarski_fandClaude Opus 5.5	60a8bf3981	rugnux --af-model (experimental): single-copy molecular replacement of a predicted model ... A predicted model (AlphaFold mmCIF, pLDDT in the B column) is trimmed at pLDDT 70, given B-factors from the pLDDT (Oeffner et al. 2022) and placed by molecular replacement, one copy in the ASU: - rotation function on a uniform 6 deg sampling of the point group's rotation ASU, each orientation scored by the correlation of E^2 with the symmetry-summed |F|^2 read off the model's molecular transform, computed once by FFT and interpolated (Navaza 1994); - full-symmetry Crowther & Blow translation function (one FFT per orientation and group, through MapFromFPhi/FFTW) for the 50 best orientations in every group with the merge's rotations; - the best three placements per group through RefineRigidBody (GPU where there is one), bulk solvent and R-free to 3 A, a C-alpha packing check against symmetry mates, Matthews, and the random-orientation null for a placed candidate. Decision: PLACED = TF Z >= 8, clash <= 5%, Vm in [1.5, 4), null >= 3 sd. Within the answer's merge the placed group with the lowest R-free is chosen where it leads every other packing group by 0.03; the twin hypothesis' merge is searched, and chosen, only where no group of the answer's merge can hold the model at all (Matthews, packing). An enantiomorph relabels the answer as --model does; another group goes to <prefix>_mr.mtz/.cif plus <prefix>_mr_model.pdb; the MODEL PRIOR report section carries the table. The search runs on the CPU in both builds; without --af-model nothing changes. Measured (RTX 5080, MR step): 3r6o twin I41 chosen over I4122 (Matthews + clash) in 1.8 s, solution 0.09 A CA rmsd from Phaser's; 11if P41 -> P43 1.2 s; 7ris P3121 -> P3221 1.7 s; 9zmu P6122 -> P6522 4.7 s; 11if's model against 3r6o's data NOT_PLACED. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-11 13:35:05 +02:00
leonarski_fandClaude Opus 5.5	d7521bf1ea	TCP pusher: never block the receiver thread; queue bounded by image-buffer slots ... TCPStreamPusher::SendImage is called from the receiver's frame-transformation threads. It used a 128-deep per-connection queue and waited when it was full. With images routed per file ((n / images_per_file) % writers), every writer receives a whole file as a burst at the full frame rate (1000 images/file at 2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all transformation threads ended up waiting on one writer. They then stopped returning FPGA frame buffers, and the FPGA dropped packets: partial images, images without a background estimate, data loss on every writer. The ZeroMQ pusher has an unbounded queue and never blocks, which is why it coped. - The per-connection queue is unbounded: every element owns a distinct image-buffer slot, so the bound is the image buffer's slot count. When slots run out the receiver drops only the write (GetImageSlot() == nullptr), after analysis. Absorbing one file burst therefore needs an image buffer of about images_per_file slots (~20 GB for a 9M at 1000 images/file); below that, images are still dropped, now as writer-queue-full instead of packet loss. - SendImage never blocks. A dead or wedged writer is still caught in SendAll (15 s silence / 60 s without progress), after which WriterThread returns the queued slots unsent. - The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed the connection of any writer whose queue stayed full for 200 ms - a normal state for a slow but healthy writer - discarding up to 128 queued images and never sending END. END now follows the whole backlog. - A per-connection enqueue mutex makes "check active, enqueue" atomic against "clear active, enqueue the end sentinel": an image could land behind the sentinel when a connection was torn down mid-run, leaking its slot (CheckIfBufferReturned then threw) or being sent as stale data in the next run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with the race widened and the lock removed). - TearDownConnection marks the connection broken first, so the destructor returns a backlog instead of sending it. Writer side: - jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was hardcoded to the kernel default). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-10 21:11:09 +02:00
leonarski_fandClaude Opus 5.5	a897b08a7d	Merge branch 'doubled-cell' into rc175: a doubled-cell hypothesis for a half-translation pseudo-symmetry that breaks down with resolution ... Narrow, guarded: the doubled pass runs only where the added class grows with resolution and is resolved; the doubled cell is adopted only where one pseudo-translation explains the added class in every shell and the point group does not rise. Fixes the one remaining lattice failure (half cell -> deposited cell, R-free 0.574 -> 0.275); p.hkl identical on the ci tier, the supercell gate set, 78 further supercell-candidate sets and all private sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 15:18:14 +02:00
leonarski_fandClaude Opus 5.5	b99cd3b5c6	rugnux doubled cell: model validation back in the doubled pass; docs for the final rules ... The doubled pass now runs only on resolved, rising classes (two sets of ~134 with probe logs), so the twenty-minute model placement in a doubled cubic cell can no longer arise, and the adopted answer is validated again (9min: R-free 0.275 against the deposited model, 0.574 in the half cell). Final rules, measured on gate + ci tier + all sets with a de-modulated axial row + 49 sets with a baseline supercell occupancy >= 8% + the private arm: 9min fail -> pass (P212121, doubled cell); every other set run keeps its verdict and a bit-identical p.hkl. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 15:16:58 +02:00
leonarski_fandClaude Opus 5.5	7580256ac7	WIP: doubled-cell rise must end at a quarter of the lattice's intensity ... On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4 standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends at 0.35-0.40, those classes at 0.06. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:14:10 +02:00
leonarski_fandClaude Opus 5.5	57543845c0	WIP: doubled-cell pass only where the class rises; no model validation in it ... A class that does not rise keeps the half cell whatever the doubled merge shows, so the doubled pass is no longer run for it - on a ci-tier crystal (R-centred, class at 1-2% of the lattice) that pass ran the card out of memory and failed the run. The doubled pass also runs without --model: placing the model in a doubled cubic crystal's 776 A R-centred cell took twenty minutes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:00:39 +02:00
leonarski_fandClaude Opus 5.5	55dd0d4f6b	rugnux: doubled-cell hypothesis - the supercell probe reads its class to the limit, and a class that stands clear is merged in the doubled cell as well ... Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice reflection predicted on the same frame lies within r3, and each resolution shell is read against the median of the six other parity classes - which a single doubling leaves empty - as an excess in units of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z beside it ("Supercell probe to the limit" in the log). Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its noise and at more than twice the strongest of them, in at least three shells), the run integrates and merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both (DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT (the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the class reaches a quarter of the lattice's intensity in the finest shell both are measured in). The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed. Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell (odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at 8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the neighbour correlation, the separation from the nearest lattice reflection after the r3 filter - separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call. Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no longer advises -C with the doubled cell, which the indexer reduces back to the half cell. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 10:47:55 +02:00
leonarski_fandClaude Opus 5.5	2b105bd259	docs: acknowledge autoPROC and STARANISO as inspiration for header distrust, beam centre and anisotropy ... Reframe the Global Phasing / autoPROC acknowledgement as inspiration and name the ideas: header metadata (beam centre, geometry, axis) must not be trusted blindly (autoPROC, Clemens Vonrhein in particular); the beam centre is determined from the images (rugnux does it its own way); and diffraction anisotropy must be analysed and reported (STARANISO, cited with the reference the STARANISO server recommends; no DOI exists). One-line pointers in data-analysis §1.4 and §13.5, and both works added to the CPU_DATA_ANALYSIS references. The Sheriff & Hendrickson, Popov & Bourenkov and Evans & Murshudov credits for the estimators are kept. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:31:17 +02:00
leonarski_fandClaude Opus 5.5	1f6d6b35e2	docs: credit Bricogne's Fourier-transform indexing, autoPROC and Global Phasing ... The de-novo FFT indexers implement the 1D-projection method of Steller, Bolotovsky & Rossmann (1997), which was already credited; the 3D Fourier transform indexing idea it descends from (Bricogne 1986, EEC PSD software workshop, Phase III, LURE, p. 28 - no DOI) was not. Add it to ACKNOWLEDGEMENT.md, the CPU_DATA_ANALYSIS references, the §5 text and the in-source credit, and add the DOI of Powell (1999) to the MOSFLM paragraph. 3D-FFT papers (Campbell 1998, Gildea et al. 2014) are not cited: the code does not compute a 3D transform. Add a section acknowledging Global Phasing (Bricogne and colleagues) and autoPROC (Vonrhein et al. 2011) as the expert-system model rugnux aims at. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 09:29:33 +02:00
leonarski_fandClaude Opus 5.5	e25b74dd1c	rugnux first pass: recognise the coincidence lattice of two twin domains ... A crystal twinned by reticular merohedry of index n - obverse/reverse rhombohedral domains, or a cubic crystal twinned by a 60 deg rotation about [111] (the spinel law, index 3) - has its two domains' reflections on a common lattice of n times the primitive volume of either domain's own cell. That coincidence cell indexes more frames and more spots than the domain's cell, so the first pass adopted it (a hexagonal/orthohexagonal or R:R cell for an F- or I-cubic crystal), and nothing downstream could recover the true lattice or Laue class. A new step after the sub-lattice hypothesis (TwinCompositeHypothesis), for n = 2 and 3, counts the validation spots on the standing cell in the four classes of the pair of index-n sub-lattices whose union holds the most spots: on both, on each alone, and on neither. Two domains leave the (1 - 1/n)^2 of the nodes on neither empty but for chance coincidences; one crystal, whatever sub-lattice a pseudo-translation favours, populates the nodes off the stronger sub-lattice alike whether they lie on the second sub-lattice or on neither. The step adopts the stronger domain's cell when the two sub-lattice cells are congruent (2 % / 2 deg), the weaker domain's own class holds at least 20 spots at more than four times the per-node density of the class on neither, and the domain's cell indexes a majority of the validation frames. The other domain is left to the leftover-lattice census. Evaluated on the private, inhouse and a twin/tNCS/rhombohedral/cubic/ long-axis selection of the open arm: it fires on the two known index-3 twins (a spinel-twinned F-cubic crystal, now indexed F-cubic; the in-house I-cubic insulin set with a 60 deg [111] second domain, R3:R -> I23) with a weak/neither per-node density ratio of 60 and more (the class on neither holds 0-8 spots against 250-2850 on the weaker domain), and nowhere else. On every other set the ratio stays between 1.0 and 1.6, the top of that range only on runs with a hundred or so validation spots, where the pair picked as the emptiest of 7 or 13 forms carries the scatter; hence the factor 4. Both twins now merge in their cubic group, and the sets re-run against the release baseline otherwise did not change. Cost when it does not fire: tens of milliseconds per first pass. Merges two prototypes: the class census and n = 2 coverage of one, the sub-cell congruence requirement of the other; no environment switch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 00:19:57 +02:00
leonarski_fandClaude Opus 5.5	0457cecafa	Merge branch 'battery-large-cells' into rc175: five large-cell open-arm sets (4mtk, 4nwv, 7d98, 9v16, 4p8r) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:36:30 +02:00
leonarski_fandClaude Opus 5.5	d8a3c6ea15	battery: add five large-cell sets to the open arm ... 4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122, c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1 16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC), tagged large-cell (plus long-axis / virus). References are the deposited space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces. 4P8R's archive ships per-frame .bz2 files, decompressed after extraction. EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes and the updated collection counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:31:28 +02:00
leonarski_fandClaude Opus 5.5	8442d55f6f	rugnux --model: score the twin hypothesis too, and take the maps from the merge the model prefers ... Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also validated against that merge: as merged, and as twinned by its law - |F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled model, one overall scale refitted per a, a scanned 0..0.5 on the working set and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU structure factors as before). The maps and the placed model are then written from whichever merge has the lower R-free; the report gives MAPS_FROM and the subgroup's plain and twinned R-work/R-free and fraction. The subgroup file follows its own validation's indexing, as the answer follows its own. The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	475df9e5cb	rugnux: write the twin hypothesis beside the answer as <prefix>_twin_<SG>.mtz/.cif ... Near-perfect twins of a subgroup are undecidable from intensities, and a merge in the higher group cannot be twin-refined. Where the adopted group may have absorbed a twin law - a twin gate refused it and was overruled, or a group above P1 was adopted on a merge that read twin-like before the search - the same observations are also merged in a subgroup and written beside the answer, with the twin law and its fraction in _pdbx_reflns_twin. The subgroup is the one a decision named (the group a refused promotion was adopted over, the indexed lattice's group a metric re-ask promoted from), else the index-2 subgroup the search confirmed as a finalist whose lost two-fold agrees worst with the intensities. It is merged from the same engine, to the answer's resolution, without writing per-frame scales back; it carries the answer's free set (flags read off the answer's merge by the adopted group's asymmetric unit) and is written in the subgroup's reference and standard setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is Yeates' H on the pairs the law relates (1/2 - median H, a lower bound). At most one file; other qualifying subgroups are named in the report (TWIN_SUBGROUP_* keys). The answer's files are byte-identical. IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the point-group ledger now carries each hypothesis's representative group. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	5eba68432e	Merge branch 'mmcif-provenance' into rc175: deposition-grade mmCIF, provenance in the reflection files, git hash for builds without git ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:32:50 +02:00
leonarski_fandClaude Opus 5.5	c0bcf1a7a9	rugnux report: XDS geometry keywords as one report section, not p_XDS.INP ... The refined geometry is no longer written as a whole p_XDS.INP (and the report's XDS_INPUT= key is gone). Instead the report carries one section, XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated, as verified with XDS IDXREF). No cell or space group: XDS determines those. The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste. The same keywords were logged by LogXDSGeometry at the start of every pass, with the axis in rugnux's sense; that printing is removed, so the report section is the one place they are stated. Docs updated (RUGNUX_REPORT, RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL). Output only; p.mtz md5-identical on 3 battery sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:09:27 +02:00
leonarski_fandClaude Opus 5.5	7ed1544136	rugnux report: BEAM_CENTRE_OFF only beyond three times the tolerance ... At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery sets - most headers are a few pixels off. It now fires beyond three times the tolerance; between one and three times the summary's Beam centre line states the offset and recommends correcting the beamline value, without a warning. The background-only branch (runs that refined no centre) uses the same bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:57:06 +02:00
leonarski_fandClaude Opus 5.5	4120b0daa8	Merged mmCIF: drop the scaled unmerged block ... The unmerged observations already leave a run as the unmerged MTZ (and the scaled fulls as the .hkl), and deposition does not come straight from rugnux, so the _diffrn_refln block made the .cif up to ten times larger for no consumer. The .cif is one data block again; the keys, software rows, metadata, R_merge/R_pim/CC* and provenance stay. ScaledFull loses the scale and frame it carried only for that block. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:51:56 +02:00
leonarski_fandClaude Opus 5.5	244f3c0f9d	rugnux: write the refined geometry as <prefix>_XDS.INP ... A rotation run now writes the geometry it integrated at as an XDS input beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the detector axes carrying the tilt, wavelength, the rotation axis the run integrated at (ProcessResult::used_rotation_axis, after any sign rescue), oscillation (with an adopted rotation scale), starting angle/frame, data range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left commented where the group is not in its number's reference setting), FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the master file as an x_??????.h5 template and a LIB= line to point at the reader plugin; a CBF series gets its ???? template and its own frame numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch (2010) Acta Cryst. D66, 125-132. The rotation axis goes out negated: rugnux turns the crystal by -phi about its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged (apart from the job list): 99.8% complete, ISa 23.3, direct beam within 0.15 px and cell within 0.07 A of rugnux's. Output only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	2632b77eb5	rugnux report: count overloads and overlapping spots, warn with advice ... Overloads: the integrated reflections whose signal disk held a saturated pixel are counted on the integration outcome before the merge, over the whole range and over the reflections coarser than 4 A, where they concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution reflections, and recommends attenuation / a shorter exposure. Measured: a strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%. Overlaps: both integration engines count the signal disks a nearer neighbour's prediction takes pixels from (the overlap ownership the default --overlap exclude already builds), and those left under --overlap-minpk of their own - BraggIntegrationCounts::overlapped / overlapped_major (log line "Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning fires from 1% in the second and recommends a longer distance, finer slicing or a smaller beam; on 10 battery sets it fires on the two long-axis ones (8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both; documented in RUGNUX_REPORT.md. Counters only; nothing reads them. p.mtz md5-identical to the rc175 battery on 10 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	5584460de0	Reflection files record their provenance: commit, command line, input file ... The merged .mtz and the unmerged .mtz carry, after the existing "From Rugnux <version>" record, HISTORY records with the git commit and date, the command line, the input file (absolute path) and a cheap identity of it - size and modification time, the master file's for HDF5, nothing hashed - and the collection date. MTZ allows 30 records of 80 characters; a longer value runs on over records starting with '+', cut so that no record ends in a blank a reader would strip. The mmCIF names the commit in _software.description and the command line, input file and identity under _jfjoch_provenance (the local prefix the other private items use; no PDBx item holds them), the first two as text fields so any character survives. OutputProvenance is filled once in the CLI and travels in ProcessConfig to every writer of a reflection file. Reflection data are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:52:43 +02:00
leonarski_fandClaude Opus 5.5	fc6b91555f	Merged mmCIF: deposition keys, R_merge/R_pim/CC*, experiment metadata, scaled unmerged block ... The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on _reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/ _diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a loop naming Rugnux for 'data reduction' and for 'data scaling'. The experiment as the input states it: _diffrn.ambient_temp from the sample temperature, the source's general class (from NXmx /entry/source/type, which the HDF5 reader now reads), facility and beamline, the detector make/model in _diffrn_detector.type (where .detector used to carry it) and its class in .detector where the name settles it, and the collection date from /entry/start_time. R_merge and R_pim are formed beside R_meas from the same per-reflection sums (stills and rotation merges) and written with CC* overall and per shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the _reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as three new columns of the shell tables. R_pim following Weiss (2001), CC* following Karplus & Diederichs (2012). A second data block, data_sample_unmerged, carries the scaled unmerged observations of a rotation merge as _diffrn_refln (measured index, frame, scaled I and sigma, pdbx_scale_value) - the same records as the .hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now carries the scale and the frame. The unmerged MTZ stays unscaled. Reflection data are unchanged: the merged _refln rows, the .mtz columns and the .hkl are written as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:50:11 +02:00
leonarski_fandClaude Opus 5.5	f462047038	rugnux report: BEAM_CENTRE_OFF warning when the file's beam centre is wrong ... The pre-scan's beam-centre check and the post-refinement both measure where the direct beam lands, but the report only carried the centre the run used. Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the log's check prints, here on the run's own cell). A shift above the tolerance is a BEAM_CENTRE_OFF warning naming the file's, refined and background centres, the difference in px and mm and the value to correct the beamline to; a run that refined no centre is warned on the background alone where its fit resolves the difference (3 sigma). The summary gains a Beam centre line. The refined centre, not the background, is the trigger wherever one exists: on a battery set where XDS's refined direct beam and rugnux's agree to 1 px, the background fit is 12 px away from both. Report-only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:07:22 +02:00
leonarski_fandClaude Opus 5.5	13badcfa39	Docs: battery report of the pushed rc.175, with commentary and figures ... Full open and in-house battery on 0b2dde214 (run 20261009-1447_0b2dde_rc175-0b2dde2-full) against the 10-07 battery on 9ced4c444, without the REFMAC check: the R-free figure and numbers are kept from the rc.174 battery. Report regenerated from the run with the current battery tools (verdicts scored by the current scorer, so KDP passes as an indistinguishable alternative), commentary written by hand. 5ky6 and the low-ISa cubic insulin set go from pass to fail, accepted losses of taking the pre-pass-2 decisions from the first part of the sweep; every other verdict is unchanged. The whole battery took 4898 s against 5795 s, most of it model validation now on the GPU (278 s against 1024 s); without it the HDF5 sets are at the same speed. Time, zoomed time, in-house time and resolution figures regenerated with paper_plots.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 16:38:23 +02:00
leonarski_fandClaude Opus 5.5	136b9703f7	Revert "rugnux: a rotation frame the sweep's cell refuses is tried on that cell scaled" ... This reverts commit a825fb6cb. The per-frame cell-scale retry restored the space group of its one target (a P2 set whose cell grows with dose) but not its data: R_meas went 26 -> 37 % and d_min 1.54 -> 1.62 A against the pre-part-B run. It patches the early, low-dose cell of pass 2 frame by frame with a +-2 % range, 0.1 % step and sqrt(n) margin chosen on that set, and runs inside AnalyzeIndexing on every set for one difficult one. A late cell belongs to rc176's far-end check of the early decisions instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 09:13:49 +02:00
leonarski_fandClaude Opus 5.5	c6412819ba	Revert "rugnux: mask a shadow that turns with the sample, found in blocks of the pre-scan" ... This reverts commit f257d51ea. The block-wise moving-shadow mask fixed its one target (a half-image lysozyme set whose pin turns with the sample) but fired on 47 sets of the full battery: on one small-molecule set the mask covered 51 % of the detector and SHELXL R1 doubled (0.10 -> 0.20), on another the mask grew from 8 to 21 %. The pre-scan cost rose by +87 s on the HDF5 sets and +499 s on the CBF sets (the commit claimed +0.1 s / +1-5 s per set). A narrowed version (regr2-mask) still costs +25-45 s battery-wide, about +0.3 s per HDF5 set, for one difficult set - more than the owner's budget allows. It is parked for rc176, where a moved shadow is to be found by a far-end check of the early decisions instead. The documentation paragraph the commit added goes with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 09:13:34 +02:00
leonarski_fandClaude Opus 5.5	f257d51ea6	rugnux: mask a shadow that turns with the sample, found in blocks of the pre-scan ... A pin, loop or capillary on the goniometer shadows one place for part of the rotation only. The beam-stop mask is one for every frame and is read off the mean over the pre-scan sample, so a pixel such hardware covers on fewer than about half the sampled frames is kept and integrated attenuated on the frames it is covered on. Since the pre-scan samples only the first part of the sweep (part B, 5b9144410), a sweep whose moving shadow is short in the first half and long in the second loses it: on the in-house half-image lysozyme sweep the mask fell from 32.8% to 21.6% of the detector, the second half of the sweep was integrated through the shadow, ISa dropped 6.9 -> 4.1, R_meas 116 -> 240%, the cell refined a != b, and the attenuated intensities erased the a-axis screw evidence (merged rc175: P4_1 2 2 instead of P4_1 2_1 2). The sample is now also read in blocks of 10 consecutive frames (rotation only); each block's shadow is found on its own with the rings about the whole sample's centre, and the blocks' pieces that reach beyond the 30 px band the whole sample's edge can grow over (PENUMBRA_MAX_PX) are added whole. A pixel shadowed on most frames of any longer stretch is shadowed on most frames of one block, so nothing finer is needed; pieces within the band are a block redrawing a static stop's edge and are left out, so a sweep without moving hardware keeps its mask bit for bit. Part B's principle is kept: everything is still decided on the first part of the sweep. Measured (GPU, against merged rc175 eac0833c8, 19 sets): half-image lysozyme P4_1 2_1 2, ISa 4.1 -> 9.7, R_meas 240 -> 113%, CC1/2 0.955 -> 0.979, R-free(model) 0.331 -> 0.282 (last night 6.9 / 116%). p.mtz md5-identical on 14 of 19 (8v4o, 9rci, 5ky6, 6yqf, 6ze4, 5reo, 9qw8, insu_I low-ISa, insu_H notwin, myob powder 1/2, myob split, nothing_1/2); changed: 5nw5 (a dim crescent at the stop edge now masked: ISa 8.4 -> 8.9, CC1/2 0.937 -> 0.980, R-free 0.462 -> 0.429), 7atg (0.85% added at the detector edge, statistics unchanged), 9min (0.03%, fail -> fail unchanged), 5jk4 (<0.01%, unchanged). Cost: pre-scan +0.1 s on HDF5 sweeps, +1-5 s on large file-per-image (CBF, marCCD) sweeps, where the blocks are re-read and their masks computed on the host. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 23:55:33 +02:00
leonarski_fandClaude Opus 5.5	a825fb6cba	rugnux: a rotation frame the sweep's cell refuses is tried on that cell scaled ... A rotation frame is refined on the sweep's one cell (orientation and beam per frame, never the cell), and a crystal whose cell grows with the dose leaves that cell behind: the per-frame gate (20 % of the in-resolution, non-ice spots on the lattice) then refuses the frames it misses as if the crystal had gone. A frame that fails the gate is now tried once more on the sweep's cell scaled isotropically - steps of 0.1 %, out to +-2 %, nearest first - to the scale that puts most of its non-ice spots on the lattice, re-refined there, and integrated on it where it then clears the gate. The scale must gain more spots than sqrt(count), so forty trial scales cannot turn a chance spot on a sparse frame into a frame. Ice-ring spots are not counted, as the gate does not count them. A frame the sweep's cell already admits is untouched, and only the per-image integration path does this (not the first-pass validation scoring). Found on 5ky6 (P 1 21 1, 564 x 0.5 deg, B rising by 46 A^2 over the sweep), which part B (5b9144410, decisions before pass 2 from the first part of the sweep) turned from P 1 21 1 into P 1 2 1. The mechanism: - The first pass now indexes on [0, 141 deg). A per-frame probe of the isotropic scale that puts most spots on the canonical lattice reads -0.1 % on frame 0, +0.4 % at 50 deg, >= +0.6 % from 115 deg on (at frame 230: 62 spots on the lattice at the sweep's cell, 112 at +0.6 %). Sampling the whole sweep (before part B) handed pass 2 a cell averaged over the dose, so frames to ~147 deg and part of the last wedge still cleared the gate; the early-range cell is the low-dose cell, and every frame from 113 deg on was refused (221 of 564 frames merged, completeness 83 %). - The axial 0k0 row is recorded in those frames: the merge held 1 screw-absent axial reflection instead of 15, so the 2_1 screw could not be decided (P2 by default), and R-free 0.243 -> 0.263. - Separately, on the early range the FFT candidate matched a false C-orthorhombic setting (gamma 92 deg) whose refutation fell straight to triclinic instead of P-monoclinic, so the cell was refined free (cell dev 0.64 -> 1.03 %). Not changed here (see the report). Integrating every frame from the sweep's cell instead (the sparse-lattice switch forced on) also recovers P 1 21 1, but at 1.63 A with ISa 5.0 and R_meas 45 %: the late frames are then integrated on a cell they have left. Result (GPU, battery verdicts against the rc175 full run 7c9a6c7d1): 5ky6 fail P 1 2 1, d 1.50, R-free 0.263, compl 83.0 % -> pass P 1 21 1, d 1.62, R-free 0.234, compl 99.6 % (pre-part-B: P 1 21 1, 1.54, 0.244, 96.9 %) 9rci pass -> pass, compl 86.5 -> 94.7 %, d 1.81 -> 1.76 9qw8 pass -> pass, R-free 0.329 -> 0.326, ISa 5.2 -> 5.5 8v4o fail -> fail (its part-B lattice loss is untouched), compl 22.3 -> 23.1 % p.mtz md5 unchanged: 5nw5, insu_I_x06da_low_isa, nothing_1, nothing_2, myob_x06da_powder_2, 7atg, 5reo, 6fvz, 6hwj, 6pxc, 8tha, 9crw, 9jzo, 6zqr, 6rym, thau_x10sa_0p1deg, cytc_x10sa. Catch2: Rugnux_Rotation, RotationIndexer* pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 21:56:46 +02:00
leonarski_fandClaude Opus 5.5	2981aedd3e	Broker: a collection can be followed from /start, its start message comes in the event stream ... With a DECTRIS detector the broker starts the writers - and so learned the collection's start message - only when the detector's own start message arrived, after /start had returned. A reader started right after /start found no collection (404) and had to wait for, and race with, the detector. - /start registers the collection in LiveCollection (Expect) as it is accepted, so GET /live/events is accepted from then on. The image pusher fills in the start message when it starts the writers (Begin) - at once on the FPGA path, when the detector's start message arrives on the DECTRIS path. - The event stream's first event is now `start`, carrying that start message (CBOR, base64) with run number, file prefix, image count and images per file; it replaces `collection`. Until it is known the stream only keeps alive. file and end follow as before. - However the measurement thread ends, it ends a collection the pusher did not end (EndIfOpen), so a cancel or a failure before the detector streamed reaches the reader as `end` with an error. The wait is event-driven: it ends with the start message or with the measurement. - GET /live/start.cbor is removed: the stream carries the start message, one route less. - rugnux takes the start message from the stream (BrokerFeed::WaitForStart), so it can be started right after /start; files reported before it attached the reader are replayed to it. Tests: LiveCollection_ExpectedAtStart; BrokerFeed_StartFromEventStream (an event stream whose start message comes late, then a file and the end). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 17:03:09 +02:00
leonarski_fandClaude Opus 5.5	509a54ef92	Broker: tell a reader which data files of the current collection are closed ... So that rugnux can process a collection while it is written, the broker reports each data file once a writer has closed it and renamed it into place: - Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the images the file holds. TCP protocol version 5. Files closed at END are reported before the END acknowledgement. - LiveCollection (image_pusher): the start message the writer of the master file received, the closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher has no back channel and offers none. - Two routes, protected by the dataset's bearer tokens like the other dataset routes: - GET /live/start.cbor: that start message; - GET /live/events: server-sent events (collection, file, end, superseded) with the history first and a keepalive every 10 s. One follower at a time; a second gets 409. Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged. - The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for the whole collection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:21 +02:00
leonarski_fandClaude Opus 5.5	5b91444102	rugnux: take the decisions before pass 2 from the first part of a rotation sweep ... The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre) and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both. Stills keep the whole range. With rugnux following a collection while it is written, nothing before pass 2 then waits for the second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is collected, and pass 2 starts reading as soon as they are done. An offline run takes the same decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run. Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit), GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34, kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 -> 18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of 120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the battery before it is merged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 14:29:32 +02:00
leonarski_fandClaude Opus 5.5	a291c01c09	rugnux: the geometry pre-pass integrates at least 120 deg of the sweep ... The two-pass geometry pre-pass integrates the first --prepass-fraction (0.5) of the sweep, but never less than 120 deg of rotation (header oscillation), so a sweep up to 120 deg is pre-passed whole and one up to 240 deg gets 120 deg. Wider sweeps are unchanged. The range is computed in one place, Rugnux::PrepassEnd. Evidence (fraction sweep with the rc175-p1-baseline binary, 10 sweeps of 360 deg and 10 of 180 deg; pre-pass distance vs the full-sweep fit): median error 0.081 % at 90 deg, 0.042 % at 120, 0.042 % at 150 and 0.033 % at 180 on the 180-deg sweeps; 0.032 / 0.021 / 0.013 % at 120 / 150 / 180 deg on the 360-deg sweeps. 120 deg on a wide sweep matches what the whole-sweep pre-pass gave a 180-deg sweep. Below ~60 deg the pre-pass misses the distance outright (6rym: held at the header, 3.4 % off; 6vww and 6fwc: distance not freed). Merged quality moves by under 1 % at any angle >= 90 deg. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 14:10:53 +02:00
leonarski_f	889418e00e	Merge branch 'xds-plugin-license' into rc175	2026-10-08 11:13:48 +02:00
leonarski_fandClaude Opus 5.5	2629f3821c	XDS plugin 1.1.0: mask module/chip edges, pass saturation through; recommend Durin ... Plugin: - Module-edge (bit 30) and chip-edge (bit 31) pixels are marked -2 (untrusted). Chip-edge pixels are split from a larger pixel with integer division and read low at low counts; feeding them to XDS as real pixels was what made the three plugins disagree in CI. - A saturated pixel in a signed image (INTx_MAX) is passed through with its value, so XDS counts it as an overload; only the error marker (INTx_MIN / UINTx_MAX) becomes -1. - Release version 1.1.0 set in CMake (library name and start message). VERSION_* in plugin.h stay 0: XDS refuses a plugin whose info-array major version is not 0. - Built and packaged on Linux only; the untested macOS archive is dropped. - The XDS linking permission is removed from LICENSE; the plugin is GPL-3.0. CI: - XDS.INP beam centre corrected to 1091/1137 (ORGY was 7.7 px off, which put IDXREF on a knife edge where one plugin's run fell into a false minimum). - Each XDS case must reach CORRECT, IDXREF spot SD must be < 2 px, and all file layouts read by the same plugin must give identical SPOT.XDS. Docs: recommend Durin (PSI or Global Phasing build); mask-bit table rewritten from measurement. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 10:56:34 +02:00
leonarski_fandClaude Opus 5.5	11122e4924	Docs: battery report of the rc.175 branch after phase 1, with commentary and figures ... Full open and in-house battery on 9ced4c444 (run 20261007-2139_9ced4c_rc175-p1-baseline) against the rc.173 battery, without the REFMAC check: R-free figure and numbers are kept from the rc.174 battery (20261005-2102_3770f4_rc174-final-refmac). No verdict changed from rc.174; 2ygz is new and fails. Two sets lost quality at the 50 % pre-pass default (outlier-laden outer shell on 5jk4, runaway cut on the in-house split myoglobin set); fraction 1 restores them, but nearby fractions do too and a fixed cut gives equal statistics, so the pre-pass only tips two existing weaknesses (an unmasked dead edge row; a noisy half-set CC1/2 on a multi-domain crystal). Time, zoomed time, in-house time and resolution figures regenerated with paper_plots.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 00:15:34 +02:00
leonarski_fandClaude Opus 5.5	1a970eef91	docs: regenerate THIRD_PARTY_NOTICES.md (Mbed TLS replaces curl, XDS plugin note) ... Generated from the root file by the update_version.sh step, which otherwise runs only on a version bump; the published copy still listed libcurl. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:21:18 +02:00
leonarski_fandClaude Opus 5.5	89207fe2bd	viewer: cpp-httplib + Mbed TLS instead of libcurl for the broker client ... The viewer's live-broker client goes back to cpp-httplib, which the broker already fetches; libcurl was only there for Kerberos, which is no longer wanted. HTTPS comes from Mbed TLS 4.2.0, fetched and statically linked for viewer builds only, so one TLS backend now serves Linux, Windows and macOS (the macOS viewer previously had no TLS at all, and Linux needed host OpenSSL >= 3.0). cpp-httplib verifies servers against the OS trust store: the Windows certificate store, the macOS Keychain, the CA bundle on Linux. CPPHTTPLIB_MBEDTLS_SUPPORT is set on jfjoch_viewer only, not on httplib::httplib, so the broker stays plain HTTP. JFJochHttpReader.cpp is the only viewer source that includes httplib.h, and the header keeps httplib::Client opaque so <windows.h> stays out of the other viewer units. Checked on Linux: a viewer-only build links Mbed TLS statically, with no curl and no system libssl; a standalone client built against the same httplib + Mbed TLS fetches from real HTTPS hosts and rejects expired, self-signed and wrong-host certificates. Windows and macOS are untested. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:16:04 +02:00
leonarski_fandClaude Opus 5.5	36366295f2	Windows: viewer installer and rugnux zip from one build ... The per-generator CPack config written for macOS is now generic and shared: the installer/.dmg generator gets the viewer, the archive generator (TGZ/ZIP) every other component. A Windows viewer-only build therefore packs NSIS;ZIP - the jfjoch-viewer installer and rugnux-<version>-win64-<variant>.zip, name unchanged - and the cuFFT DLL is installed into both components. CI folds build-rugnux-windows into build-windows (cuda and nocuda, so the Windows rugnux gains a -cpu zip) and checks the zip holds rugnux.exe, no viewer, and in the CUDA variant the cuFFT DLL. JFJOCH_RUGNUX_ONLY remains for the aarch64 cross build. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:04:17 +02:00
leonarski_fandClaude Opus 5.5	07ae3c3a40	XDS plugin: GPL linking permission, own tgz with its notices, macOS build; one portable CI build ... LICENSE: the section-7 addendum is rewritten as numbered additional permissions. CUDA libraries are treated as System Libraries (Intel MKL dropped - never linked). New: the XDS plugin may be linked with the closed-source XDS, valid only while it contains no third-party GPL code. The plugin links HDF5, zlib-ng, Zstandard and Compression (LZ4, Bitshuffle, Bitshuffle h-perf) and nothing else - no FFTW. In a portable (viewer-only) build it is its own CPack component, shipped as jfjoch_xds_plugin-<version>-<os>-<arch>.tar.gz with LICENSE, a short xds-plugin/THIRD_PARTY_NOTICES.md and only those six licence texts. On macOS it now exports only plugin_* (--exclude-libs does the same on Linux). The RPM/DEB packaging is unchanged. The viewer-only build already compiled everything rugnux needs, so it now packs one archive per program: Linux three .tar.gz (viewer, rugnux, plugin); macOS the viewer .dmg plus rugnux and plugin .tar.gz, each generator getting its components through CPACK_PROJECT_CONFIG_FILE. CI merges build-viewer-tgz + build-rugnux-tgz into build-portable-tgz (cuda and nocuda, so Linux rugnux gains a -cpu archive) and the two macOS jobs into build-macos. Both check that FFTW is neither on the plugin's link line nor in its symbol table and that it exports only plugin_*. The bare .so upload from the RPM job is dropped. RUGNUX_ONLY stays for aarch64 and Windows. Verified locally (Linux, CPU, CI flags): three tarballs, the plugin link line is exactly the list above, 0 FFTW symbols, four exports, system-only dynamic deps; XDS loads the packaged plugin. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 20:29:19 +02:00