198 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5.5	7580256ac7	WIP: doubled-cell rise must end at a quarter of the lattice's intensity ... On strong data a class at one or two percent of the lattice rises significantly (0.009 to 0.059 at 4 standard errors on a ci-tier triclinic crystal, which was then adopted doubled and failed), and a half cell missing a few percent of its intensity still describes the crystal. The doubled deposition ends at 0.35-0.40, those classes at 0.06. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:14:10 +02:00
leonarski_fandClaude Opus 5.5	57543845c0	WIP: doubled-cell pass only where the class rises; no model validation in it ... A class that does not rise keeps the half cell whatever the doubled merge shows, so the doubled pass is no longer run for it - on a ci-tier crystal (R-centred, class at 1-2% of the lattice) that pass ran the card out of memory and failed the run. The doubled pass also runs without --model: placing the model in a doubled cubic crystal's 776 A R-centred cell took twenty minutes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3	2026-10-10 14:00:39 +02:00
leonarski_fandClaude Opus 5.5	55dd0d4f6b	rugnux: doubled-cell hypothesis - the supercell probe reads its class to the limit, and a class that stands clear is merged in the doubled cell as well ... Supercell probe, log-only part: the class it reads over 20-3 A is followed out to the run's own limit (a second prediction of the same supercell on the same 60 frames, so the 20-3 A band and the SUPERCELL_* keys are unchanged). Half-integer reflections are counted only where no lattice reflection predicted on the same frame lies within r3, and each resolution shell is read against the median of the six other parity classes - which a single doubling leaves empty - as an excess in units of the lattice's intensity and as a z in the class's own standard errors, the strongest sibling's z beside it ("Supercell probe to the limit" in the log). Doubled-cell hypothesis: where that class stands clear of the six empty ones (measured above its noise and at more than twice the strongest of them, in at least three shells), the run integrates and merges the doubled cell too, forcing the last pass's indexing result with its lattice doubled along the class (orientation, geometry and axis kept). One of the two is written as the answer, the other's merged .mtz/.cif/.hkl beside it as <prefix>_doubled or <prefix>_halfcell; the report names both (DOUBLED_CELL_*). Which one is the answer is a policy behind one constant: ADDED_CLASS_SIGNIFICANT (the doubled cell wherever the class stands clear) or HALF_CELL_BREAKS_DOWN (default: only where the class reaches a quarter of the lattice's intensity in the finest shell both are measured in). The doubled lattice is classed with a near-zero tolerance: the default lattice search promoted a triclinic doubled cell with an 88.2 deg angle to orthorhombic C, and no frame indexed. Measured on forced doubled-cell merges (rc175 b530c2d base): the open-arm crystal deposited in the doubled cell merges at R-free 0.275 against its deposited model, against 0.574 in the half cell (odd/even 0.02 at >8 A rising to 0.65 at 1.95 A, odd half-set CC1/2 0.93 at 8-5 A). Five crystals deposited in the half cell read classes the doubled merge holds as well (odd CC1/2 0.73-0.98 at 8-4 A): two rise with resolution like the first (to 0.4-0.5 at 2.5 A), one is strong from low resolution (0.3-0.5), two stay flat at 0.1-0.2. Nothing measured on intensities - the trend, the neighbour correlation, the separation from the nearest lattice reflection after the r3 filter - separated the doubled deposition from the rising half-cell ones, so the policy is the owner's call. Also: the SUPERCELL_POSSIBLE prose no longer says correct cells "read the same", and the warning no longer advises -C with the doubled cell, which the indexer reduces back to the half cell. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 10:47:55 +02:00
leonarski_fandClaude Opus 5.5	e25b74dd1c	rugnux first pass: recognise the coincidence lattice of two twin domains ... A crystal twinned by reticular merohedry of index n - obverse/reverse rhombohedral domains, or a cubic crystal twinned by a 60 deg rotation about [111] (the spinel law, index 3) - has its two domains' reflections on a common lattice of n times the primitive volume of either domain's own cell. That coincidence cell indexes more frames and more spots than the domain's cell, so the first pass adopted it (a hexagonal/orthohexagonal or R:R cell for an F- or I-cubic crystal), and nothing downstream could recover the true lattice or Laue class. A new step after the sub-lattice hypothesis (TwinCompositeHypothesis), for n = 2 and 3, counts the validation spots on the standing cell in the four classes of the pair of index-n sub-lattices whose union holds the most spots: on both, on each alone, and on neither. Two domains leave the (1 - 1/n)^2 of the nodes on neither empty but for chance coincidences; one crystal, whatever sub-lattice a pseudo-translation favours, populates the nodes off the stronger sub-lattice alike whether they lie on the second sub-lattice or on neither. The step adopts the stronger domain's cell when the two sub-lattice cells are congruent (2 % / 2 deg), the weaker domain's own class holds at least 20 spots at more than four times the per-node density of the class on neither, and the domain's cell indexes a majority of the validation frames. The other domain is left to the leftover-lattice census. Evaluated on the private, inhouse and a twin/tNCS/rhombohedral/cubic/ long-axis selection of the open arm: it fires on the two known index-3 twins (a spinel-twinned F-cubic crystal, now indexed F-cubic; the in-house I-cubic insulin set with a 60 deg [111] second domain, R3:R -> I23) with a weak/neither per-node density ratio of 60 and more (the class on neither holds 0-8 spots against 250-2850 on the weaker domain), and nowhere else. On every other set the ratio stays between 1.0 and 1.6, the top of that range only on runs with a hundred or so validation spots, where the pair picked as the emptiest of 7 or 13 forms carries the scatter; hence the factor 4. Both twins now merge in their cubic group, and the sets re-run against the release baseline otherwise did not change. Cost when it does not fire: tens of milliseconds per first pass. Merges two prototypes: the class census and n = 2 coverage of one, the sub-cell congruence requirement of the other; no environment switch. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-10 00:19:57 +02:00
leonarski_fandClaude Opus 5.5	0457cecafa	Merge branch 'battery-large-cells' into rc175: five large-cell open-arm sets (4mtk, 4nwv, 7d98, 9v16, 4p8r) ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:36:30 +02:00
leonarski_fandClaude Opus 5.5	d8a3c6ea15	battery: add five large-cell sets to the open arm ... 4MTK (P61, c 653 A, PILATUS 6M miniCBF, SBGrid 235), 4NWV (I222 virus-like particle, 370-411 A axes, Rayonix MX-300 marCCD, SBGrid 9), 7D98 (P6122, c 602 A, EIGER 4M miniCBF at 1.9 A, XRDa 13), 9V16 (P41212, c 451 A, EIGER1 16M HDF5, XRDa 338) and 4P8R (P6522, c 701 A, Rayonix MX-225 marCCD, IRRMC), tagged large-cell (plus long-axis / virus). References are the deposited space group, cell and resolution. 7D98 is pinned to its 250-frame miniCBF sweep: the HDF5 masters beside it repeat frames 61-260 in 60-frame pieces. 4P8R's archive ships per-frame .bz2 files, decompressed after extraction. EXTERNAL_TEST_DATA.md gains the five rows (DOIs resolved), the archive notes and the updated collection counts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 20:31:28 +02:00
leonarski_fandClaude Opus 5.5	8442d55f6f	rugnux --model: score the twin hypothesis too, and take the maps from the merge the model prefers ... Where a twin hypothesis (<prefix>_twin_<SG>) was written, the model is also validated against that merge: as merged, and as twinned by its law - |F_twin(h)|^2 = (1-a)|F_model(h)|^2 + a|F_model(Th)|^2 from the placed, scaled model, one overall scale refitted per a, a scanned 0..0.5 on the working set and R-free read at it (ValidateAgainstModel's new twin_law argument; GPU structure factors as before). The maps and the placed model are then written from whichever merge has the lower R-free; the report gives MAPS_FROM and the subgroup's plain and twinned R-work/R-free and fraction. The subgroup file follows its own validation's indexing, as the answer follows its own. The model chooses the maps only: the space group, <prefix>.mtz and R_FREE stay the answer's. Documented in RUGNUX_REPORT.md, RUGNUX_ADVANCED.md and CPU_DATA_ANALYSIS_DECISIONS.md (14.9); Yeates (1997) credited. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	475df9e5cb	rugnux: write the twin hypothesis beside the answer as <prefix>_twin_<SG>.mtz/.cif ... Near-perfect twins of a subgroup are undecidable from intensities, and a merge in the higher group cannot be twin-refined. Where the adopted group may have absorbed a twin law - a twin gate refused it and was overruled, or a group above P1 was adopted on a merge that read twin-like before the search - the same observations are also merged in a subgroup and written beside the answer, with the twin law and its fraction in _pdbx_reflns_twin. The subgroup is the one a decision named (the group a refused promotion was adopted over, the indexed lattice's group a metric re-ask promoted from), else the index-2 subgroup the search confirmed as a finalist whose lost two-fold agrees worst with the intensities. It is merged from the same engine, to the answer's resolution, without writing per-frame scales back; it carries the answer's free set (flags read off the answer's merge by the adopted group's asymmetric unit) and is written in the subgroup's reference and standard setting (a C 1 1 2_1 description is reduced to P 1 2_1 1). The fraction is Yeates' H on the pairs the law relates (1/2 - median H, a lower bound). At most one file; other qualifying subgroups are named in the report (TWIN_SUBGROUP_* keys). The answer's files are byte-identical. IndexTwoTwinSubgroups and EstimateTwinLawFraction are new and tested; the point-group ledger now carries each hypothesis's representative group. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:40:22 +02:00
leonarski_fandClaude Opus 5.5	5eba68432e	Merge branch 'mmcif-provenance' into rc175: deposition-grade mmCIF, provenance in the reflection files, git hash for builds without git ... Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:32:50 +02:00
leonarski_fandClaude Opus 5.5	c0bcf1a7a9	rugnux report: XDS geometry keywords as one report section, not p_XDS.INP ... The refined geometry is no longer written as a whole p_XDS.INP (and the report's XDS_INPUT= key is gone). Instead the report carries one section, XDS GEOMETRY KEYWORDS, with the XDS.INP keywords that carry the geometry the result was integrated at: ORGX/ORGY (PONI, 1-based), DETECTOR_DISTANCE, the detector axes where the detector is tilted, INCIDENT_BEAM_DIRECTION with X-RAY_WAVELENGTH, and ROTATION_AXIS in XDS's sense (rugnux's axis negated, as verified with XDS IDXREF). No cell or space group: XDS determines those. The lines carry a "# XDS.INP> " prefix, so sed lifts them ready to paste. The same keywords were logged by LogXDSGeometry at the start of every pass, with the axis in rugnux's sense; that printing is removed, so the report section is the one place they are stated. Docs updated (RUGNUX_REPORT, RUGNUX_INTEGRATION, RUGNUX, RUGNUX_TUTORIAL). Output only; p.mtz md5-identical on 3 battery sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 19:09:27 +02:00
leonarski_fandClaude Opus 5.5	7ed1544136	rugnux report: BEAM_CENTRE_OFF only beyond three times the tolerance ... At one times BeamCenterNeed_pxl the warning fired on 107 of the 250 battery sets - most headers are a few pixels off. It now fires beyond three times the tolerance; between one and three times the summary's Beam centre line states the offset and recommends correcting the beamline value, without a warning. The background-only branch (runs that refined no centre) uses the same bound. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:57:06 +02:00
leonarski_fandClaude Opus 5.5	4120b0daa8	Merged mmCIF: drop the scaled unmerged block ... The unmerged observations already leave a run as the unmerged MTZ (and the scaled fulls as the .hkl), and deposition does not come straight from rugnux, so the _diffrn_refln block made the .cif up to ten times larger for no consumer. The .cif is one data block again; the keys, software rows, metadata, R_merge/R_pim/CC* and provenance stay. ScaledFull loses the scale and frame it carried only for that block. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:51:56 +02:00
leonarski_fandClaude Opus 5.5	244f3c0f9d	rugnux: write the refined geometry as <prefix>_XDS.INP ... A rotation run now writes the geometry it integrated at as an XDS input beside the report: ORGX/ORGY (the PONI, 1-based), DETECTOR_DISTANCE, the detector axes carrying the tilt, wavelength, the rotation axis the run integrated at (ProcessResult::used_rotation_axis, after any sign rescue), oscillation (with an adopted rotation scale), starting angle/frame, data range, NX/NY/QX/QY, OVERLOAD, sensor thickness, space group and cell (left commented where the group is not in its number's reference setting), FRIEDEL'S_LAW per -A and the merged resolution range. HDF5 inputs get the master file as an x_??????.h5 template and a LIB= line to point at the reader plugin; a CBF series gets its ???? template and its own frame numbers. The report names the file (XDS_INPUT=). Conventions per Kabsch (2010) Acta Cryst. D66, 125-132. The rotation axis goes out negated: rugnux turns the crystal by -phi about its axis (IndexAndRefine), XDS by +phi about ROTATION_AXIS. Checked with XDS IDXREF on two exports (one CBF, one HDF5 via durin): the negated axis indexes 99.5% / 14% of the spots, rugnux's own sign 37% / 0.04%. On the CBF set XDS then ran DEFPIX/INTEGRATE/CORRECT from the file unchanged (apart from the job list): 99.8% complete, ISa 23.3, direct beam within 0.15 px and cell within 0.07 A of rugnux's. Output only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	2632b77eb5	rugnux report: count overloads and overlapping spots, warn with advice ... Overloads: the integrated reflections whose signal disk held a saturated pixel are counted on the integration outcome before the merge, over the whole range and over the reflections coarser than 4 A, where they concentrate (log line "Overloads:", report OVERLOADED_REFLECTIONS*). An OVERLOADS warning fires from 0.05% of all, or 0.25% of the low-resolution reflections, and recommends attenuation / a shorter exposure. Measured: a strongly diffracting in-house sweep reads 0.017% / 0.42% (warned), an ordinary sweep with a few hundred overloaded partials 0.001% / 0.012%. Overlaps: both integration engines count the signal disks a nearer neighbour's prediction takes pixels from (the overlap ownership the default --overlap exclude already builds), and those left under --overlap-minpk of their own - BraggIntegrationCounts::overlapped / overlapped_major (log line "Overlapping spots:", report OVERLAPPED_REFLECTIONS*). An OVERLAPS warning fires from 1% in the second and recommends a longer distance, finer slicing or a smaller beam; on 10 battery sets it fires on the two long-axis ones (8.3% and 13.4%), the rest read 0.0-0.44%. Summary lines for both; documented in RUGNUX_REPORT.md. Counters only; nothing reads them. p.mtz md5-identical to the rc175 battery on 10 sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 18:36:25 +02:00
leonarski_fandClaude Opus 5.5	5584460de0	Reflection files record their provenance: commit, command line, input file ... The merged .mtz and the unmerged .mtz carry, after the existing "From Rugnux <version>" record, HISTORY records with the git commit and date, the command line, the input file (absolute path) and a cheap identity of it - size and modification time, the master file's for HDF5, nothing hashed - and the collection date. MTZ allows 30 records of 80 characters; a longer value runs on over records starting with '+', cut so that no record ends in a blank a reader would strip. The mmCIF names the commit in _software.description and the command line, input file and identity under _jfjoch_provenance (the local prefix the other private items use; no PDBx item holds them), the first two as text fields so any character survives. OutputProvenance is filled once in the CLI and travels in ProcessConfig to every writer of a reflection file. Reflection data are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:52:43 +02:00
leonarski_fandClaude Opus 5.5	fc6b91555f	Merged mmCIF: deposition keys, R_merge/R_pim/CC*, experiment metadata, scaled unmerged block ... The .cif now follows the PDBx/mmCIF dictionary's keys: _entry.id, the entry_id of _cell/_symmetry/_reflns, pdbx_ordinal and pdbx_diffrn_id on _reflns and _reflns_shell, diffrn_id on _diffrn_source/_diffrn_detector/ _diffrn_radiation, and _diffrn_radiation_wavelength.id. _software is a loop naming Rugnux for 'data reduction' and for 'data scaling'. The experiment as the input states it: _diffrn.ambient_temp from the sample temperature, the source's general class (from NXmx /entry/source/type, which the HDF5 reader now reads), facility and beamline, the detector make/model in _diffrn_detector.type (where .detector used to carry it) and its class in .detector where the name settles it, and the collection date from /entry/start_time. R_merge and R_pim are formed beside R_meas from the same per-reflection sums (stills and rotation merges) and written with CC* overall and per shell (_reflns.pdbx_Rmerge_I_obs/pdbx_Rpim_I_all/pdbx_CC_star and the _reflns_shell counterparts), as report keys R_MERGE/R_PIM/CC_STAR and as three new columns of the shell tables. R_pim following Weiss (2001), CC* following Karplus & Diederichs (2012). A second data block, data_sample_unmerged, carries the scaled unmerged observations of a rotation merge as _diffrn_refln (measured index, frame, scaled I and sigma, pdbx_scale_value) - the same records as the .hkl, which the wwPDB accepts as scaled unmerged data. ScaledFull now carries the scale and the frame. The unmerged MTZ stays unscaled. Reflection data are unchanged: the merged _refln rows, the .mtz columns and the .hkl are written as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:50:11 +02:00
leonarski_fandClaude Opus 5.5	f462047038	rugnux report: BEAM_CENTRE_OFF warning when the file's beam centre is wrong ... The pre-scan's beam-centre check and the post-refinement both measure where the direct beam lands, but the report only carried the centre the run used. Section 1 now sets the file's direct beam (FILE_DIRECT_BEAM) beside the background's (BACKGROUND_DIRECT_BEAM, with its sigma), the shift of the refined centre from the file's (BEAM_CENTRE_SHIFT) and the tolerance the geometry asks for (BEAM_CENTRE_TOLERANCE: BeamCenterNeed_pxl, the number the log's check prints, here on the run's own cell). A shift above the tolerance is a BEAM_CENTRE_OFF warning naming the file's, refined and background centres, the difference in px and mm and the value to correct the beamline to; a run that refined no centre is warned on the background alone where its fit resolves the difference (3 sigma). The summary gains a Beam centre line. The refined centre, not the background, is the trigger wherever one exists: on a battery set where XDS's refined direct beam and rugnux's agree to 1 px, the background fit is 12 px away from both. Report-only; no result changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 17:07:22 +02:00
leonarski_fandClaude Opus 5.5	13badcfa39	Docs: battery report of the pushed rc.175, with commentary and figures ... Full open and in-house battery on 0b2dde214 (run 20261009-1447_0b2dde_rc175-0b2dde2-full) against the 10-07 battery on 9ced4c444, without the REFMAC check: the R-free figure and numbers are kept from the rc.174 battery. Report regenerated from the run with the current battery tools (verdicts scored by the current scorer, so KDP passes as an indistinguishable alternative), commentary written by hand. 5ky6 and the low-ISa cubic insulin set go from pass to fail, accepted losses of taking the pre-pass-2 decisions from the first part of the sweep; every other verdict is unchanged. The whole battery took 4898 s against 5795 s, most of it model validation now on the GPU (278 s against 1024 s); without it the HDF5 sets are at the same speed. Time, zoomed time, in-house time and resolution figures regenerated with paper_plots.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 16:38:23 +02:00
leonarski_fandClaude Opus 5.5	136b9703f7	Revert "rugnux: a rotation frame the sweep's cell refuses is tried on that cell scaled" ... This reverts commit a825fb6cb. The per-frame cell-scale retry restored the space group of its one target (a P2 set whose cell grows with dose) but not its data: R_meas went 26 -> 37 % and d_min 1.54 -> 1.62 A against the pre-part-B run. It patches the early, low-dose cell of pass 2 frame by frame with a +-2 % range, 0.1 % step and sqrt(n) margin chosen on that set, and runs inside AnalyzeIndexing on every set for one difficult one. A late cell belongs to rc176's far-end check of the early decisions instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 09:13:49 +02:00
leonarski_fandClaude Opus 5.5	c6412819ba	Revert "rugnux: mask a shadow that turns with the sample, found in blocks of the pre-scan" ... This reverts commit f257d51ea. The block-wise moving-shadow mask fixed its one target (a half-image lysozyme set whose pin turns with the sample) but fired on 47 sets of the full battery: on one small-molecule set the mask covered 51 % of the detector and SHELXL R1 doubled (0.10 -> 0.20), on another the mask grew from 8 to 21 %. The pre-scan cost rose by +87 s on the HDF5 sets and +499 s on the CBF sets (the commit claimed +0.1 s / +1-5 s per set). A narrowed version (regr2-mask) still costs +25-45 s battery-wide, about +0.3 s per HDF5 set, for one difficult set - more than the owner's budget allows. It is parked for rc176, where a moved shadow is to be found by a far-end check of the early decisions instead. The documentation paragraph the commit added goes with it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-09 09:13:34 +02:00
leonarski_fandClaude Opus 5.5	f257d51ea6	rugnux: mask a shadow that turns with the sample, found in blocks of the pre-scan ... A pin, loop or capillary on the goniometer shadows one place for part of the rotation only. The beam-stop mask is one for every frame and is read off the mean over the pre-scan sample, so a pixel such hardware covers on fewer than about half the sampled frames is kept and integrated attenuated on the frames it is covered on. Since the pre-scan samples only the first part of the sweep (part B, 5b9144410), a sweep whose moving shadow is short in the first half and long in the second loses it: on the in-house half-image lysozyme sweep the mask fell from 32.8% to 21.6% of the detector, the second half of the sweep was integrated through the shadow, ISa dropped 6.9 -> 4.1, R_meas 116 -> 240%, the cell refined a != b, and the attenuated intensities erased the a-axis screw evidence (merged rc175: P4_1 2 2 instead of P4_1 2_1 2). The sample is now also read in blocks of 10 consecutive frames (rotation only); each block's shadow is found on its own with the rings about the whole sample's centre, and the blocks' pieces that reach beyond the 30 px band the whole sample's edge can grow over (PENUMBRA_MAX_PX) are added whole. A pixel shadowed on most frames of any longer stretch is shadowed on most frames of one block, so nothing finer is needed; pieces within the band are a block redrawing a static stop's edge and are left out, so a sweep without moving hardware keeps its mask bit for bit. Part B's principle is kept: everything is still decided on the first part of the sweep. Measured (GPU, against merged rc175 eac0833c8, 19 sets): half-image lysozyme P4_1 2_1 2, ISa 4.1 -> 9.7, R_meas 240 -> 113%, CC1/2 0.955 -> 0.979, R-free(model) 0.331 -> 0.282 (last night 6.9 / 116%). p.mtz md5-identical on 14 of 19 (8v4o, 9rci, 5ky6, 6yqf, 6ze4, 5reo, 9qw8, insu_I low-ISa, insu_H notwin, myob powder 1/2, myob split, nothing_1/2); changed: 5nw5 (a dim crescent at the stop edge now masked: ISa 8.4 -> 8.9, CC1/2 0.937 -> 0.980, R-free 0.462 -> 0.429), 7atg (0.85% added at the detector edge, statistics unchanged), 9min (0.03%, fail -> fail unchanged), 5jk4 (<0.01%, unchanged). Cost: pre-scan +0.1 s on HDF5 sweeps, +1-5 s on large file-per-image (CBF, marCCD) sweeps, where the blocks are re-read and their masks computed on the host. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 23:55:33 +02:00
leonarski_fandClaude Opus 5.5	a825fb6cba	rugnux: a rotation frame the sweep's cell refuses is tried on that cell scaled ... A rotation frame is refined on the sweep's one cell (orientation and beam per frame, never the cell), and a crystal whose cell grows with the dose leaves that cell behind: the per-frame gate (20 % of the in-resolution, non-ice spots on the lattice) then refuses the frames it misses as if the crystal had gone. A frame that fails the gate is now tried once more on the sweep's cell scaled isotropically - steps of 0.1 %, out to +-2 %, nearest first - to the scale that puts most of its non-ice spots on the lattice, re-refined there, and integrated on it where it then clears the gate. The scale must gain more spots than sqrt(count), so forty trial scales cannot turn a chance spot on a sparse frame into a frame. Ice-ring spots are not counted, as the gate does not count them. A frame the sweep's cell already admits is untouched, and only the per-image integration path does this (not the first-pass validation scoring). Found on 5ky6 (P 1 21 1, 564 x 0.5 deg, B rising by 46 A^2 over the sweep), which part B (5b9144410, decisions before pass 2 from the first part of the sweep) turned from P 1 21 1 into P 1 2 1. The mechanism: - The first pass now indexes on [0, 141 deg). A per-frame probe of the isotropic scale that puts most spots on the canonical lattice reads -0.1 % on frame 0, +0.4 % at 50 deg, >= +0.6 % from 115 deg on (at frame 230: 62 spots on the lattice at the sweep's cell, 112 at +0.6 %). Sampling the whole sweep (before part B) handed pass 2 a cell averaged over the dose, so frames to ~147 deg and part of the last wedge still cleared the gate; the early-range cell is the low-dose cell, and every frame from 113 deg on was refused (221 of 564 frames merged, completeness 83 %). - The axial 0k0 row is recorded in those frames: the merge held 1 screw-absent axial reflection instead of 15, so the 2_1 screw could not be decided (P2 by default), and R-free 0.243 -> 0.263. - Separately, on the early range the FFT candidate matched a false C-orthorhombic setting (gamma 92 deg) whose refutation fell straight to triclinic instead of P-monoclinic, so the cell was refined free (cell dev 0.64 -> 1.03 %). Not changed here (see the report). Integrating every frame from the sweep's cell instead (the sparse-lattice switch forced on) also recovers P 1 21 1, but at 1.63 A with ISa 5.0 and R_meas 45 %: the late frames are then integrated on a cell they have left. Result (GPU, battery verdicts against the rc175 full run 7c9a6c7d1): 5ky6 fail P 1 2 1, d 1.50, R-free 0.263, compl 83.0 % -> pass P 1 21 1, d 1.62, R-free 0.234, compl 99.6 % (pre-part-B: P 1 21 1, 1.54, 0.244, 96.9 %) 9rci pass -> pass, compl 86.5 -> 94.7 %, d 1.81 -> 1.76 9qw8 pass -> pass, R-free 0.329 -> 0.326, ISa 5.2 -> 5.5 8v4o fail -> fail (its part-B lattice loss is untouched), compl 22.3 -> 23.1 % p.mtz md5 unchanged: 5nw5, insu_I_x06da_low_isa, nothing_1, nothing_2, myob_x06da_powder_2, 7atg, 5reo, 6fvz, 6hwj, 6pxc, 8tha, 9crw, 9jzo, 6zqr, 6rym, thau_x10sa_0p1deg, cytc_x10sa. Catch2: Rugnux_Rotation, RotationIndexer* pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 21:56:46 +02:00
leonarski_fandClaude Opus 5.5	2981aedd3e	Broker: a collection can be followed from /start, its start message comes in the event stream ... With a DECTRIS detector the broker starts the writers - and so learned the collection's start message - only when the detector's own start message arrived, after /start had returned. A reader started right after /start found no collection (404) and had to wait for, and race with, the detector. - /start registers the collection in LiveCollection (Expect) as it is accepted, so GET /live/events is accepted from then on. The image pusher fills in the start message when it starts the writers (Begin) - at once on the FPGA path, when the detector's start message arrives on the DECTRIS path. - The event stream's first event is now `start`, carrying that start message (CBOR, base64) with run number, file prefix, image count and images per file; it replaces `collection`. Until it is known the stream only keeps alive. file and end follow as before. - However the measurement thread ends, it ends a collection the pusher did not end (EndIfOpen), so a cancel or a failure before the detector streamed reaches the reader as `end` with an error. The wait is event-driven: it ends with the start message or with the measurement. - GET /live/start.cbor is removed: the stream carries the start message, one route less. - rugnux takes the start message from the stream (BrokerFeed::WaitForStart), so it can be started right after /start; files reported before it attached the reader are replayed to it. Tests: LiveCollection_ExpectedAtStart; BrokerFeed_StartFromEventStream (an event stream whose start message comes late, then a file and the end). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 17:03:09 +02:00
leonarski_fandClaude Opus 5.5	509a54ef92	Broker: tell a reader which data files of the current collection are closed ... So that rugnux can process a collection while it is written, the broker reports each data file once a writer has closed it and renamed it into place: - Writer: FileWriter takes a callback, called for every closed data file. StreamWriter uses it on TCP to send an acknowledgement with ack_for = FILE_CLOSED (new TCPFrameType 10): file number and the images the file holds. TCP protocol version 5. Files closed at END are reported before the END acknowledgement. - LiveCollection (image_pusher): the start message the writer of the master file received, the closed files, the end. Fed by the TCP pusher (FILE_CLOSED acknowledgements; end once every writer acknowledged END) and by the in-process HDF5 pusher (the FileWriter callback). The ZeroMQ pusher has no back channel and offers none. - Two routes, protected by the dataset's bearer tokens like the other dataset routes: - GET /live/start.cbor: that start message; - GET /live/events: server-sent events (collection, file, end, superseded) with the history first and a keepalive every 10 s. One follower at a time; a second gets 409. Documented in jfjoch_api.yaml; they add no schema, so the generated C++ model is unchanged. - The HTTP server's thread pool is set explicitly (16, up to 64): an event stream holds a thread for the whole collection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 15:52:21 +02:00
leonarski_fandClaude Opus 5.5	5b91444102	rugnux: take the decisions before pass 2 from the first part of a rotation sweep ... The pre-scan sample (beam-stop shadow, defective pixels, spot width, the background beam centre) and the first pass's frames (the spread and wedge schemes, the validation frames, the prefetch) now come from [start, PrepassEnd) - the part of the sweep the geometry pre-pass integrates - instead of the whole sweep. Rugnux::PrepassEnd is the one place that range is defined, for the pre-pass and for these decisions alike, so any change to the pre-pass range (such as a minimum rotation) moves both. Stills keep the whole range. With rugnux following a collection while it is written, nothing before pass 2 then waits for the second part of the sweep: the pre-scan, both first passes and the pre-pass run while it is collected, and pass 2 starts reading as soon as they are done. An offline run takes the same decisions from the same frames, so the streamed result still equals the offline one by md5 (in-house JUNGFRAU 4M thaumatin replayed at one data file per second, also at -N 8 and with the replay paused and resumed): 2.4 s from the last data file to the result, against 8.0 s for the offline run. Result-changing. p.mtz md5 at af3ea9cca -> here (with the 120 deg minimum of the previous commit), GPU: myob de2ab820 -> 9bdd6916, cytc 3e5d2f91 -> 62e744a1, thau bc5e3de3 -> 3ec88c34, kdp cf83ee5d -> cdc2c068, 8a1a 50ab659e -> 309ea9a8, 8qaw d3287922 -> c2849d52; CPU: myob 3f9d12d8 -> 18c7ac2f, cytc d30c9969 -> 40e17d97, thau 4cdbdd94 -> f14e52da, kdp f1d801f3 -> eb930714. Sweeps of 120 deg or less are unchanged by this commit (their early range is the whole sweep). Needs the battery before it is merged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 14:29:32 +02:00
leonarski_fandClaude Opus 5.5	a291c01c09	rugnux: the geometry pre-pass integrates at least 120 deg of the sweep ... The two-pass geometry pre-pass integrates the first --prepass-fraction (0.5) of the sweep, but never less than 120 deg of rotation (header oscillation), so a sweep up to 120 deg is pre-passed whole and one up to 240 deg gets 120 deg. Wider sweeps are unchanged. The range is computed in one place, Rugnux::PrepassEnd. Evidence (fraction sweep with the rc175-p1-baseline binary, 10 sweeps of 360 deg and 10 of 180 deg; pre-pass distance vs the full-sweep fit): median error 0.081 % at 90 deg, 0.042 % at 120, 0.042 % at 150 and 0.033 % at 180 on the 180-deg sweeps; 0.032 / 0.021 / 0.013 % at 120 / 150 / 180 deg on the 360-deg sweeps. 120 deg on a wide sweep matches what the whole-sweep pre-pass gave a 180-deg sweep. Below ~60 deg the pre-pass misses the distance outright (6rym: held at the header, 3.4 % off; 6vww and 6fwc: distance not freed). Merged quality moves by under 1 % at any angle >= 90 deg. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 14:10:53 +02:00
leonarski_f	889418e00e	Merge branch 'xds-plugin-license' into rc175	2026-10-08 11:13:48 +02:00
leonarski_fandClaude Opus 5.5	2629f3821c	XDS plugin 1.1.0: mask module/chip edges, pass saturation through; recommend Durin ... Plugin: - Module-edge (bit 30) and chip-edge (bit 31) pixels are marked -2 (untrusted). Chip-edge pixels are split from a larger pixel with integer division and read low at low counts; feeding them to XDS as real pixels was what made the three plugins disagree in CI. - A saturated pixel in a signed image (INTx_MAX) is passed through with its value, so XDS counts it as an overload; only the error marker (INTx_MIN / UINTx_MAX) becomes -1. - Release version 1.1.0 set in CMake (library name and start message). VERSION_* in plugin.h stay 0: XDS refuses a plugin whose info-array major version is not 0. - Built and packaged on Linux only; the untested macOS archive is dropped. - The XDS linking permission is removed from LICENSE; the plugin is GPL-3.0. CI: - XDS.INP beam centre corrected to 1091/1137 (ORGY was 7.7 px off, which put IDXREF on a knife edge where one plugin's run fell into a false minimum). - Each XDS case must reach CORRECT, IDXREF spot SD must be < 2 px, and all file layouts read by the same plugin must give identical SPOT.XDS. Docs: recommend Durin (PSI or Global Phasing build); mask-bit table rewritten from measurement. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-08 10:56:34 +02:00
leonarski_fandClaude Opus 5.5	11122e4924	Docs: battery report of the rc.175 branch after phase 1, with commentary and figures ... Full open and in-house battery on 9ced4c444 (run 20261007-2139_9ced4c_rc175-p1-baseline) against the rc.173 battery, without the REFMAC check: R-free figure and numbers are kept from the rc.174 battery (20261005-2102_3770f4_rc174-final-refmac). No verdict changed from rc.174; 2ygz is new and fails. Two sets lost quality at the 50 % pre-pass default (outlier-laden outer shell on 5jk4, runaway cut on the in-house split myoglobin set); fraction 1 restores them, but nearby fractions do too and a fixed cut gives equal statistics, so the pre-pass only tips two existing weaknesses (an unmasked dead edge row; a noisy half-set CC1/2 on a multi-domain crystal). Time, zoomed time, in-house time and resolution figures regenerated with paper_plots.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-08 00:15:34 +02:00
leonarski_fandClaude Opus 5.5	1a970eef91	docs: regenerate THIRD_PARTY_NOTICES.md (Mbed TLS replaces curl, XDS plugin note) ... Generated from the root file by the update_version.sh step, which otherwise runs only on a version bump; the published copy still listed libcurl. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:21:18 +02:00
leonarski_fandClaude Opus 5.5	89207fe2bd	viewer: cpp-httplib + Mbed TLS instead of libcurl for the broker client ... The viewer's live-broker client goes back to cpp-httplib, which the broker already fetches; libcurl was only there for Kerberos, which is no longer wanted. HTTPS comes from Mbed TLS 4.2.0, fetched and statically linked for viewer builds only, so one TLS backend now serves Linux, Windows and macOS (the macOS viewer previously had no TLS at all, and Linux needed host OpenSSL >= 3.0). cpp-httplib verifies servers against the OS trust store: the Windows certificate store, the macOS Keychain, the CA bundle on Linux. CPPHTTPLIB_MBEDTLS_SUPPORT is set on jfjoch_viewer only, not on httplib::httplib, so the broker stays plain HTTP. JFJochHttpReader.cpp is the only viewer source that includes httplib.h, and the header keeps httplib::Client opaque so <windows.h> stays out of the other viewer units. Checked on Linux: a viewer-only build links Mbed TLS statically, with no curl and no system libssl; a standalone client built against the same httplib + Mbed TLS fetches from real HTTPS hosts and rejects expired, self-signed and wrong-host certificates. Windows and macOS are untested. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:16:04 +02:00
leonarski_fandClaude Opus 5.5	36366295f2	Windows: viewer installer and rugnux zip from one build ... The per-generator CPack config written for macOS is now generic and shared: the installer/.dmg generator gets the viewer, the archive generator (TGZ/ZIP) every other component. A Windows viewer-only build therefore packs NSIS;ZIP - the jfjoch-viewer installer and rugnux-<version>-win64-<variant>.zip, name unchanged - and the cuFFT DLL is installed into both components. CI folds build-rugnux-windows into build-windows (cuda and nocuda, so the Windows rugnux gains a -cpu zip) and checks the zip holds rugnux.exe, no viewer, and in the CUDA variant the cuFFT DLL. JFJOCH_RUGNUX_ONLY remains for the aarch64 cross build. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 21:04:17 +02:00
leonarski_fandClaude Opus 5.5	07ae3c3a40	XDS plugin: GPL linking permission, own tgz with its notices, macOS build; one portable CI build ... LICENSE: the section-7 addendum is rewritten as numbered additional permissions. CUDA libraries are treated as System Libraries (Intel MKL dropped - never linked). New: the XDS plugin may be linked with the closed-source XDS, valid only while it contains no third-party GPL code. The plugin links HDF5, zlib-ng, Zstandard and Compression (LZ4, Bitshuffle, Bitshuffle h-perf) and nothing else - no FFTW. In a portable (viewer-only) build it is its own CPack component, shipped as jfjoch_xds_plugin-<version>-<os>-<arch>.tar.gz with LICENSE, a short xds-plugin/THIRD_PARTY_NOTICES.md and only those six licence texts. On macOS it now exports only plugin_* (--exclude-libs does the same on Linux). The RPM/DEB packaging is unchanged. The viewer-only build already compiled everything rugnux needs, so it now packs one archive per program: Linux three .tar.gz (viewer, rugnux, plugin); macOS the viewer .dmg plus rugnux and plugin .tar.gz, each generator getting its components through CPACK_PROJECT_CONFIG_FILE. CI merges build-viewer-tgz + build-rugnux-tgz into build-portable-tgz (cuda and nocuda, so Linux rugnux gains a -cpu archive) and the two macOS jobs into build-macos. Both check that FFTW is neither on the plugin's link line nor in its symbol table and that it exports only plugin_*. The bare .so upload from the RPM job is dropped. RUGNUX_ONLY stays for aarch64 and Windows. Verified locally (Linux, CPU, CI flags): three tarballs, the plugin link line is exactly the list above, 0 FFTW symbols, four exports, system-only dynamic deps; XDS loads the packaged plugin. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-07 20:29:19 +02:00
leonarski_fandClaude Opus 5.5	f97700e084	battery: read 8owm and 8rud as the deposited .cbf.gz; 6ZR0's zip is intact ... The MXRDR zips for both sets hold gzip-compressed frames. Our staging had expanded them to .cbf. rugnux reads .cbf.gz natively, so the manifest now points at the files exactly as deposited. 6ZR0's zip, as the Keele repository serves it, has a readable central directory with all 1060 members. Our earlier download was short, which lost the last frame. Remove the statement that the repository's zip is damaged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 19:59:40 +02:00
leonarski_fandClaude Opus 5.5	1227d4168f	rugnux: drop the P1 cross-check; --finalist-ledger makes its own P1 merge ... The P1 cross-check (<prefix>_P1.mtz) and --no-p1-crosscheck are gone. The tail waited for that merge after the reflection files were written: 12-18 s on a large rotation set (7200 frames), ~2 s on a small protein. A user who wants a P1 merge re-runs with -S 1. --finalist-ledger keeps working: it now starts a P1 merge of its own beside the analyses (made only when the flag is given) and folds the ledger from it. The twin-immune zone evidence, which was read on the cross-check merge, moves with it and is reported only with --finalist-ledger. The unmerged MTZ was built in the shadow of the cross-check merge; it is now built beside the reflection files, from the point where the model validation has settled the indexing. WriteMtzReflections loses its amplitudes flag (only the cross-check wrote an intensity-only MTZ). The merged outputs (p.mtz, .hkl, _unmerged.mtz) are byte-identical to the base on the gate matrix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 19:04:40 +02:00
leonarski_fandClaude Opus 5.5	570b8ac31c	1.0.0-rc.175 ... VERSION and the strings update_version.sh derives from it (API spec, generated broker model and Python client docs, frontend, docs, FPGA and PCIe driver versions). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:14:39 +02:00
leonarski_fandClaude Opus 5.5	08ca1d5764	Move the hot-pixel finder from rugnux/ to image_analysis/hot_pixels/ ... A pure move: HotPixels.{h,cpp} and HotPixelsGPU.{h,cu} are compiled into JFJochImageAnalysis (like beam_stop/) instead of JFJochRugnux. Only include paths and the CMake source lists change; the objects are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 14:05:37 +02:00
leonarski_fandClaude Opus 5.5	280ff0a4a0	rugnux: remove --estimate-beam-center and the spot-symmetry beam-centre estimator ... --estimate-beam-center was opt-in, off by default and used by no battery run. Removed with it: the half-turn spot-symmetry estimator (image_analysis/geom_refinement/BeamCenterFromSpots and its tests, used nowhere else), its pre-scan branch and paired-frame sample, the sparse-sweep extra reads, the BEAM_CENTER_* constants used only there, beam_center_placed_, the spindle fit that only ran inside it (--fit-spindle / --no-fit-spindle, ProcessConfig::fit_spindle; its estimate was only logged), the viewer's "Measure beam centre before indexing" job option and the RugnuxCommandLine emission. The background centre (FindBeamCenter on the pre-scan projection), the asynchronous beam-centre capture, the beam-centre check that indexes at both centres and the measured centre's bound on the post-refinement are unchanged, and so is the pre-scan sample for the shadow, the defective pixels and the spot width. Warnings that pointed at --estimate-beam-center now point at --beam-x/--beam-y. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:35:40 +02:00
leonarski_fandClaude Opus 5.5	082f77eaa8	rugnux: remove the beam-centre ladder (--beam-center-search) ... The ladder stepped the beam centre a pixel at a time after a rotation first pass that indexed fewer than half the validation frames. A battery with it switched off changed no output, so it goes: the option, its usage text, ProcessConfig::beam_center_search_pxl, the once-per-run flag that kept probes, walks and later passes from re-running it, and the docs. The background centre measurement, the beam-centre check that indexes at the measured centre, the merge-judged arms and the post-refinement of the centre are unchanged. rugnux now rejects --beam-center-search as an unknown option. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:35:40 +02:00
leonarski_fandClaude Opus 5.5	8c7c222e7d	rugnux: the two-pass geometry pre-pass integrates the first half of the sweep by default ... --prepass-fraction now defaults to 0.5; --prepass-fraction 1 restores the pre-pass over the whole sweep. The first-pass indexing and the pre-scan still sample the whole sweep, and the second pass integrates every frame and is the only one written. A full battery at 0.5 (with the ladder off) was quality-neutral against the rc174 baseline except for one small-molecule set that crashed, fixed in the previous commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:35:40 +02:00
leonarski_fandClaude Opus 5.5	37b7fdd38d	rugnux: the terminal shows numbered steps with timing; the full log goes to <prefix>_log.txt ... Standard mode prints only steps, one-line results with the time since the line above, a rule closing each step with its total, warnings (!!) and errors (** ERROR:), and a summary at the end. Every log line of every module goes to <prefix>_log.txt instead, at debug level and timestamped, opened with version, build, GPU, threads, command line and the header geometry. -v puts the whole log back on the terminal. - RugnuxObserver gains OnStep / OnNote. RunPipeline posts them only on a pass that makes the result (the observer is held in steps_, nullptr on probe passes and on copies run beside the main pass); RunAllPasses opens steps for the checks between passes (rotation scale, metric symmetry, geometry walk) and notes its re-runs. OnPhase is written to the log as a marker. - RugnuxConsole (CLI only): on a terminal a progress bar and a spinner status line drawn in place with \r, durations only; not a terminal, plain ASCII with the time of day on every line and quarter milestones for long passes. isatty/_isatty, no environment variables. A warning repeated by later merges is shown once more on one line and then counted per step. - The CLI's former stdout blocks (space-group search, merging statistics, per-image cost, processing time, tNCS/twinning/anisotropy in --mode scale) go to the log; the calibration result block stays on the terminal as that mode's summary. The MX summary takes its rows from the report's own SUMMARY (ReportDocument::summary_rows); WriteResultReport returns the document it rendered. HDF5's automatic error-stack printing is switched off. - The banner keeps the "Version X (git abcdef" form battery.py reads off a bare invocation; battery.py's failure note now reads p_log.txt, whose lines are not wrapped. Output only: p.mtz is bit-identical on the three in-house reference sets. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-07 12:11:21 +02:00
leonarski_fandClaude Opus 5.5	76b3956e11	French-Wilson once, on the written merge; none for the P1 cross-check ... Every full-statistics rotation merge computed French-Wilson amplitudes - the search candidates, the pinned low/high pair, the final, and the P1 cross-check - although only the written merge's amplitudes are used, and those were recomputed anyway with the anisotropic prior. RotationScaleMerge gets SetFrenchWilson(); Rugnux turns it off for its merges and makes the amplitudes once after the anisotropy analysis (anisotropic prior when a tensor was fitted, isotropic otherwise). --mode scale keeps the engine's own amplitudes. p_P1.mtz is now intensities only (owner decision): the F/SIGF/F(+)/F(-) columns are omitted; every intensity column is bit-identical to before. French-Wilson was 29% of the CPU samples in the large-cell tail (8tyy P1 cross-check window, ~84 core-s). p.mtz md5 unchanged on myob, cytc, thau, kdp, 8tyy (GPU build). Wall: 8tyy 173.9 -> 150.9 s; cytc/thau -0.2..-0.3 s, myob unchanged (3 interleaved repeats). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi	2026-10-06 23:10:26 +02:00
leonarski_f	a395f358ef	1.0.0-rc.174 (#84) ... * Rugnux: Performance improvements on GPU and CPU (more of the pre-scan and of scaling on the GPU, faster CPU spot finding and crystal refinement), with unchanged results. * Rugnux: More robust processing - patches of persistently hot pixels are masked, an inconsistent merge triggers a retry at the measured beam centre, and builds targeting different CPU levels give the same results. * Rugnux: Improved scaling and merging - reflections with an overloaded pixel are dropped, as in XDS, sparse rotation sweeps are scaled more reliably, and French-Wilson amplitudes use an anisotropic Wilson prior. * Rugnux: Improved space-group determination - glide planes in groups without a centre of symmetry, screw axes from short or weak axial rows kept when a higher group is adopted, and more reliable decisions on twinned and pseudo-symmetric crystals. * Rugnux: Improved small-molecule processing - spots that grow wider than the integration disk and split spots are integrated over their measured footprint, sparse lattices are integrated on every frame, and the `.hkl` file holds unmerged scaled reflections (SHELX HKLF 4). * Rugnux: Reads Rigaku d*TREK SMV images (Saturn CCD), including detector 2theta and encoded pixel overflows; home-source (rotating-anode) datasets were added to the validation battery. * jfjoch_viewer: Fixed processing failing at the end with "Wrong JPEG library version" on Linux; the merge window shows the space group with proper subscripts and a checklist of crystal pathologies. Reviewed-on: #84 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-10-06 14:03:18 +02:00
leonarski_f	84228bf8be	v1.0.0-rc.173 (#83) ... * jfjoch_broker: Optional per-dataset authentication - statistics, images and plots can require a bearer token, which jfjoch_viewer supports. * jfjoch_viewer: Dark mode and a theme-matched colour scheme, a magnifier panel, and simpler contrast and background controls. * Rugnux: Multiple performance improvements on GPU and CPU (CPU-only processing up to 40% faster, faster image decoding on ARM), with unchanged results. * Rugnux: `--model` rigid-body refinement runs on the GPU, and the model-validation check is faster and more reliable. * Rugnux: Improved scaling and merging - error model, outlier rejection, absorption correction and French-Wilson amplitudes now agree more closely with XDS and ctruncate. * Rugnux: Improved integration - radial background on powder and ice rings, crowded rotation data keep their reflections, and CPU-only builds integrate large unit cells as GPU builds do. * Rugnux: More robust detector geometry - measured beam centre, X-ray bandwidth and goniometer rate, and geometry refinement accepted only on significant evidence. * Rugnux: Merged files are written in the standard setting, or in the setting of a reference MTZ, structure-factor mmCIF or model, with its free-R flags. * Rugnux: Richer report - ice and powder rings, further lattices, superstructure candidates and mosaicity, with warnings worded as prompts to check. * Rugnux: Clear error messages when a data set needs more GPU or host memory than is available. Reviewed-on: #83 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-29 15:57:32 +02:00
leonarski_f	6dfe065365	v1.0.0-rc.172 (#82) ... * Fixed `jfjoch_broker` cancelling every data collection with a CUDA "out of memory" error after long operation: GPU memory no longer leaks with each collection. * Rugnux scales a rotation sweep until the per-frame scales settle instead of for a fixed three rounds, and says so when they did not - merged intensities, and the space group, resolution cut and frame rejection read off them, change accordingly; `--scaling-iterations` is now the cap on that loop (default 100). * Rugnux places every frame of a marCCD, SMV or miniCBF series at the spindle angle its own header states, so a series with missing frames, or with angles written modulo 360, is no longer read at the wrong geometry or refused. * Every rotation run writes two diagnostic files beside its reflections: `<prefix>_detector.jpg`, the detector projection with the pixel mask and the detected beam-stop shadow drawn on it, and `<prefix>_plot.txt`, one row per image. Reviewed-on: #82 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-22 06:48:37 +02:00
leonarski_f	30b6800289	v1.0.0-rc.171 (#81) ... * Rugnux: basic support for CCD images (marCCD, SMV) and for gzipped miniCBF. * `jfjoch_viewer`: opens the CCD formats, and fixes to the dataset plots. * Documentation updates. Reviewed-on: #81 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-17 14:42:52 +02:00
leonarski_f	cb5a2f032a	v1.0.0-rc.170 (#80) ... * Fixed a `jfjoch_broker` crash during indexing: sorting no longer misbehaves on non-finite values, and GPU FFT indexer kernel launches are now error-checked. * rugnux needs about a third less peak memory to scale, merge and post-refine rotation data, with identical results. * `rugnux --model`: the placed coordinate file carries the space group its own coordinates obey, and says so when that is not the group the reflection files beside it carry. * `jfjoch_viewer`: fixes in the dataset plots, inspector and layout; spot markers lose their black outline by default (a checkbox under "Image features" restores it) and the highest-pixel markers are white boxes around the pixel. Reviewed-on: #80 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-16 18:17:46 +02:00
leonarski_f	9aae0c2ba7	v1.0.0-rc.169 (#79) ... * Building Jungfraujoch no longer needs zlib or Eigen installed on the machine, and the dependencies the build fetches are pinned and updated to current releases. * rugnux: improvements in indexing, lattice selection and geometry post-refinement, which index crystals that previously returned no lattice and keep the better of the two geometries a run measures. * rugnux: improvements in beam-centre measurement, beam-stop detection and space-group determination. * rugnux: the unit cell reported with a determined space group now obeys that group - a cell whose symmetry was confirmed from the intensities is re-refined under it, and a cell the group cannot describe is reported with a warning rather than as it stands. * rugnux drops the stretches of a rotation sweep whose removal measurably improves the merged intensities and reports what became of every frame, and decides the resolution cut on the crystal's own diffraction rather than on its ice rings. * The rugnux results report is machine-readable - every line that is not `KEY= value` data starts with `#` - and states the build it was written by, its authorship and its terms of use (`REPORT_VERSION= 8`). * `jfjoch_viewer`: improvements in the file manager (CBF frames beside HDF5 datasets, a remembered root), the dataset plots, the inspector and the image statistics, plus a settable font size, a view of the rugnux results report, usable performance over a remote display (`ssh -X`) and a reset of all settings to defaults; the reciprocal-space window is removed. * Broker fixes around DECTRIS collections and dark-mask calibration: re-initialising after a run that never started no longer freezes the broker, a cancelled calibration is abandoned instead of reported as done, and a collection whose start message never arrives ends by itself. Reviewed-on: #79 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-15 17:09:31 +02:00
leonarski_f	77bc0cfe52	v1.0.0-rc.168 (#78) ... * rugnux is substantially faster - a corpus of 145 rotation datasets processes in about two thirds of the time - with identical results. * A crystal whose lattice looks more symmetric than it is because the beam centre is off is no longer processed on the wrong cell. * rugnux prints at startup, and writes at the foot of every results report, a short acknowledgement of the X-ray research community whose methods it implements and of the open-source projects it builds on; `ACKNOWLEDGEMENT.md` now ships in every package beside `LICENSE` and `THIRD_PARTY_NOTICES.md`. Reviewed-on: #78 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-10 13:51:16 +02:00
leonarski_f	a39fd29f77	v1.0.0-rc.167 (#77) ... * `rugnux --model` reports CC(model, data) - the correlation of the merged intensities with the placed, scaled model - by resolution shell, on the same shells as CC1/2, with the reflection count and a significance for each. * `rugnux --model` fits the model's scale, anisotropic B and bulk-solvent parameters on the working reflections only, so the R-free it reports is measured against a model no free reflection helped scale. * The bulk-solvent parameters of `rugnux --model` are searched over their physically meaningful range instead of being fitted without bounds, so a model is never scaled with a solvent term that has silently switched itself off. * The rigid-body placement of `rugnux --model` uses the same bounded bulk solvent as the reported fit, so a model is no longer placed against a target carrying a solvent term with no physical meaning. * `rugnux --model` puts the model into the data's own description of the lattice before placing it, so a model whose cell is written on other axes - I-centred where the run indexed C-centred, a different unique axis, a permuted orthorhombic cell - is placed rather than scored where it was read; `MODEL_CHANGE_OF_BASIS=` and `MODEL_SETTING_AS_READ=` report it when it happens. * The rugnux results report opens with a summary - `VERDICT=` (`OK`, `WARNINGS`, `UNUSABLE`, `FAILED`), `VERDICT_TEXT=`, `PATHOLOGY_FLAGS=` with one closed-vocabulary code per condition that warned, and the `WARNING:` lines, which used to close the file - and the sections after it are renumbered 1-5 with no gaps. * `rugnux --developer` writes the full results report - the pipeline-internal keys and the long explanations the default report now leaves out - and `--finalist-ledger` adds the evidence for every space group the search considered, not only the one it adopted. * The results report warns when the merged data carry no usable signal and when too little of reciprocal space was measured inside the fitted resolution, and omits `FITTED_RESOLUTION` where the CC1/2 curve it is fitted on never falls off. * rugnux detects translational pseudo-symmetry and reports it under the `PSEUDO_TRANSLATION` flag as `TNCS_DETECTED=` and the `TNCS_*` keys - a translation the merged data are exactly invariant under is reported as `UNDECLARED_LATTICE_TRANSLATION=` under `LATTICE_TRANSLATION` instead - and a detected pseudo-translation can no longer buy a false screw axis in the space-group search or hide a twin from the L-test (`L_TEST_VS_TNCS=`). * The space-group search determines glide planes from zonal systematic absences, so a non-Sohncke space group such as P 2_1/c or Pbca is named where the run previously stopped at its Sohncke subgroup; `SOHNCKE_SPACE_GROUP=` carries the best Sohncke group beside it on every run that searched, and a centre of symmetry is never claimed. * Where the cell metric carries more rotational symmetry than the Bravais class the indexer named, the extra rotations are put to the intensities and the space-group search is asked again on the metric's own cell - adopted only where the intensities confirm the higher symmetry - so a lattice that is nearly but not exactly hexagonal, or whose reduction landed in a sub-cell, still reaches its true point group. * Systematic-absence calls rest on the evidence rather than on counts: a screw axis whose absent class the data show extinct is no longer refused because a handful of reflections in it read as present, and `SPACE_GROUP_ALTERNATIVES=` no longer drops a candidate that differs only on a zone the sweep never measured. * A reference correlation measured on too few reflections is refused instead of scored zero, so a run given a reference MTZ is no longer reindexed on an operator that mapped almost everything outside the reference's coverage. * A frame counts as indexed from 6 spots on its lattice rather than 9, so a weakly diffracting crystal whose frames cannot carry 9 is no longer refused the lattice it fits; `--min-indexed-spots` overrides it. * `-C` accepts a known cell in any equivalent description - conventional or primitive, centred or not - instead of only the reduced primitive form, so a centred cell given the way it is published no longer makes the run report that it found no lattice. * Each reflection is corrected for the sensor's quantum efficiency at the angle it meets the detector (attenuation lengths from the NIST tables, which also fixes the spot-width parallax term on CdTe) and for the attenuation of the flight path between the sample and its pixel; `--flight-path air|helium|vacuum` declares the medium - default air, since no file states it - and the report says what was assumed and what it was worth. The unmerged MTZ records the factors in new `QE` and `FLIGHT` columns beside `LP`, so raw counts are `I / LP * QE * FLIGHT`, and `_process.h5` in new optional `qe` and `flight` datasets. * Rotation geometry post-refinement fits the crystal and the detector at once, against the observed spot positions and the observed rocking angles together, so the refined distance depends far less on how wrong the file's distance was. * A coarsely sliced sweep integrates correctly: partials are joined into one rocking event by angle rather than by frame count, so two crossings of the Ewald sphere are no longer summed into one full, and at 0.5 degrees per image or coarser the per-frame geometry refinement accepts a spot whose miss the exposure's own rotation accounts for. * `rugnux --mode scale` reports the detector tilt and direct beam of the geometry it re-scaled at, instead of zeros that read as a flat detector, and no longer warns that no image was indexed on a run whose lattice came from its input file. * Every rotation run that determined a space group and merged reports what the mounting cost: `SPINDLE_LOST_UNIQUE_FRACTION=` is the fraction (0-1) of unique reflections the mounting made unmeasurable under the measured point group, also written to the master as `/entry/MX/spindleLostUniqueFraction` and what the mounting warning fires on; `SPINDLE_SYMMETRY_AXIS_ANGLE_DEG=` / `SPINDLE_SYMMETRY_AXIS_ORDER=` describe the mounting in the `--developer` report. * Stills and grid scans carry a per-image `spindle_blind_fraction` - how much of a rotation sweep's blind cone this orientation would make unrecoverable, 0.5 and above calling for a second orientation - through the CBOR stream, HDF5 (`/entry/MX/spindleBlindFraction`), the plot and scan-result APIs, and the viewer and frontend plots; an absent value means the frame could not be assessed and is not a 0. * The results report's `REPORT_VERSION` is 7. Reviewed-on: #77 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-09-09 07:25:13 +02:00