18 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	6dd36436b6	rugnux: the unmerged MTZ is written on request again, not on every run ... 0fed51f76 made <prefix>_unmerged.mtz default-on so a run that turned out to need it would not have to be repeated. It costs more than that is worth: measured on a dense rotation dataset, it adds 2.6 s to a 7.0 s --mode scale run (+38%) and writes 272 MB - larger than the merged .mtz, .cif and .hkl put together, and paid by every run whether or not anything will read it. --export-unmerged asks for it again. --no-export-unmerged stays accepted and now does nothing, the mirror of what --export-unmerged did while the default was on. The two fixes that came with the default-on change are kept: the file is written in --mode mx and --mode scale and with --no-merge, and both write sites are guarded on the output prefix, so a prefix-less run no longer writes a file called "_unmerged.mtz". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016L1qig74oYQzfUJJZbbxFh	2026-08-27 16:04:52 +02:00
leonarski_fandClaude Opus 5	dee539a713	rugnux: --model takes the model's enantiomorph as a label, and stops flipping I(+)/I(-) ... Where the data were merged in the enantiomorph of the model's space group, --model reindexed the observed reflections by change_of_hand_op(). That was wrong, and the anomalous site list added yesterday is what showed it. The two groups of an enantiomorphic pair differ only in the translations of their operations. Their rotations are identical, so they transform hkl identically, share a reciprocal ASU, and split into Bijvoet hands identically - to_asu_sign returns the same answer for either, verified over 200k random indices for all eight enantiomorphic families. The label therefore carries no handedness at all, and there was never anything about it to undo. What does carry the hand is the indexing the data already have, from the diffraction geometry, and with it the sign of every anomalous difference. change_of_hand_op is the inversion, so reindexing by it swapped I(+) with I(-): it did not correct the hand, it flipped it, on the strength of a label the space-group search itself reports as undetermined. So the model's group is now adopted as a label and no reflection moves. On a tetragonal reference dataset the site list goes from ten unrelated atoms at around 3 sigma to the two ordered halides at 15.6 and 10.3 sigma followed by the methionine and every cysteine sulfur at 6.6 to 7.3 - the correct map was there all along and the reindex was inverting it. Whether the model and the data agree about the hand is a real question, and now an answerable one: a map whose deepest hole at an atom is below -5 sigma and deeper than its highest peak says they disagree, and the run warns and names that atom. It is deliberately not repaired by reindexing, which would make the two agree by construction and destroy the evidence for which of them is wrong. Verified by mirroring a model: the warning fires naming the atom at -10.1 sigma, where the unmirrored model reads +10.1. R-free is identical to four decimals across that mirror, which is exactly why it could never have arbitrated this. MODEL_HAND_OPERATOR becomes MODEL_ENANTIOMORPH_ADOPTED, there being no operator to report. REPORT_VERSION stays 4: both that version and the MODEL VALIDATION section it describes were added in this same unreleased cycle, so no report carrying the old key has ever left it and there is nothing for a consumer to have depended on. The unmerged export follows only the indexing operator now, the enantiomorph being a label. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014zTy4Bpi4pPHw4bybf7q2R	2026-08-27 15:09:24 +02:00
leonarski_fandClaude Opus 5	bc500d1578	rugnux: report what --model found, and attribute GEMMI's bundled headers ... Two things left open by the previous commit. A model that was validated left nothing in the results report. R-work, R-free, the bulk-solvent parameters and the map filenames went to the log and nowhere else, so a run given --model was, in the file anyone reads afterwards, indistinguishable from one that was not - and the WARNING line added last commit was the only trace of a model that failed. Section 10. MODEL VALIDATION now carries it: MODEL_FILE, the space group the model is in, R_WORK / R_FREE with their reflection counts, the bulk-solvent k_sol / b_sol and overall scale, the mean 2Fo-Fc density at the atom centres, the hand and indexing operators the written reflections were brought into the model's frame with, and MAPS_PREFIX. A model that could not be used gets MODEL_VALIDATION= NOT_PERFORMED and the reason instead, in the shape the merging section already uses. A run that asked for no model has no section at all. That moves WARNINGS from 10 to 11, which is what REPORT_VERSION is for: it is now 4. No existing key was renamed and no table column moved, so a consumer that greps for what it needs is unaffected; one that keys on the section number of WARNINGS is not, hence the bump. The four third-party headers GEMMI has always bundled - fast_float, half, pocketfft and tinydir - were never listed in THIRD_PARTY_NOTICES.md, riding instead under GEMMI's single MPL-2.0 row. They are other authors' code under other licences (Apache-2.0 OR MIT OR BSL-1.0, MIT, BSD-3-Clause, BSD-2-Clause), and MPL-2.0 does not speak for them. Each now has its own row and its own licence text, alongside the PEGTL and sajson entries added with the CIF parser, and a note records why the six are listed separately from GEMMI. Their terms live in the headers rather than in LICENSE files, so five of the six texts are kept by hand; COLLECT.sh says so and copies the one - PEGTL - that ships a file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 13:50:13 +02:00
leonarski_fandClaude Opus 5	2e8040c60c	rugnux: read mmCIF models with --model, and say when a model could not be used ... --model called gemmi::read_pdb unconditionally, so a coordinate file in mmCIF - which is how the PDB serves coordinates by default - was refused outright: Model validation: cannot read model 6G8A.cif: Incorrect file format (perhaps it is cif not pdb?) Worse than the refusal was what followed it. ValidateAgainstModel logged, returned an empty result, and the run went on to finish with exit code 0, no R-free, no maps and nothing in the report - indistinguishable from a run that was never given --model at all. A script that passed a .cif, checked the exit code and grepped for R-free simply got nothing back. Both read sites now go through read_structure_gz with CoorFormat::Detect, so PDB, mmCIF and mmJSON are all read, gzipped or not, and the format comes from the file's content rather than from its name. Detect is passed explicitly: GEMMI otherwise takes the extension and only falls back to the content when it does not recognise one, and a model arrives named however whoever produced it named it. That needed a part of GEMMI the vendored subset had trimmed away. mmread.hpp was already here but its CIF parser was not, so this restores what upstream GEMMI ships: read_cif/mmcif/json/mmread_gz, plus GEMMI's own copies of PEGTL (MIT, Colin Hirsch and Daniel Frey; 155 headers, 672 kB) and sajson (MIT, Chad Austin; one header). Both are listed in THIRD_PARTY_NOTICES.md with their own licence texts, PEGTL's collected by COLLECT.sh and sajson's kept by hand, its terms being a comment block rather than a file. json.cpp carries a one-line change: upstream keeps sajson at the root of its repository, outside the include tree this subset copies, so the include points at where the other bundled headers live. ModelValidationResult gains failure_reason, and a model that was asked for and could not be used now reaches the results report as WARNING: Model validation did not run: model bogus.pdb has no atoms or no unit cell in --mode mx and --mode scale alike. The run still succeeds - a merge that is good is not made bad by a model that was not - but it no longer does so quietly. One log line was making a claim it could not support: the indexing-ambiguity message said the ambiguity was "resolved against the supplied model" while being printed before the model is read, so it announced a resolution that a failed model never performed. It now says the model will be used; the reference branch, where the work really has already happened, keeps the past tense. Verified on the rotation test dataset with a deposited mmCIF, the same file gzipped, an mmCIF under a .pdb name, a PDB, and an unreadable file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 13:42:48 +02:00
leonarski_fandClaude Opus 5	e65cfdc675	rugnux: record in the report how the run was invoked, what it cost and what it ran on ... The results report said what the run determined but nothing about how it was produced, so a report read next to a lost shell history could not be reproduced or compared. Four keys in the header block, all new, none renamed - a consumer that greps for what it needs is unaffected, so REPORT_VERSION does not move: COMMAND_LINE= rugnux -o myrun --model model.pdb dataset_master.h5 WALL_TIME= 262.41 GPU_COUNT= 4 GPU= 4x NVIDIA A100-SXM4-80GB The command line is argv as one shell-ready line; an argument that would not survive being pasted back into a shell is single-quoted, so a file prefix with a space comes back as the one argument it was. It replaces the raw argv echo at the top of the run, which had no quoting at all. WALL_TIME is the whole invocation, timed from the top of the CLI. It is deliberately not result.total_time_s, which starts inside Rugnux::Run and so counts neither opening the file nor setting up the analysis - and which --mode scale never sets at all, having no ProcessResult of its own. It is printed on stdout as well, next to the processing time it is slightly larger than. The GPUs are the reason rugnux is fast, and until now nothing said whether any were being used. get_gpu_names() reports them per device and get_gpu_description() collapses repeats, so four identical cards read as one line rather than the same name four times and a mixed machine keeps one group per model. Both have a CPU-only implementation, so the JFJOCH_USE_CUDA=OFF build reports GPU_COUNT= 0 rather than failing to link. The same line is printed at startup, before the run rather than after it: a machine that turns out to have no GPU - a driver mismatch, a CUDA_VISIBLE_DEVICES left over from another job - is worth knowing about while there is still time to stop, not once the run has taken an order of magnitude longer than it should. GPU_COUNT= 0 is written with no GPU= line beside it, because the absence is the statement. The header comment claiming timing is deliberately absent from the report is now wrong and says so: rates and per-image costs stay on stdout, the total does not. Verified on the rotation test dataset in both build configurations, with and without CUDA_VISIBLE_DEVICES, in --mode mx and --mode scale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 13:09:54 +02:00
leonarski_fandClaude Opus 5	0fed51f767	rugnux: write the unmerged MTZ on every run, not on request ... <prefix>_unmerged.mtz is what another scaling program needs - aimless, pointless, careless, iotbx.merging_statistics - and a run that did not ask for it up front had to be repeated to get it. It is now written on every run that has an output prefix, in --mode mx and --mode scale and with --no-merge, alongside the merged files and replacing none of them. --no-export-unmerged turns it off; --export-unmerged is still accepted and now does nothing. The summed form is the default one, as before: each rocking event's partials combined into one full, written at the batch its rocking curve is centred on. --export-unmerged-partials still additionally writes the unsummed <prefix>_unmerged_partials.mtz for a program that would rather sum them itself. Both write sites gated on the export flag alone and would have written a file called "_unmerged.mtz" when there was no output prefix. That could not happen while the export was opt-in and a prefix was implied by asking for it; with the export on by default it would fire on any prefix-less run. They now use the same guard the merged writes use. Verified on the rotation test dataset: written by default, by --mode scale and under --no-merge, absent with --no-export-unmerged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016WmryXe8ASbNi632sUMfsa	2026-08-27 12:24:54 +02:00
leonarski_fandClaude Opus 5	5dafb9d21c	Docs: stop the rugnux examples implying that parallelism has to be asked for ... Every example command carried -N 32, which reads as though a run is single-threaded until told otherwise. It is not: the thread count follows the machine, and -N is there to limit it, or to lift the per-image loop's default ceiling of 16 workers per GPU. The examples drop the flag and the quick start says what -N is actually for. Two things the quick start left to be discovered elsewhere: -S takes a Hermann-Mauguin symbol or a space-group number, either spelling; and --model, where the model is isomorphous, pins the enantiomorph - which no measurement can decide - so the merged reflections come out in the model's hand and its space group, and on anomalous data that is what puts I(+) and I(-) the right way round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn	2026-08-26 21:31:13 +02:00
leonarski_fandClaude Opus 5	daac3c2da1	rugnux: report nine resolution shells, as XDS does ... The binning rule was already XDS's: equal steps in 1/d^2 between the lowest- and the highest-resolution reflection the merge kept, anchored on the data rather than on the nominal low-resolution cut. Only the count differed - ten shells against XDS's nine - so at the same resolution limits neither the boundaries nor the reflection populations matched, and a per-shell number could not be read across. Forced to nine on a tetragonal rotation dataset, rugnux now gives 4.27 3.03 2.47 2.14 1.92 1.75 1.62 1.52 1.43 against CORRECT.LP's 4.26 3.02 2.47 2.14 1.92 1.75 1.62 1.52 1.43; the hundredth of an angstrom in the first two shells is each program anchoring on its own lowest-resolution reflection, the two sets of survivors not being identical. This matters most to the comparisons that are made shell by shell. rugnux_vs_xds.py hands XDS's resolution range to rugnux but not its shell count, so its low-resolution R_meas column compared a shell running to 4.49 A against one running to 4.26 A - 814 unique reflections against 943. The outer shell was nearly immune, its inner edge being d_min*sqrt(n/(n-1)), which is 1.51 at ten shells and 1.52 at nine. --resolution-shells still sets any count. rugnux_stills_ab.py pins the count for both its arms and now pins nine, so its tables read the same way as everything else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn	2026-08-26 20:24:34 +02:00
leonarski_fandClaude Opus 5	e576184e2f	Docs: a quick start at the top of the rugnux page, and the indexing ambiguity written down ... The rugnux page opened with installation and reached its quick start around line 500 of 840, which is not where someone who wants to process a dataset looks. It now opens with four commands - the defaults, a reference MTZ, a model, and a pinned cell and space group, which is what people actually ask for - followed by the five files a run leaves and a few pointers. The old quick start keeps the detailed walkthrough under "Running rugnux", moved up to sit after the orientation sections. The indexing ambiguity had no prose anywhere, and neither did -z, which had one row of a table for four different jobs. A new section says what a reference MTZ does, what the ambiguity is, that it costs a rotation run an arbitrary frame and a serial run its CC1/2 outright, how to resolve it in each of the four cases, and that it is neither the enantiomorph nor twinning - the two things it is most often taken for. The three long reference pages carry a table of contents, from the MyST contents directive rather than a hand-written list, so it cannot go stale. The theme's sidebar already lists each page's H2 headings, so this adds the level below that and an in-page map; splitting the pages was the alternative and would have broken every anchor other documents link to. Two things the code says that the docs did not. --mode scale does not accept a reference MTZ on rotation data - the rotation scaler refuses it and the run stops - where the page offered it as one of the things to re-merge with. And CPU_DATA_ANALYSIS 10.9 still said stills use the reference as their per-image scale target, which stopped being true when stills were changed to scale against their own merge; the reference is their per-image ambiguity test and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vi1gV6Z45aZL5wLwe85Ksn	2026-08-26 20:04:45 +02:00
leonarski_f	dd67cffde6	Docs: bring the release notes and the rugnux page up to rc.164 ... The changelog section had grown to 22 entries written from the commits rather than for a user. Collapsed to 13, each saying what is gained before what changed, with related work merged - the six anisotropy entries into one, five space-group entries into two, five integration-radius entries into one - and four things added that had no entry at all: the packaging split, the statically linked cuFFT, the Ceres back-end drop and REPORT_VERSION reaching 3. CPU_DATA_ANALYSIS gains 9.5, the adaptive integration radius: the r80 measurement, the clamp, the constant-ring-area r3, the convergence test, why it applies to the final pass only and the density fallback. Three passages there had gone stale against the code: 13.1 still described the I/sigma quantile fallback that b90fcfb98 deleted, and 13.5 said the gate tests delta_B when it tests delta_B_linear and that the high-symmetry caution fires wherever the Laue class leaves one free direction, which 0da593b32 narrowed. RUGNUX gains an Installation section - there was no page anywhere saying where rugnux comes from - a synopsis, a worked first run naming the files that actually appear and the report keys to grep, and sections on the unmerged export and on diffraction anisotropy. Its report section list was one section behind: anisotropy is 9, warnings is 10. The packaging change reached further than the changelog implied, so the pages describing what a release ships were corrected too: the viewer tarball no longer carries rugnux or the two developer utilities, the Linux archives link cuFFT statically rather than shipping it beside the binary under an $ORIGIN rpath, the aarch64 archive has a higher glibc floor than the RHEL 8 ones and is untested on hardware, the archives unpack with no top-level directory, and the notices live per package under share/doc/jfjoch_<component>. The RHEL 9.5 vm_flags workaround in FPGA_PCIE_DRIVER is obsolete and now says so.	2026-08-26 15:16:02 +02:00
leonarski_fandClaude Opus 5	7e3ab8150d	Integration: widen the radius for the final pass only ... 48008e144 set the integration radius from measured spot width, and on one crystal that cost 21% of the indexed frames. The cause is not the indexer. The geometry pre-pass integrates, post-refines the detector distance and beam from what it integrated, and discards the intensities. PostRefine takes its positions, its weights and its reflection set from the integrator, and the observed position is a raw first moment over the signal disk with the background still in it. A flat background contributes nothing to the numerator, because the disk is centred on the prediction, but it adds n*b to the denominator - so every measured offset is pulled toward its prediction by I/(I+n*b), and n goes from 50 pixels at r1 = 4 to 113 at r1 = 6. A wider disk under-corrects the geometry. Measured: 225.978 mm against 225.944, and a slightly worse held-out residual. That is the entire difference the pre-pass hands on: 33 microns of distance and 0.02 pixels of beam. It is enough because that crystal's metric is 0.54 degrees off orthorhombic, so the de-novo Bravais choice sits on a knife edge - scanning the beam in hundredth-pixel steps flips it back and forth in islands, and both sides are bit-reproducible. Snapping the cell to 90 degrees costs a contiguous 66-degree stretch of the sweep, which is where the frames go. So the pre-pass now keeps the radius the run started with and only the final pass integrates at the measured width. Pass 1 becomes bit-identical to a fixed-radius run, so the geometry it hands on is too, by construction rather than by tuning: on all twelve crystals the rule moves, the post-refined distance and the pass-2 validation frame count match the fixed-radius values digit for digit. On the crystal that regressed, the indexing rate goes 0.7900 to 1.0000, observations up 30.1%, multiplicity 5.22 to 6.68, and low- resolution R_meas 13.1% to 10.7% against XDS's 10.8%. Against the right reference - the rule off, same lattice, same geometry, matched indexing rate - the widened radius on that crystal is better in 10 of 10 shells on R_meas and 10 of 10 on CC1/2. It looked worse against the shipped arm only because that arm merges a fifth fewer frames, and dropping the hardest part of a sweep flatters a pooled statistic. Taking the position centroid over a fixed aperture regardless of r1 was tried and does not work: the intensities, the sigmas and the surviving reflection set carry the perturbation too, and the geometry moved further from the baseline, not closer. Battery: space group identical on all 38 and 35 agreeing with the reference in every arm, merged intensities byte-identical on the 26 crystals the rule does not move, over a baseline reproducible to zero. No runtime cost. The density guard moves with it and now reads the final pass's own count. Exercising its re-run branch with a lowered bound turned up two reporting bugs, fixed here. Not fixed, and worth its own task: that crystal is still one hundredth of a pixel from losing a fifth of its data, because the indexer computes the unconstrained cell for exactly this comparison but only scores it once the constrained one has already failed outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P	2026-08-26 01:58:43 +02:00
leonarski_fandClaude Opus 5	48008e1447	rugnux: set the integration radius from the crystal's own spot width ... On rotation data the signal radius is now r1 = clamp(round(2*r80), 4, 6), where r80 is the 80% encircled-flux radius of the crystal's own spots. The background ring keeps its area (r3 = sqrt(r2^2 + 133)), so r1 = 4 is the shipped default bit for bit and 26 of the 38 battery crystals come out byte-identical. The width had to be measured somewhere new. rugnux already has one - shell_sigma2[].tan - but it is a second moment taken inside the r1 disk it would be setting, and it saturates at r1/2, so feeding it back measures the cap and not the crystal. SpotWidth instead measures encircled flux over an aperture fixed for the whole file (14 px, normalised at 8), in the pre-scan, from spots the finder already produces on the frames the beam-stop projection already reads. It touches no integrator output and runs before the first integration pass, so there is no loop, it costs no extra frame reads, and both passes - including the space-group search, which runs in pass 1 - see the same radius. A default run pays a median 1.9 s. k = 2 is not fitted. For a Gaussian r80 = 1.794 sigma, so r1 = 2*r80 is 3.59 sigma, where the truncated second moment recovers 0.990 of sigma^2. The new test checks the estimator returns 1.794 sigma on a known Gaussian. Battery, 38 crystals, both arms run twice: the space group is identical on all 38 and 35 agree with the reference in both arms. Per shell on the 12 crystals the rule moves, 6 win and 4 tie, with mean per-shell <I/sigma> up 30.6, 24.6, 15.8, 9.1, 8.0 and 5.1 per cent and R_meas down as much as 23.8. Runtime is neutral - 19m13s against 22m00s warm. One crystal is a real cost and is named in docs/RUGNUX.md with its workaround: an I222 case that is simultaneously the widest-spot and among the highest-mosaicity in the set loses 28.5% of its observations at unchanged completeness, because at r1 = 6 its predicted reflection density leaves the background ring too few clean pixels. No cheap guard separates it - its predicted spacing is mid-table, larger than five crystals that survive r1 = 12 - and the guard that would, on the measured drop rate out of pass 1, needs a diagnostic channel out of both integration engines and is not yet validated. This depends on 3ea120677: at the previous twin-law bound of 1.70 the wider radius costs one crystal its 422, refused on an H ratio of 1.73 even though every operator correlation confirms the point group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P	2026-08-25 22:45:02 +02:00
leonarski_fandClaude Opus 5	db908c6ec1	_process.h5: record the change of basis, and the space group ... A _process.h5 was internally inconsistent whenever the space group re-seated the lattice. The per-image reflections and lattices go to file as each image is processed, but the conventional setting is only chosen afterwards, so the file kept pre-reindex indices beside a post-reindex cell. Measured on an affected dataset, max|d_file - d(hkl, stored cell)| was 18.6 A. --mode scale then compared each frame's lattice against the stored cell, found none of 1800 acceptable, rejected every observation and died in the merge with "resolution calculation failed". /entry/MX/reindexMatrix now carries M with hkl_cell = M . hkl_written, and the reader applies it, so everything it hands out is in the setting of /entry/sample/unit_cell. Absent means the identity, so a file written before this reads exactly as before. On the affected dataset M comes out [[1,1,0],[0,1,1],[1,0,1]], det 2 - the primitive-to-body-centred basis its volume ratio implied - and the same measure falls from 18.6 A to 1.8e-5. Writing the reflections in the final setting instead was rejected: the per-image writer is shared with the broker, which streams and cannot buffer a run; and h,k,l, predicted_x/y and the per-image lattice are one consistent statement about one image, which retro-editing the indices would silently break. Two things turned up while fixing it. There are three re-seat sites, not one - the space-group search's own centred-lattice test re-seats too, and logged nothing - so the matrix composes over all of them. And the space group itself was never written: it was set only on the arm that searches, while a two-pass rotation run reuses pass 1's group and takes the other arm, so the canonical file carried a cell but no group and --mode scale merged in P1. --mode scale now reproduces --mode mx on the affected dataset: same space group and cell, 39329 unique reflections both, 100% of reflections common, CC 0.99985, sum|dI|/sum|I| = 0.0081. The residual is three reflections in 11.09 M crossing an ice-band edge, because mx carries the integrator's d and scale recomputes it from the cell. An unaffected dataset is byte-identical in .mtz, .hkl and .cif. An older affected file still cannot merge - M is not recoverable from it - but now says so in 51 s, naming both cells and the -S/-C override to use, instead of failing inside the merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CHMmeM1d489zvNFT7ZMN2P	2026-08-25 21:21:55 +02:00
leonarski_fandjungfrau	4dc2534dbf	v1.0.0.rc-162 (#72) ... **Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-25 08:21:39 +02:00
leonarski_f	538f3504d3	v1.0.0.rc-161 (#71) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-08-13 17:03:10 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00