v1.0.0-rc.166 #76

Open

leonarski_f wants to merge 183 commits from rc166 into main

pull from: rc166

merge into: :main

:main

:rc166

:gitea-pages

:rc165

:rc164

:rc163-unshare-doc-files

:2608-rc-162

:2608-performance

:adaptive-spot-finding

:stills-partiality-postrefine

:2607-res-prediction

:2607-rc-160

:french-wilson-model-maps

:2607-win-viewer

:2607-state-machine

:2606-rc.156-fixes

:viewer-windows

:2606-viewer-processing

:2606-tcp

:2606-pixel-refine

:2606-eiger-calib

:2606-q-spacing

:2606-eiger-module-fix

:2606-force-cpu-azint

:2606-xds-plugin

:2606-azint

:2506-rot-all

:2606-sgsearch

:2605-scaling

:2605-hdf5-vds

:2605-hdf5-enospc

:2605-hdf-fixes

:2605-b-factor

:2605-speed-up-preview-start

:2604-index-and-refine-fix

:2604-cuda-cleanup

:2604-azint-mapping-parallel

:2604-more-fixes

:2604-start-speed

:2604-async-start

:2605-sigma-correction

:2604-hdf5-errors

:2604-rc.136

:2604-rc.135

:2604-processing-options

:2603-single-file

:2604-detector-parallel-logic

:2603-pipeline-upgrades

:2603-multilattice

:2603-rc.131-2

:2603-rc.131

:2603-rc.130

:2603-rc.129

:2603-ep

:2602-scaling-d-limit

:2602-residual

:2601-completness

:2601-1.0.0-rc.124

:2601-1.0.0-rc.123

:2512-scaling-merging

:2512-1.0.0-rc.122

:2512-1.0.0-rc.121

:2511-1.0.0-rc.120

:2511-1.0.0-rc.119b

:2511-1.0.0-rc.119

:2511-1.0.0-rc.118

:2511-1.0.0-rc.117

:2511-1.0.0-rc.116

:2511-1.0.0-rc.115c

:2511-1.0.0-rc.115

:2511-1.0.0-rc.114b

:2511-1.0.0-rc.114

:2511-1.0.0-rc.113

:2511-1.0.0-rc.112

:2511-1.0.0-rc.111

:2511-1.0.0-rc.110

:2511-1.0.0-rc.109

:2511-1.0.0-rc.108

:2511-1.0.0-rc.107

:2511-1.0.0-rc.106

:2511-1.0.0-rc.105

:2511-1.0.0-rc.104

:2511-rc.103

:2511-1.0.0-rc.102

:2511-viewer-enh-2

:2511-viewer-enh

:2511-eiger-mask-3

:2511-eiger-mask

:2510-viewer-improvements

:2510-viewer-3D

:2510-fpga-clamp

:2510-rc.88

:2510-release

:2509-rc.82

:2509-gitea

Conversation 0 Commits 183 Files Changed 383

+20910 -3936

183 Commits

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	e421053b30	docs: the viewer's calibration fix is user-visible, so the release note says it ... It rides the existing viewer entry rather than taking a bullet of its own - the release note is deliberately about a dozen lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:44:46 +02:00
leonarski_fandClaude Opus 5	b8fa8e67d5	docs: the pages catch up with the last day on rc166 ... An audit of docs/ against the code at HEAD, concentrating on what landed after the previous documentation audit: the balanced half-set split and CCanom, the declared-range completeness denominator, the one rule for a quantity nobody measured, and the PONI the calibration now refuses to write. Five statements the code had made false: - The rotation merge's CCref column shows a dash, not "nan". - Two pages still promised unweighted 2Fo-Fc / Fo-Fc maps; they have been sigma_A-weighted since the rigid-body work, and one of the two sat six lines from a paragraph that said so. - Inserting the CCanom prose into the SigAno paragraph left the sentence about the PDBx items and the SigAno column with CCanom as its subject, so it read as a claim about a quantity that is in neither. CCanom is also rotation-only and is not in the mmCIF, which nothing said. - Two cross-references did not resolve - a heading that was renumbered, and a slug spelled without the hyphen MyST puts in "TCP/IP". Added where the behaviour is new and a user meets it: the report's contract for a quantity a run did not measure - no key, and a dash in the table, which is not the same claim as a measured zero; the half-set rule behind CC1/2, which is why a CUDA and a non-CUDA build now agree on it and on CCanom; and the second reason a calibration writes no .poni, a detector whose stored image is mirrored or quarter-turned, which the PONI format cannot state. The rc.166 changelog stays a release note. One entry is widened from the shell table to the rule it is a case of, and two are added for output that was wrong rather than merely undocumented: FITTED_RESOLUTION was the P1 cross-check's, and the unmerged MTZ carried the reference setting where the merged file carried the adopted one. Sphinx builds clean with -W on the pinned docs/requirements.txt, and every intra-doc anchor resolves against the generated HTML. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 17:44:20 +02:00
leonarski_fandClaude Opus 5	b2cbf2723b	calibration: write the JSON before anything the poni can refuse ... The orientation refusal added an hour ago throws out of WritePoniFile, which ran first inside the same try - so the run aborted before WriteCalibrationJson, and its own error text told the user "the .json beside it carries the full geometry" where there was no .json at all. The JSON is the file that can state everything: the verdict, the uncertainty, and a geometry a PONI has no fields for. It goes first, so a poni that cannot be written costs only the poni. Found by the documentation audit, which went to write down what the failure produces and found the two did not agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:44:20 +02:00
leonarski_fandClaude Opus 5	2456bbe7f6	viewer: calibrate a sample cell against its own space group, as the CLI does ... The powder-calibration widget built its ring list with CalculateXtalRings(cell), which assumes a primitive lattice. The overload taking a space group exists precisely because the fit pairs the innermost OBSERVED ring with the innermost LISTED one, so a list opening with a reflection the symmetry forbids scales the whole calibration by the ratio between that ring and the first real one. rugnux_cli was moved to the new overload when it landed; the viewer was not. So on any centred sample cell the viewer's calibration was scaled wrong - the exact failure the overload was added to prevent. The widget already takes the cell from the loaded dataset; it now takes the group with it and uses both. Where the dataset has no group the old call stands, which is the same P-lattice assumption as before and no worse. Found by a whole-branch review, which saw the two call sites side by side where a per-commit read of either could not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:38:40 +02:00
leonarski_fandClaude Opus 5	7bb74a9dc8	review fixes: two silent wrong outputs, and one rule for a quantity nobody measured ... From four whole-branch code reviews of rc166. Reviewing the net diff rather than the commits found what per-commit review structurally cannot: a later commit leaving an earlier one's claim standing, and two cases of a later commit quietly undoing an earlier one. THE FITTED RESOLUTION WAS THE P1 CROSS-CHECK'S. The block that merges in P1 to write <prefix>_P1.mtz saves and restores the error model around itself, because that merge is not the run's answer. A later commit taught the same function to report the CC1/2 resolution fit and did not extend the list, so every de-novo rotation run in a non-P1 group has been quoting FITTED_RESOLUTION - the number the report itself calls the one to quote - from a merge with n_ops times the unique reflections at a fraction of the multiplicity. AND IT ROUND-TRIPPED THE SPACE GROUP THROUGH ITS NUMBER. A number names only the reference setting, which stopped being enough when the search learned to adopt P 1 1 2(1) or I 1 2 1. Everything written after that block - the unmerged MTZ included - therefore carried the reference setting while the merged file carried the adopted one: two files describing one dataset in two different settings. It now carries the group. A PONI CANNOT STATE A MIRRORED OR QUARTER-TURNED DETECTOR, and the fits became orientation-aware on this branch while the writer did not. It wrote five numbers that silently described a different geometry from the one measured; it now refuses, and says the JSON beside it has the full one. A REFUSED FIT'S ERROR BARS COULD BE HANDED BACK AS AN ACCEPTED GEOMETRY'S. RingOptimizer::Run writes its uncertainty only where the solve is usable, and CalibrateFromSpots runs it twice - tilt free, then tilt pinned. A failed second fit kept the first's sigmas, valid flag and all. It is cleared on the way in. ONE RULE FOR NOT MEASURED. The report had four conventions for it and printed the same missing quantity two ways on adjacent lines: SIGANO as the literal "nan" and CC_ANOM by absence, for a Friedel-merged run that split no Bijvoet pair - which is the default. A quantity a run did not measure now writes no key, and the shell table's dash follows the same rule rather than a 0.0% that reads as a measured total failure. ANISOTROPY_D_MIN_BEST also stops printing nan when only its first principal direction is unmeasured. Four claims that a later commit made false are corrected where they stand: the merge header promising an order-independence the balancing rule gave up, the reference page arguing against CCanom 28 minutes before it shipped, the screw threshold whose "three dead reflections clear it" the evidence floor caps at 19.2 nats, and a shell comment calling equal width in 1/d^2 equal volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:27:56 +02:00
leonarski_fandClaude Opus 5	8661193712	merge: balanced half-sets by rank, and CCanom read off them ... Two statistics and the machinery they share. THE SPLIT. CC1/2 correlates two half-set means, so a reflection whose observations all land in one half has no second mean and contributes nothing. The half was a hash of the image index alone, which does that to 2^(1-n) of the reflections at multiplicity n: half of the doubly measured ones, a quarter of the triples. Measured on in-house rotation data that is 0% of a 26-fold redundant sweep but 6-30% of a 3-to-5-fold one, and 35-60% of its outer shell - the shell the automatic resolution cutoff is read from. An observation's half is now the PARITY OF ITS RANK among its reflection's observations, ordered by a key built from the raw Miller index and the peak frame. That is exactly balanced - floor(n/2) against ceil(n/2), the split cctbx's compute_cc_one_half uses and phenix.merging_statistics through it - and, because a rank is a property of the set rather than of the order it is walked in, it is the same on every path. A sequential "put it wherever the counts are more even" rule is balanced too but not that: the device holds the fulls in emit order and the host in frame order, and on three test crystals that alone moved CC_HALF between a CUDA and a JFJOCH_USE_CUDA=OFF build by up to 0.0077, with 4 to 12 shell rows differing. Both now agree bit for bit, on the whole scaling-and-merging section, while the radiation-damage B in the same report still differs between the builds - so the agreement is a property of the split, not of the two pipelines being identical. It is also stable across repeated runs and across -N 1, 4 and 12. Ranking is quadratic in a reflection's multiplicity - tens - so it is one multiplicity-weighted sweep of the observations, assigned once per pass and read by both the host loop and MergeAccumKernel. The kernel no longer decides anything, which is why the two cannot drift apart. CCANOM. SigAno was the only anomalous quality statistic reported, and it is a ratio against the error model: an optimistic sigma raises it, and it cannot separate real anomalous signal from an underestimated sigma. CCanom carries no sigma. For each acentric reflection the anomalous difference is formed twice, once per half, and the two are correlated over every pair where both hands split into two non-empty halves - per shell, and overall as one correlation rather than a mean of shells. The halves for it are balanced within each MATE, not within the reflection: a mate left entirely in one half loses the whole pair, and balancing per pair instead measured 0.223 where per mate gives 0.260. It hangs on the anomalous accumulation that already existed for the I(+)/I(-) export, which runs whether or not the merge is Friedel-averaged - so CCanom is reported on a default run too, and -A only changes the counting basis. The whole-mate sums SigAno and the reflection files read are the two halves added back together, so neither moves. Against external programs on the same observations: AIMLESS 0.264 and cctbx 0.266 where this gives 0.260, per shell within about a point. XDS's CORRECT.LP has a column named `Anomal Corr` which is NOT this quantity - two to three times larger in the low shells, with a total below every one of its own shells - so the report says so where it describes the key. On data with no anomalous signal and around two observations per mate the statistic is unstable and goes strongly negative; phenix reads -0.83/-0.45/-0.42 where this reads +0.12/-0.33/-0.41 on the same file, so that is the statistic, not this implementation, and it is reported as measured. REPORTING. A shell prints `-` where it could not measure a quantity instead of `nan` - which every run printed for CCref, and a multiplicity-1 shell printed for most of the row - and CC_ANOM is omitted from the report rather than written as a placeholder when no pair could be split in both hands. A dash and an absent key are the same statement; a measured value always prints, including a negative one. Reported CC1/2 moves, so baselines keyed on CC_HALF need regenerating, and the automatic resolution cutoff reads the same statistic and can move a shell edge with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 17:07:32 +02:00
leonarski_fandClaude Opus 5	0c3e1462ef	merge: the completeness denominator is the declared range, not the surviving one ... CalcPossibleReflections was handed d_min/d_max derived from the reflections that came out of the merge, so a loss at either extreme took the numerator and the denominator with it. At the high end that is right: d_min is the finest d reached anywhere and the denominator is the full sphere down to it, so anisotropic loss shows. At the low end it was a tautology - d_max was the coarsest reflection that happened to survive, so anything the beam-stop shadow mask (on by default), a detector mask or the low-resolution limit itself removed left the denominator along with the data and could not be reported as missing. Both statistics paths now bin, and count, between the DECLARED low-resolution limit and the finest d reached: MergeOnTheFly::MergeStats (stills) and RotationScaleMerge::MergeAndStats (rotation). The grid and the denominator keep sharing their bounds, so no possible reflection falls outside a shell. An undeclared low limit is the whole sphere - 1/d^2 down to 0 - spelled as an infinite d_max, which ResolutionShells already handles and which gemmi's for_all_reflections special-cases; the change therefore reads correctly whether or not the 50 A default stays. The innermost shell keeps a finite d_max label, falling back to the coarsest reflection measured when the bound is infinite. This makes the shell boundaries the ones the integration document already claims: XDS lays its nine 1/d^2 bins between INCLUDE_RESOLUTION_RANGE's two values, not between the extremes of the surviving data, and counts POSSIBLE against the declared low limit - which is why its innermost shell reports the beam stop's loss. Verified against a CORRECT.LP: all nine boundaries reproduce to the printed precision from the declared 50 A, and not from the coarsest observed reflection. Measured on stored merges of seven rotation datasets, small-molecule and protein, re-scaled with --mode scale: the overall denominator moves by 0 to 2 reflections out of 70,000-100,000, because on every one of them the coarsest reflection the declared limit allows was itself measured - the corpus has no dataset whose stop eats a whole low-resolution class. What does move is the shell grid: the innermost boundary shifts by 0.1-0.4% in d (e.g. 7.21 -> 7.22 A), which changes the innermost shell's counts by up to a few per cent and its R_meas by around 0.1 percentage points. Stored battery baselines for rmeas_lo must therefore be regenerated, not compared across this commit. Two decisions read merged completeness (the two-pass wrong-cell guard, which only fires above 100.5% and only under -S); a larger denominator can only lower the figure, so the guard can fire less often, never more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 16:39:40 +02:00
leonarski_fandClaude Opus 5	a0db11193a	docs: the low-resolution limit guards a depressed background, not the beam stop ... The comment, the usage text and the advanced docs all said reflections coarser than 50 A are "behind or beside the beam stop", which invites the conclusion that beam-stop masking makes the limit redundant. Measured, it does not. On raw images, the azimuthal background in the pixels the shadow mask leaves OPEN is 30-44% of the field value out to ~34 px and recovers to 95% of it only at d ~ 52 A: the stop suppresses air scatter well beyond the shadow it casts. A background ring inside that zone over-estimates the background, so the intensity comes out negative - 74% of the reflections coarser than 50 A on that geometry, against 2% in the shell just inside it, and they sit a median 15 px clear of the mask (reflections touching the mask are only 33% negative). The mask covers 47-79% of the area coarser than 50 A across three datasets, so the part that matters is exactly the part it leaves open. The zone is set by the stop's angular size, so it tracks wavelength rather than detector distance, which is what makes a fixed d limit reasonable. Freeing the limit on the most affected large-cell dataset in the corpus adds 1329 observations (+0.11%), moves I/sigma 7.01 -> 7.00 and completeness not at all, and stretches the declared low-resolution range from 46 A to 102 A on the strength of mostly-negative data. The advanced docs additionally told large-cell users to change the setting, implying recoverable data; corrected. "The value XDS configurations use" is also sharpened: every dataset here whose XDS.INP leaves INCLUDE_RESOLUTION_RANGE unset reports 50.000 in CORRECT.LP, so 50 A is XDS's own program default rather than a habit of our inputs. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 16:26:20 +02:00
leonarski_fandClaude Opus 5	7f72c2f622	rugnux: two usage-message lines caught up with the behaviour ... --fft-min-unit-cell claimed a crystal below the 10 A floor cannot be indexed at all; since the default-on short-axis second hypothesis it can, so the line says so. --model promised 2Fo-Fc/Fo-Fc maps; the maps have been sigma_A-weighted 2mFo-DFc/mFo-DFc since the rigid-body work, and the placed model is written beside them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 15:48:14 +02:00
leonarski_fandClaude Opus 5	b971f99b53	docs: audit of the rugnux and analysis pages against the code ... The one outright falsehood: the R-free convention paragraph in RUGNUX_INTEGRATION.md still described the pre-flip FreeR_flag numbering (0 = work, phenix/CNS) and told REFMAC5 users to pass FREE 1 - on a current file that keyword designates the 95% working set as free and REFMAC stops. The file has carried 0 = free (CCP4) since the flip; the paragraph now says so, the keyword is gone from the worked script, and a note keeps the old error message findable for files written before it. A contradiction within RUGNUX_ADVANCED.md: the --mode scale section said the mode never reindexes the reflections it writes, while the indexing- ambiguity section (correctly) said --mode scale --model does resolve a rotation ambiguity. The code adopts the model's frame before writing, so the former now agrees with the latter, and says what genuinely cannot be repaired: a stills _process.h5 integrated without a reference. Half-updated model-validation prose: the pages that predate the hypothesis gate still described --model deciding the enantiomorph and the indexing unconditionally. Every such statement (quick start, tutorial, the ambiguity table and bullets, the validation section) now carries the gate: the model decides nothing unless it beats the null of its own random placements, and the indexing choice must also beat the null's margin. The map names now say sigma_A-weighted 2mFo-DFc/mFo-DFc. Missing files: the tutorial's output-file list did not mention the --model outputs at all; it now lists the maps, the map-coefficient MTZ, the anomalous map and the placed model in both formats, and points out that <prefix>.cif is reflections while <prefix>_model.cif is coordinates. _model.pdb is added beside _model.cif everywhere the placed model is described, with the PanDDA/dimple reason it exists. Undocumented indexing behaviour: the axis-harmonic spot-count arbiter and the default-on short-axis second hypothesis existed only as changelog lines; CPU_DATA_ANALYSIS_INDEXING section 6 now describes both, and the --fft-min-unit-cell texts no longer claim a crystal below the 10 A floor cannot be indexed at all. Small corrections in RUGNUX_REPORT.md: sweep-quality warnings live in section 11, not 9; JFJOCH_DATASET_SETTINGS carries the poni_rot*_rad angles too whenever any is non-zero. The tutorial's post-refine sentence now states the real commit bounds (distance under 1%, beam within 15 px of the header or the run's own measured centre, whichever is nearer) instead of "restrained toward the header", which f98442077 made stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 15:48:14 +02:00
leonarski_fandClaude Opus 5	79a0b1358f	docs: keep the viewer's merge-plot axis fix in the changelog ... Collapsing the section dropped it as minor. It is small but it is the kind of thing a user notices and then wonders about - a CC1/2 curve drawn over the full height with tick labels covering only the top tenth of it - so someone who saw that should be able to find out it is fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:38:37 +02:00
leonarski_fandClaude Opus 5	ea07d7c9dd	docs: the rc.166 changelog collapsed into a release note ... The section had grown one line per commit, 38 entries, a development log. It is now twelve statements a user of rugnux or the broker can act on, each folding the lines that shared a user-visible consequence: - the model work is two entries (the hypothesis gate; the outputs, now including <prefix>_model.pdb) plus its report keys. - reading foreign data - miniCBF, Eiger 1.x, third-party NXmx, rugnux and viewer alike - is one entry. - lattice, indexing-on-the-true-cell, point group, setting, absences and the -S refusal are one entry; beam centre, tilt and the 2theta arm are one; the unmerged-MTZ lines are one; the reflection-file conventions (FreeR direction, HKL_base, _refln.status) are one. - the six report lines and both REPORT_VERSION bumps are one entry that says what the report carries and states the version once. - the calibration mode's .json, its refusal to return a non-measurement (non-zero exit) and --no-refine-tilt are one entry. - the recorded-metadata lines (direct_beam, incident_beam_size, peakCountUnfiltered, smargon.chi_deg) are one; the two documentation lines are one. Kept standalone, because holders of existing files need it: the snake-grid mirroring fix. Kept explicit inside their entries: the FreeR_flag direction and the DETERMINED_FROM_MODEL -> ASSUMED_FROM_MODEL rename. Dropped entirely: the goniometer-axis direction-vs-length line (internal consistency of the first pass) and the viewer merge-plot label fix; the null-replicate cost work, tests, regenerated python-client docs and review-fix internals never earned lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 15:38:27 +02:00
leonarski_fandClaude Opus 5	232fd649ae	model validation: write the placed model as PDB as well as mmCIF ... The file exists for fragment screening, and the tools it is meant to feed take a PDB: PanDDA's per-dataset input is <name>.pdb beside <name>.mtz, and dimple produces that same pair. Writing only mmCIF left the one consumer the file was for unable to read it. Same coordinates, same frame, both formats. gemmi::write_pdb was already linked - to_mmcif.cpp needs use_hetatm out of to_pdb.cpp - so this costs a call and no dependency. A PDB cannot express every cell, so it is written where it can be and skipped with a line saying so where it cannot; the mmCIF is unconditional. Measured on one rotation run: <prefix>_model.pdb and <prefix>_model.cif carry the same 1156 atoms, and both read back with the space group and cell of the .mtz beside them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:32:34 +02:00
leonarski_fandClaude Opus 5	847e44d718	indexing: the short-axis pass stays out of a given cell, and four stale comments ... From the code review of the indexing work. The short-axis pass took its settings from experiment_, which never carries the max bound RaiseFFTBoundForKnownCell raises for a given cell - that runs on a local copy. So with -C naming a cell longer than the default search, the short pool could not represent the cell that was asked for and could only return a sub-multiple of it; a sub-multiple indexes every frame its parent does, which is the premise of this whole comparison, so it would tie and be adopted over the user's own cell. It now skips a run with a known cell entirely, which is also where it had least to offer: the same call has already lowered the FLOOR against that cell for the standard pass, so the two schemes were returning the same lattice and paying two extra FFTs for it. Four comments asserted rules the code no longer follows. The one directly above the decision still said the larger cell is taken as spurious "regardless of scheme order" - the opposite of what the arbiter does - and a dead assignment below it stated the same thing a second time. The added-class enumeration is described as exhaustive; it covers the index-n sublattices with a CYCLIC quotient, which is all of them at n = 2 and n = 3 but not at n = 4, where a supercell doubling two axes has a (Z/2)^2 quotient and reads at the chance value instead. That is a report and decides nothing, but it is quoted in the argument for bounding n at 4, so the bound rests on thinner evidence at n = 4 than it appears to. And std::rint where the identical loop in IndexAndRefine uses it and says why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:22:19 +02:00
leonarski_fandClaude Opus 5	ab4a89e0a9	review fixes: the calibration accept test, a NaN warning, and two grid defects ... From the code review of the writer, API and geometry work. The powder calibration's re-bin accept test was one-directional. The best-of fold twelve lines above it is symmetric on purpose - a fit that is not a measurement never beats one that is, whatever its residual - but the final test applied the residual clauses regardless, so a CONVERGED re-fit could lose to a non-converged first pass on rms alone. That is exactly the motivating case: a first pass that declined the tilt and pinned the header's. It now matches the fold. The anisotropy warning seeded max_element and min_element on an array that can hold NaN, and every comparison against NaN is false, so both returned the same entry and the sentence named one direction twice with nan for both limits. The non-finite entries are skipped, and where fewer than two directions have a limit the warning states the anisotropy without naming one. The grid-scan plot added a SIGNED half-step to a grid laid out in |step|, so with a negative step every cell centre landed one whole step outside the extent - the same family as the snake-parity fix, in the same configurations. And one guaranteed no-op: /entry/MX/imageScaleFactor was written twice. Changelog: the short-axis pass had no entry at all, and the mmCIF one carried its rationale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:20:49 +02:00
leonarski_fandClaude Opus 5	b2c0a5ea48	model validation: review fixes - the run's output, the origin gauge, and two claims ... From two independent code reviews of the model-validation work. Nothing here changes a verdict: the acceptance set still reads +17.69 / +1.70 / -0.37 sigma and the rejected runs still write files byte-identical to a run with no model. THE RUN'S OUTPUT. Model validation runs BEFORE the reflection files are written, and two paths through it could throw: the null's replicates (rotated models fed to a scaling path that fails outright on data it cannot pair up - the adversarial input for it), and the mmCIF coordinate writer. Either would have taken the .mtz, .cif, .hkl and _unmerged.mtz with it, after the merge had already been paid for. A null that cannot be built is a question that could not be put, which is the NOT_TESTED state this design already has; a coordinate file that cannot be written is a lost convenience. Both now degrade instead of aborting. THE ORIGIN GAUGE. Translating the whole cell content along a free-origin direction - all three in P1, the unique axis in a polar group - leaves every |F| exactly unchanged. The code said the LM damping and the R-free gate made that harmless between them. Neither does: the gate is a function of |F| and is blind to exactly this, and the gauge column of the Jacobian is not zero but noise divided by the difference step. It is now projected out after every zone, against the group's own common fixed subspace. P2_1 alone is a large share of deposited structures, and the reported shift was partly fiction in every one of them. TWO CLAIMS THAT WERE FALSE. The report told the user R-work carries the decision "because nothing was refined against it", six lines from where six placement parameters are refined against it; the real argument is that the null is placed the same way, so the optimism is common-mode and cancels. And the constant's own comment quoted a +4 vs +1 sigma gap where the measurement is +17.7 vs +1.7. Also: the map file's phase columns are back to [0, 360), the convention they carried before sigma_A weighting; with no data space group the map file follows the reflections into P1 rather than taking the model's group; the sigma is floored against a near-zero null spread rather than only an exactly-zero one; the model is restored whether or not the solver reported a usable answer; and the zone list is the ladder walked rather than the ladder planned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:18:55 +02:00
leonarski_fandClaude Opus 5	4f84b86276	model: say what NOT_TESTED means, which is not that the groups agree ... The log line and the report both said a not-tested model "is already in the space group the data were merged in". That is one way to claim nothing, not the only one, and it is false for the rest: the gate is "no enantiomorph candidate and the indexing winner is identity", and a model whose group differs from the data's without being its enantiomorphic partner sets no candidate either. Measured: data merged in P 43, model in P 43 21 2 - different groups, no enantiomorphic relation between them - and the run printed that the model was already in the data's group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:00:24 +02:00
leonarski_fandClaude Opus 5	0903a8eee3	maps: the map file states the group the reflections are in, and the wavelength ... Two things wrong with <prefix>_maps.mtz beside <prefix>.mtz. It took the space group from the MODEL, unconditionally. That was invisible while a model's hand was always adopted; now that a model decides nothing until it beats its own null, a rejected model that asserts the other enantiomorph leaves the two files disagreeing - the reflections in the group the data were merged in, the map coefficients labelled with the model's. An enantiomorphic pair indexes identically, so the coefficients are the same numbers either way and only the label moves, but the two groups have different screw translations: a reader that expands symmetry out of the map file was doing it in the wrong group. It now takes the group the same way AdoptModelFrame does a few lines below - the model's only where the hand was adopted, the data's otherwise. And it carried no wavelength at all, DWAVEL 0.00000 on both datasets, because nothing in ValidateAgainstModel knew it; it is now passed in from the experiment the two callers already hold. Measured on one rotation data set: with the model accepted both files read P 43 21 2, with an unrelated model rejected both read P 41 21 2, and both carry the collection wavelength. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:54:54 +02:00
leonarski_fandClaude Opus 5	63a2f6744a	docs: say what the MTZ base-dataset fix actually changed, and what it did not ... The page said the fix makes mtzinfo and mtzdmp report the collection wavelength, which implies mtzdmp was wrong before. It was not. Verified across 226 files from two battery arms, one built before the fix and one after: every pre-fix merged MTZ reports 1.54187 under mtzinfo and the true wavelength under mtzdmp, truncate, ctruncate, gemmi, iotbx and phenix.xtriage. No program was found whose output or behaviour differs between the two files - cad silently repairs the layout on the way through. So the fix buys a conformant file and a correct mtzinfo line, not a rescued phasing run, and the f'/f'' consequence is the risk it removes rather than a measured effect. Also records that <prefix>_unmerged.mtz still reads 1.54187 under mtzinfo and is not a regression: its columns sit on HKL_base deliberately, which is what POINTLESS expects, and the per-batch wavelength AIMLESS and POINTLESS actually read is correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:49:10 +02:00
leonarski_fandClaude Opus 5	46c50ac744	tests: the merged MTZ's base dataset, which nothing covered ... WriteReflections had no test at all, and the defect it hid was invisible to two of the three libraries that read an MTZ: with the data dataset written at id 0 - the id reserved for HKL_base - gemmi and iotbx still returned the real wavelength, and only CCP4's mtzinfo fell back to the 1.54187 A Cu K-alpha default. iotbx reading such a file reports two datasets both numbered 0. The test writes a merged MTZ through the real writer at a wavelength well away from that default, reads the file back, and asserts the layout the report depends on: HKL_base at id 0, the data dataset at id 1 carrying the wavelength, and every data column owned by the data dataset rather than the base - that ownership is what the reported wavelength is read off. Cell and space group are checked with it, since they travel in the same header. Verified against the pre-fix writer (base absent, H K L added on the data dataset): it fails on the dataset count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:48:48 +02:00
leonarski_fandClaude Opus 5	28a58e94b9	model validation: write the model as it was placed ... `--model` re-fractionalizes the model into the data cell and then places it as one rigid body, but the placed coordinates never reached disk. On a lysozyme sweep against a non-isomorphous deposited model the move is 3.058 deg and 1.035 A, so a user overlaying their input model on rugnux's maps was out by exactly that, and no file on disk corresponded to the maps at all. `<prefix>_model.cif` is that file: the input's chains, residues, ligands, waters, B-factors, occupancies and anisotropic Us, at the coordinates the maps were computed from. `<prefix>.cif` is already the merged reflections, hence the suffix. The cell and space group come from the same two values WriteReflections is given - the unit cell and DiffractionExperiment::GetSpaceGroupOrP1() after AdoptModelFrame has settled the enantiomorph - so the coordinate file and the .mtz beside it always agree. Taking them from the input model would not: with data merged in P4(1)2(1)2 and a P4(3)2(1)2 model, the written reflections take the model's group, which is neither the data's original label nor, when the model is rejected, the model's own. Written whenever the maps are, not only where the rigid-body step was committed. The model is re-fractionalized and may be relabelled whatever the placement decided, so an unmoved model is still not the input file; and a model the null rejected is scored, placed and mapped like any other - the negative result, and the case where the density is most worth looking at. Nothing in the tree could write coordinates: gemmi_gph declared to_mmcif.hpp but src/to_mmcif.cpp had been trimmed from the vendored subset. Both it and to_pdb.cpp (to_mmcif.cpp calls its use_hetatm) are vendored from the same gemmi 0.7.5 the rest of gemmi_gph comes from, unmodified, and every header they include was already there. Same package, same MPL-2.0, same LICENSE.txt already collected into licenses/gemmi.txt and already listed against `gemmi_gph/` in THIRD_PARTY_NOTICES.md, so no new row and no new licence text. Verified end to end: read back with gemmi the file differs from the re-fractionalized input by exactly the reported 3.058 deg / 1.035 A with 0.0000 A rms about that rigid move, and REFMAC5 at zero cycles against rugnux's own .mtz starts at R-free 0.3667 where rugnux reports 0.3826 - against 0.5832 for the unplaced input model, where rugnux reports 0.5911 before the placement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:47:38 +02:00
leonarski_fandClaude Opus 5	8a0bbae3c4	model: nine null replicates, now that they cost the slowest and not the sum ... The verdict is (real - mean)/sd of the null sample, so its reliability is set by how well the SPREAD is pinned, not the mean. The relative error on an sd from n draws is 1/sqrt(2(n-1)), and a sample that happens to come out narrow is exactly what turns a model that does not fit into one that appears to. Measured on the case that sits nearest the gate - an unrelated protein, 1.7-1.8 sigma against a threshold of 3 - the chance of it reading over the gate on a different seed is 31% at n=3, 17% at n=5 and 6% at n=9. Nine is affordable only because the replicates now run concurrently: the null costs the slowest of them rather than their sum, and the slowest of nine is barely above the slowest of five. Measured end to end on the accepted case, 5.98 s against 6.46 s for five - inside the run-to-run scatter, so the four extra replicates are free. Verdicts unchanged, on more evidence: correct model ACCEPTED at +17.69 sigma (+14.96 at n=5), an unrelated protein REJECTED at +1.70, the same model rotated 90 degrees REJECTED at -0.37. Both rejected runs still write .mtz, .cif, .hkl and _unmerged.mtz byte-identical to a run with no model at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:47:10 +02:00
leonarski_fandClaude Opus 5	53a8c428c3	model: run the null's replicates at once ... The replicates are independent by construction - the same model under a different random rotation, scored the same way, nothing flowing between them - so each takes a copy of the model and of its structure factors and they run concurrently on the run's own -N budget. The real model is not moved by any of them, which also drops the save/restore/recompute the serial loop needed to put it back. Measured on an idle machine, decision-pending --mode scale --model: the null 10.0 s -> 2.5 s (median of 5), the whole run 13.9 s -> 6.5 s. The speedup saturates at 4.0x, not 5x, because the replicates cost 1.58-2.31 s each (how many placement evaluations an orientation needs) and the null now costs the slowest one; sum/max on the serial run is 4.13x, so 97% of what is there to get. More threads than replicates buy nothing. The rotations are drawn up front, in order, from the same fixed seed, so replicate i gets the same orientation whatever order the threads run in. A verdict that depended on the interleaving would not be a measurement. Verified: every reflection file, map, map-MTZ and report is byte-identical to the serial build's, on the accepted, both rejected and both no-decision cases, and identical again with -N 1. Thread safety was read out of the vendored gemmi source, not assumed. Each replicate owns its DensityCalculator, SolventMasker, Scaling and Structure; their statics are the const IT92 tables (constant-initialised, never written on these paths), pocketfft is built with POCKETFFT_CACHE_SIZE 0 and POCKETFFT_NO_MULTITHREADING so it holds no plan store, Scaling's Levenberg-Marquardt is stack-local per call, and nothing on these paths writes into the Structure it was handed. Ceres defaults num_threads to 1, so the rigid-body solve adds no threads of its own. Peak RSS is unchanged - the merge, not the null, is the high-water mark. Five concurrent replicates add 83 MB over the serial null on a 78x78x37 A cell at 1.56 A, ~17 MB each, which is the two grids at d_min/(2*rate). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:41:28 +02:00
leonarski_fandClaude Opus 5	b9443b4daf	indexing: run the short-axis hypothesis by default ... The pass was landed behind RUGNUX_SHORT_AXIS_PASS so that a battery arm could measure it before it ran on every dataset. This turns it on and keeps the variable as an escape hatch, spelled the other way round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 14:32:45 +02:00
leonarski_fandClaude Opus 5	0eb9fb8a8b	docs: the phenix label line, and why the MTZ carries no DANO ... Two measurements, neither of which changes what we write. phenix's "Multiple equally suitable arrays" is a tie between the two INTENSITY arrays, IMEAN and I(+)/I(-); iotbx scores F and F(+)/F(-) below them, so writing amplitudes as well is not what causes it. ctruncate's own output ties in the same place, so this is what phenix does with a CCP4 merged file rather than something rugnux does to a user. Column order, dataset and project names, and dropping the amplitudes all leave the tie exactly where it was; only removing one of the two intensity arrays clears it, and removing the Bijvoet columns would take the SHELX route with it. So the file stays as it is and the page now carries the label line per program, both formats, including the quoting the anomalous one needs. A program that asks iotbx for anomalous data by preference - hyss, find_peaks_holes, molprobity, the autosol import - needs nothing at all, which is now said as well. The page also said the MTZ form of that message names no choices. It names both, exactly as the mmCIF form does; corrected. DANO/SIGDANO stay out. Against a ctruncate file, DANO is F(+)-F(-) and SIGDANO is the quadrature sum of the two sigmas, bit-identical on every reflection, so the pair is a restatement of columns we already write - and the quadrature sum is the convention whether or not the mates share a scale model. Every consumer in the documented routes takes the Bijvoet columns directly, CCP4's own bp3 and afro ask for them in preference to F/DANO, and adding the pair costs 12.5% of the merged file while changing nothing phenix or Phaser sees. fft's anomalous Fourier is the one caller with no other spelling; the ctruncate command that makes it is now on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:12:45 +02:00
leonarski_fandClaude Opus 5	4af23e9b27	model: build the null only where the model claims something ... The null costs about 12 s - five replicates, each a full fit and a rigid-body placement - and it gates exactly two decisions: the enantiomorph label and the merohedral indexing. A model that claims neither, already in the group the data were merged in on a crystal with no indexing ambiguity, has nothing for the null to arbitrate, and paying for it there is 12 s spent gating a decision nobody is making. That is the isomorphous fragment-screening run - hundreds of crystals of one form against one apo model, a map wanted seconds after the last image - and it is the case that has to be fast. So the null runs only when a decision is pending: an enantiomorph candidate exists, or the indexing probe returned a winner that is not the identity. The probe is a handful of scaling fits and had to run first anyway, so the question is answered before the expensive part starts. Measured on a rotation dataset, --mode scale --model, interleaved on a loaded machine: the isomorphous case is 3.6-4.7 s before this whole line of work and 3.8-4.7 s with the gate, against 16.5 s while the null ran unconditionally. The case that still pays is unchanged at 17 s, and the three wrong-model cases give bit-identical verdicts and bit-identical files to before. The rigid-body refinement of the replicates stays: 3 of 5 commit a placement, so without it the null would score a weaker procedure than the one it judges. MODEL_FIT gains a third value, NOT_TESTED, alongside ACCEPTED and REJECTED - the question was never put, which is not the data answering it badly - and the MODEL_FIT_NULL_* and MODEL_FIT_SIGMA keys are absent with it rather than reporting a null that was never built. REPORT_VERSION stays 6: MODEL_FIT is itself new in this release, so its vocabulary has never shipped and a consumer cannot be relying on the two-value form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 14:06:29 +02:00
leonarski_f	c999883eb1	model: --model is a hypothesis, and it decides nothing until it fits ... A model supplied with --model rewrote the space group of every reflection written out on the strength of its file having parsed. Measured on a rotation dataset merged in P4(1)2(1)2: an unrelated protein and the correct model rigidly rotated 90 degrees each produced a .mtz, .cif and _unmerged.mtz byte for byte identical to what the crystal's own model produced - relabelled P4(3)2(1)2 - at R-free 0.601 and 0.674 against the correct model's 0.591, with no warning. The adoption was pure space-group-number arithmetic and ran before the model had been fitted at all. A model changes exactly two things on the rotation path, and both rewrite the data: the enantiomorph label and the merohedral indexing. Both now wait for the fit. Everything else a model produces - R-factors, maps, the rigid-body placement - is a statement about the MODEL, cannot corrupt a reflection, and is computed and reported either way. The gate is not a threshold on R, because no threshold works: the classical acentric random value is 0.586 at unit scale but 0.550 at the R-minimising scale, observed nulls land at 0.599-0.615, and the value moves with the model's atom count and B-factors as much as with the data. Instead the same model is re-oriented at random about its own centroid five times and run through the identical path - same scaling, same rigid-body placement, same R - and the real fit is asked how far above that distribution it sits. R-work carries the decision: nothing is refined against the working set here, and it has 12615 reflections to R-free's 709. Measured on the case above: the crystal's own model +15.0 sigma, the unrelated protein +1.8, the 90-degree rotation -1.0, and the two rejected runs now write files byte-identical to a run with no model. The nulls are rigid-body refined like the real fit, or the comparison would be between a placed model and unplaced nulls. That is what the null costs: about 12 s for the five replicates, on a --mode scale run that merges in 2 s. The merohedral margin gets the same treatment - a random placement also picks a winner, and measured, by a comparable lead - and the candidate operators are now enumerated from the DATA's space group. Taking them from the model's enumerated zero operators wherever the two groups differ, which is exactly the case the probe exists for: a model in P4(3)2(1)2 against data merged in P4(3) probed nothing at all, and now probes the twin law. The anomalous difference map is the only measurement here sensitive to the hand - inverting the model through the origin moves R-work by less than 1e-4, since |F(h)| of the inverted structure is |F(-h)| - so where it says the hands disagree it vetoes the adoption outright, fit or no fit. Report: MODEL_FIT, MODEL_FIT_SIGMA and the null beside it, MODEL_DECISIONS_TAKEN, the indexing margin against its null, and MODEL_VALIDATION= PERFORMED as the counterpart of the failure line. SPACE_GROUP_ENANTIOMORPH= DETERMINED_FROM_MODEL becomes ASSUMED_FROM_MODEL and is written only where the model was accepted: nothing here measured the hand, the model asserted it. That is a reason code changing name and meaning, so REPORT_VERSION is 6. Stills are untouched: the per-image indexing hand a model can set at integration time is not reachable on the rotation path and is not gated here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:34:52 +02:00
leonarski_fandClaude Opus 5	d5cf3a6e58	docs: worked Phaser and SHELX routes, and the MTZ base dataset they need ... Both programs were run on merged rugnux output and the commands are the ones that worked, error messages included. Phaser takes myrun.mtz with no LABIN, picking IMEAN/SIGIMEAN and the cell, group and resolution out of the file; MODE MR_AUTO already searches both hands of an enantiomorphic pair and returned the hand opposite the one in the header on a tetragonal test. SGALTERNATIVE SELECT ALL covers the screw variants the report lists as indistinguishable, MODE CCA prints the list without searching, and a wrong point group has to be re-merged instead. mmCIF is refused by 2.8.3 in both the CCP4 and the phenix build; gemmi cif2mtz gives the same solution. SHELXC reads only myrun.hkl - it refuses an MTZ and exits 0 while doing it - and needs CELL and SPAG repeated, since HKLF 4 carries no metadata. A default rotation merge already writes the Bijvoet split, so no flag is needed and --no-export-unmerged hides nothing. On a 5 keV cubic sweep SHELXD separated a space-group pair the merged intensities could not (CFOM 69.4 against 52.0) and SHELXE solved it, 42.9 % against 15.3 % autotrace CC between the two hands; a tetragonal sweep at lower completeness and multiplicity did not separate at all, and that is recorded too. Verifying it exposed one defect. The merged MTZ was written with its data dataset at id 0, the id MTZ reserves for HKL_base, so mtzlib dropped the wavelength and mtzinfo reported the 1.54187 A default - the wrong edge for anything taking f'/f'' from the file. Writing HKL_base first, as the unmerged writer already does, puts the real wavelength back; Phaser's solution is unchanged either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:07:03 +02:00
leonarski_fandClaude Opus 5	6b738713e8	model validation: place the model, then weight the maps by sigma_A ... --model re-fractionalized the model into the data cell and then left it there. On a non-isomorphous pair that is a placement error, not a cell error: the box is squeezed, the body inside it is not moved. Six parameters now recover it - an angle-axis rotation about the model's centroid and a translation, refined over a 6 / 4.5 / 3.5 A ladder, the scale (k_overall, anisotropic B, k_sol, b_sol) re-fitted at every evaluation so the target measures the placement and not the scale. The refinement sees only the working reflections and the step is committed only if R-free, on the free set it never saw, drops; otherwise the model goes back where it was read. Measured on merged lysozyme data against a deposited lysozyme model whose cell differs by 3.4% in c: R-work 0.559 -> 0.400, R-free 0.591 -> 0.383. Over the same 3.5 A range the external arbiter (REFMAC rigid body through dimple) works in, 0.524 -> 0.330 against REFMAC's 0.522 -> 0.355, and the recovered movement agrees with REFMAC's to 0.25 deg and 0.03 A (3.05 deg / 1.04 A vs 2.76 / 0.98). 2.4 s of added wall clock, 234 structure-factor evaluations. The map coefficients become 2mFo-DFc and mFo-DFc. sigma_A is estimated by maximum likelihood per resolution shell on the free reflections only, with the number of shells taken from the size of the free set so no shell is thin; centric and acentric reflections carry their own likelihoods, and a centric reflection's bias-free coefficient is mFo. Cross-checked against CCP4 SIGMAA on the same reflections: mean FOM 0.404 against its 0.396, with the same per-shell structure. The figure of merit is written to _maps.mtz so the weighting can be undone. The per-shell scaling refusal in fit_model stands - Fobs is never rescaled and the R-factors are untouched - but m and D are per-dataset, so maps from one campaign are no longer scaled identically. That is argued at the code and in docs/CPU_DATA_ANALYSIS_DECISIONS.md 14.4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:06:05 +02:00
leonarski_fandClaude Opus 5	5cc2f811b3	indexing: an axis harmonic is a small multiple, and the Wilson B is XDS's ... Two unrelated notes, one on each side of what the merge reports. The axis-harmonic precondition accepted any near-integer volume ratio. Its window is absolute in a unit integer spacing, so a ratio between two UNRELATED lattices passes it about a third of the time whatever the multiple is, and at 124x the pair is not a cell and its harmonic in any sense. That branch was harmless while the rule was "always take the smaller cell"; deciding the pair on the evidence makes it reachable, so it is now bounded at 4. Measured over the corpus: of 44 firings, all 36 at n <= 4 read 1.1 to 51.3 points BELOW the chance occupancy (n-1)/n, so an index-n sub-lattice really exists in each of them; all 8 above it - two datasets, both decided by 8 to 34 sigma, so not a margin problem - read within 7.8 points OF chance, so none does. The largest n at which a real sub-lattice was ever seen is 3, and no dataset moves either way. Separately, WILSON_B and _reflns.B_iso_Wilson_estimate are fitted to log<I> directly, which is what XDS's Wilson line does and is not what TRUNCATE, ctruncate or phenix.xtriage do - they divide out Sigma = sum f^2(s) first. Leaving Sigma in the slope inflates B by 2-8 A^2 on our own merges, and against those two programs on the same files this number runs 10-36 A^2 high, in the same direction every time. The docs said "the analogue of XDS's Wilson-line B", which is accurate but easy to read past; they now say the two conventions are not comparable and which one this is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 13:03:25 +02:00
leonarski_fandClaude Opus 5	476a849c0e	indexing: decide an axis harmonic on which cell explains more of the spots ... When two first-pass schemes return cells whose primitive volumes differ by a small integer, the validation-FRAME count cannot tell them apart: a spurious axis multiple indexes every frame its true sub-cell does, so both reach 60/60 and the count saturates. The rule that then decided the pair was unconditionally against the larger cell, so on a crystal with a real pseudo-translation the true cell could not win in any scheme order. Ask the same question at the granularity where it does not saturate: how many of the validation frames' SPOTS does each cell account for? That comparison leans towards the smaller cell by construction, and needs no threshold to do so. Acceptance is a fractional-Miller test, so multiplying an axis by n multiplies that axis's residual by n: the larger cell places every shared reflection n times less accurately than the sub-cell does, and loses outright the spots that sit in the tolerance margin. The only thing that can pay for that loss is the class of reflections the larger cell ADDS - empty for a spurious multiple, the superstructure's satellite rows for a real one - so the larger cell wins the count only when the extra periodicity is really there. The count is taken over the validation frames' whole spot lists, which reach far deeper into each frame's intensity distribution than the first pass's own accumulation cap, and a superstructure layer is faintest exactly where that cap cuts. Measured over the five crystals of this corpus where the two schemes return an integer-related pair, the larger cell accounts for 1.37x and 1.98x the spots on the two whose true axis was being halved, and 0.30x, 0.36x and 0.71x on the three where the doubling is spurious. All five come out right: the two keep the true cell and its deposited space group, the three reproduce the answer the old rule gave, to the digit. (One of the three has a bistable first pass - four builds give three answers, one of them without this change at all - so it is not evidence either way; the other two are reproducible.) What the added class holds is computed and reported next to the decision, because it is the physics the count is a consequence of. It is deliberately NOT thresholded, and that is the part of this that took the measuring. Refuted along the way: - An occupancy floor, which is how this was first written. Over the five crystals the arbiter is asked about, the emptiest index-n class reads 48.2, 41.6, 37.3, 25.4 and 7.1 %. The two the larger cell should win are the 41.6 and the 37.3, so the three it should lose bracket them on both sides, and a real superstructure elsewhere on the corpus reads 3.4 %, below all five. Recomputing the same statistic on the merged intensities over a sweep of I/sigma cuts leaves the ordering unchanged, so this is a continuum and not two populations: no floor separates them, and no amount of extra data would. The bimodality a floor needs was an artefact of a calibration set that contained no failure. - Requiring the two cells to stand in a genuine sub/super-lattice relation, the change of basis being integral. Measured, all five pairs are index-n relations to within 0.016 of an integer - the volume ratio is not the weak link. - Deciding it on the merge, by integrating and merging both cells. The worst failure does announce itself there (CC1/2 0.9994 -> 0.9566, ISa 18.6 -> 1.4), but it costs a second full integrate-and-merge, and the successes lose 13-30 % of their ISa where another failure loses 27 %, so the metric does not separate them either. - Requiring the sub-lattice class to be the STRONGER of the two, which is the right mechanism but the wrong observable at this point in the run: the sign it turns on lives in integrated intensities, and the spot finder reports no spot at all where a class is absent, so at first pass the same ratio reads 0.90 against 0.39 and 0.27. The ordering survives, the sign does not. The occupancy is maximally wrong on the worst failure because that cell is not a superstructure at all. A beam-centre error along the spindle translates the derotated cloud rigidly, and a lattice shifted by half a spacing is indexable only on a doubled axis - the shift needed scales as 1/(2L), so a long axis is the easy one to half-offset. In that doubled setting the even class is empty and the zero layer reads a negative mean intensity, which no crystal can do, while the odd class carries everything. Such a cell fits no index-n sublattice at all, so its added class reads the chance value (n-1)/n, the largest the occupancy can take, and the occupancy test reports the artefact as more real than any genuine superstructure. The spot count sees it for what it is, at 0.30x. So the beam-centre warning below is now suppressed only when the larger cell WINS. Declining it is a fall back to the default, and that warning - which names the beam centre and offers --estimate-beam-center - is then the most useful thing the run can say; on the half-offset mode it is the correct diagnosis. The same question asked unconditionally of the committed cell, the halved-axis probe, is not included. Measured over the committed cells of 29 crystals, 19 of 189 axes read above the 2 % floor it would have used and the largest read 30 %, and those largest readings are on crystals whose committed cell is already wrong - where doubling an axis is the wrong response. It rescues one crystal whose superstructure layer reads 3.4 %. That is not worth the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 12:46:57 +02:00
leonarski_f	54f2de31bb	grid scan: say how the stationary spindle angle is stated, and pin it ... A grid scan is a set of stills at a stationary spindle, and the angle it stood at is what relates one grid to another taken elsewhere on the circle. Users were finding an all-zero omega in the file and concluding the angle could not be recorded at all. It already can, and has since the goniometer and the grid scan stopped being alternatives: send the axis with step 0 and its start angle, and that angle is written per image into the NXmx sample chain, read back by reader/, and taken by dials.import as a set of stills. Measured on a generated 12-image grid: the placeholder file carries omega = 0 x 12, the same file with the axis sent at step 0 carries omega = 90 x 12, and dials.import reports "still: 1, sweep: 0" for both. Nothing in the code needed changing, so nothing was; what was missing was that nobody could tell, and that no test held the behaviour down. So: the API and the HDF5 documentation now say it in as many words, and three tests pin the three legs the value crosses - the OpenAPI request (which used to drop the grid scan whenever an axis was present, unpinned until now), the CBOR start message, and the file round trip. Also corrects a claim two comments and the HDF5 page were making. NXmx can express "no rotation" perfectly well - a sample may depend_on "." - so the placeholder is not there for the standard's sake. It is there because dxtbx cannot read a sample chain of translations alone: strip the rotation axis from a grid scan master and dials.import dies in get_dxtbx_goniometer with a matmul dimension mismatch. Recorded so nobody removes the placeholder on the strength of the standard. One thing the change does not fix, because it cannot: a stationary angle is invisible to DIALS when a grid scan is present. dxtbx picks the first varying axis as the scan axis, which is a grid translation, so the oscillation reads (0, 0); and with exactly one rotation axis in the chain it builds a single-axis goniometer whose fixed rotation is the identity, never consulting the angle. The same angle IS visible when it is the only candidate (oscillation reads (90, 0)) or when a Smargon head puts a second rotation axis in the chain (the setting rotation then carries it). The value is in the file and correct either way.	2026-09-02 11:04:29 +02:00
leonarski_fandClaude Opus 5	92a615085d	grid scan: the snake reverses on the acquisition row, not the display row ... GetElementPosFast_step took the snake parity from GetElementPosSlow_step, which is the DISPLAY row: it is the acquisition row r = image / n_fast, flipped to (n_slow-1) - r when the slow step is negative. So for a negative slow step the parity it hands back is parity(n_slow-1) XOR parity(r), and with an even n_slow that is inverted on every row - the whole raster comes out mirrored along the fast axis. An odd n_slow leaves it correct, so the same scan collected with 20 or 25 images disagreed about where image 0 sat: n_fast=5, fast +1.5 um, slow -2.5 um, snake on gives images 0..4 at fast index 4,3,2,1,0 with 4 rows and 0,1,2,3,4 with 5 rows. A positive slow step was correct at both counts, and so was every non-snake configuration. Snake means the stage reverses direction on alternate rows in acquisition order, so the parity has to come from the acquisition row. Taking it from image_number / n_fast directly makes the fast index independent of the slow axis and of the row count, and drops the call into the display-row function that caused the coupling. vertical_scan only relabels which axis is fast, so it was wrong in exactly the same way and is fixed by the same line. Affected files: written by an affected build, with snake on, a negative grid slow step (step_y for a horizontal scan, step_x for a vertical one), and an even number of rows. Their /entry/sample/transformations/grid_scan_x or _y is mirrored along the fast axis, as was the grid map in the frontend and the viewer - both mirrored together, which is why neither showed it. Tests: the interaction of snake with the step signs was never asserted, only each in isolation, so add a table over snake x {+,- fast step} x {+,- slow step} x {even, odd row count} x {horizontal, vertical} asserting positions, plus a case running one affected configuration through GetXContainer_m / GetYContainer_m and Rearrange. Every pre-existing assertion is unchanged and still passes; only the four negative-slow, even-row cells of the product move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:57:15 +02:00
leonarski_fandClaude Opus 5	3be79fb1b6	report: name the direction the anisotropy warning is about ... The warning said the diffraction limit "runs from 3.34 to 2.45 A depending on direction" and stopped there, so a reader was told the crystal is anisotropic and given no direction to act on. It fires on 19 of 113 datasets in the battery. The eigenvectors are already measured and already written to the merged mmCIF as `_reflns.pdbx_aniso_B_tensor_eigenvector_N_ortho`; they had simply never reached the human report. The tensor is fitted on s = frac.mat * (h,k,l), so in that Cartesian frame a*, b*, c* are the rows of frac.mat and naming the axis is one dot product per eigenvector. The label is exact in every Laue class that has a free tensor direction except triclinic, and the cosine is printed so a loose fit shows as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:53:31 +02:00
leonarski_fandClaude Opus 5	7b1caa6ce5	mmcif: the free set is _refln.status = f, not a column of our own ... The merged mmCIF carried the R-free flag in `_refln.status_free` as 1/0 and wrote `o` into `_refln.status` on every row. `_refln.status_free` is not in the PDBx/mmCIF dictionary - it is absent from mmcif_pdbx v4.0, v5.0, v5.288 and v5.362, from CCP4's and phenix's shipped copies, and its wwPDB item page is a 404 - and no deposited structure-factor file uses it. `_refln.status` is the item that carries the free set, with `f` for a test reflection and `o` for a working one; on a deposition that also carries `_refln.pdbx_r_free_flag` the two agree exactly. Measured on a real merged file this run wrote: CCP4 cif2mtz refuses the file outright - "Unexpected context type for category REFLN" from its dictionary-validating parser, exit 1, a 12-byte truncated MTZ. Dropping the non-dictionary column is what fixes it: the same file without it converts. gemmi converts, but its cif2mtz spec knows only `status` and `pdbx_r_free_flag`, so FreeR_flag comes out 1 everywhere and the free set is silently lost - R-free would then be computed on the working set. phenix worked, but only by a filename heuristic matching the words "status" and "free". Writing `f` while keeping the extra column is worse than either, because phenix then finds two candidate free-set arrays and refuses the file, so the column goes in the same change. After it, all three read the same 5% test set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:53:31 +02:00
leonarski_fandClaude Opus 5	90d0d3c3f9	docs: the python client reference, for the two schemas that gained fields ... docs/python_client/docs is generated from the API spec and copied in by update_version.sh, so it goes stale between releases. The calibration convergence gate and the beam size both added properties without it; this is the same generator run those commits should have carried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 10:35:12 +02:00
leonarski_fandClaude Opus 5	aa6073d41f	api: chi is a reported angle, not a travel limit ... The Smargon at the SLS 2.0 MX beamlines reports its own axis positions with readout noise, so a chi parked at zero comes back as about -1e-7 degrees. The schema bounded chi_deg to [0, 90], so an ordinary "chi is at zero" setup was refused - and the same holds at the other end of the arc, where a chi parked at 90 reads just above it. Both end-stops are exactly where a static positioner is left, so widening the range would only move the problem. The bound bought nothing. Chi never enters any computation: OpenAPIConvert puts it in SmargonPosition, DiffractionExperiment::BuildTransformationChain hands it to a rotation transformation verbatim, HDF5NXmx writes it and HDF5MetadataSource reads it back. No downstream reads its sign, and a rotation is defined for any angle. phi, the sibling angle in the same object with the same semantics, has never been bounded. What was left was a restatement of a hardware travel limit that the goniometer enforces itself, and its only observable effect was to refuse a value the instrument genuinely reported. It was also not enforced where a server-side check would matter: the cpp-pistache-server generator does not recurse into a nested object model, so Dataset_settings::validate never calls the Smargon model's. The rejection was raised by the generated python and TypeScript clients, which do check. Regenerated the C++ server model, the TypeScript client and redoc-static.html; python-client is gitignored and comes from gen_python_client.sh. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:33:30 +02:00
leonarski_fandClaude Opus 5	8a04773d1d	api: record the beam size at the sample, and write it where NXmx puts it ... dataset_settings gains beam_size_x_um and beam_size_y_um, the horizontal and vertical size of the X-ray beam where it meets the sample. They follow the same route total_flux takes - OpenAPI, DatasetSettings, the CBOR start message, the HDF5 master, and back out of a stored file - and nothing consumes them; this is metadata a beamline can state and a downstream program can read. NXmx puts this in the application definition rather than the base class: not NXbeam's extent (rank 2, nP x 2, per scan point, always FWHM of a rectangular aperture) but NXmx's own incident_beam_size, a recommended rank-1 two-element array in the order x, y. Both are live and neither is deprecated, so the choice matters; the MX definition wins in an MX file. Written as one array with a units attribute of "m", like every other length in the master, so the settings hold micrometres and FillMessage converts once. The unit table of ReadLength_m becomes LengthUnitFactor so the array read can share it: a master written elsewhere may state this in millimetres. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:59:34 +02:00
leonarski_fandClaude Opus 5	7890149ad6	writer: peakCountUnfiltered in the master, like its four siblings ... /entry/MX/peakCountUnfiltered was written only by the data-file plugin. The four other per-image spot counts are written to both the data files and the master, so a reader holding just the master got every count except the unfiltered one. It was not only a missing dataset. HDF5MetadataSource already reads /entry/MX/peakCountUnfiltered from the master and falls back to /entry/MX/nPeaks when it is absent - and nPeaks is the number of spots *stored* for an image, i.e. after the spot budget truncates. On a VDS master the fallback therefore substituted the post-filter count for the unfiltered one silently, and the two differ precisely on the images that hit the budget. EndMessage::spot_count is the right member: it and the data-file plugin's spot_count_total both come from DataMessage::spot_count (the end-message copy via ScanResultElem::spot_count), and CountSpots() sets that from spots.size() before FilterSpotsByCount() applies the budget - "spots found before filtering", as docs/HDF5.md already described it. The CBOR end block already carried spot_count on both encode and decode, so no message or protocol change was needed. SaveVectorIfMissing keeps the NXmxIntegrated case correct, where the data-file plugin has already written the dataset into the same file. Verified by writing files in all three formats and dumping them: the new dataset appears in the legacy, VDS and integrated masters and matches its sibling peakCountLowRes in value, datatype and read-back precedence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:37:53 +02:00
leonarski_fandClaude Opus 5	ef5da29319	calibration: a fit that hands back the file's own tilt is not a calibration ... A powder calibration is run because the file's geometry is in doubt, so a fit that quietly returns part of that file has answered nothing - and it is indistinguishable from one that worked, down to the residual and the sigmas arranged around it. On one of four LaB6 exposures of one detector the tilt came out at 2.93x its own sigma, a hundredth under the significance gate, so it was declined and pinned - at the master's hardcoded rot1 -0.08, rot2 -0.22 deg. That is eight times the tilt just refused, on no evidence, and worth 10 px of PONI at 190 mm. rugnux printed it to four decimal places, wrote the .poni, and exited 0. Judge the result on provenance instead of on any residual: a geometry is a measurement only if every parameter in it came from this data. Two ways out of the fits do not qualify - a covariance that never conditioned, so the fit cannot say what it determined, and a declined tilt pinned at a non-zero value from the file. A declined tilt over a file stating no tilt still qualifies, because reporting no tilt is then exactly what was measured; so does --no-refine-tilt, because a hold that was asked for is a stated choice and not a silent substitution. No single number separates the four. rms is 2.465 px against 1.44-1.64; the significance of all four lies between 2.93 and 4.47, so the gate is nearly a coin flip at these distances and moving it would only recalibrate on one population; and the failed fit has the TIGHTEST parameter sigmas of the set, because pinning the tilt removes the tilt/centre correlation that inflates a good fit's. The spot cross-check reads 13.5 px against 0.98-2.66, but 10.4 px of that is the pinned tilt moving the PONI - the same defect one step downstream, not independent evidence. On a failure rugnux says so, writes no .poni - a PONI file states where the detector is and has no field in which to say it does not know - writes the JSON with converged false and the reason beside it, and exits non-zero. The re-binning pass now prefers a converged refit over a non-converged one whatever its residual, so a tilt an earlier pass measured is not what a later one gets pinned at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:37:46 +02:00
leonarski_fandClaude Opus 5	4bb3d44983	viewer: label the merge plot over the range it is drawn on ... The merge-statistics window asked for an absolute y-axis - CC1/2 and CCref on 0..100, everything else from 0 - by setting the range on the chart's value axis after JFJochSimpleChartView::UpdateData had already built the chart. The visible tick labels are a separate QCategoryAxis whose entries UpdateData had generated from the range of the data, and those entries were not rebuilt. So the plot was drawn over 0..100 while the labels down its left edge covered only 91.5..100 and crowded into the top tenth; the numbers matching the drawn range appeared only on the right-hand grid axis, which the same call had made visible. CC1/2 showed it worst because its range is the narrowest, but every metric was affected. Give UpdateData the range instead, so the ticks and both axes come from one number. The side-panel azimuthal-integration chart passes no range and renders pixel-identically. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:23:02 +02:00
leonarski_fandClaude Opus 5	34e1ec8fb9	docs: keep the analysis landing page and drop a trailing transition ... The four-way split leaves CPU_DATA_ANALYSIS.md as the landing page the toctree points at, and one part ended on a horizontal rule, which docutils refuses at the end of a document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:20:27 +02:00
leonarski_fandClaude Opus 5	d6c6547e0e	docs: point the two source comments at the pages their sections moved to ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	889c9b6cbb	docs: one changelog line for the rugnux documentation work ... Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	67bf380401	docs: credit DENZO/SCALEPACK and MOSFLM for the 2D-then-merge architecture ... Both programs were acknowledged for specifics (profile-fit variances, FFT autoindexing, post-refinement practice) but not for the paradigm rugnux's rotation pipeline is built on: integrate each image in 2D, then combine the partials into fulls, as against XDS's 3D profiles. One paragraph names it; the citations were already on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	ee4c23b9a2	docs: the phenix label override, for the mmCIF as well as the MTZ ... Both merged formats hit the same refusal - the file carries the mean and the Bijvoet pairs - but the label vocabularies differ per format and the MTZ incantation fails on the mmCIF with a fresh error. Give both measured commands, and say it is one behaviour in two formats, not a difference between our files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	64182bb295	docs: an overview page - what a rugnux run does, in order ... The page that did not exist: one paragraph per stage from opening the file to the written reflections, each linking into the data-analysis reference part that carries the depth, with the stills differences at the end. First entry after the landing page, and the landing page says to read it first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	20f869c0b8	docs: split the data-analysis reference into four parts along the pipeline ... CPU_DATA_ANALYSIS.md becomes a short landing page (scope, part map, references) over four parts in pipeline order - images to spots (0-3), indexing and geometry (4-7), integration/scaling/merging (8-12), space group and validation (13-14). Pure moves: the section numbering is continuous and unchanged, since the rest of the documentation and the source cite sections by number. Inbound topical links now land on the right part; the build has zero warnings and the rendered-HTML anchor check finds no dead link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:19:16 +02:00
leonarski_fandClaude Opus 5	376aa0a2e0	docs: split RUGNUX.md into one page per job, and put rugnux first ... The 1274-line page becomes a landing page (quick start, the page map, where it fits) plus seven pages a reader can answer one question from: installing, what rugnux reads, running it, integration with other programs, the results report, advanced usage, and powder calibration. Content is moved, not rewritten - only the connective sentences at each page top are new. Every internal anchor is remapped to its new page and every inbound link (DEPLOYMENT, TOOLS, HDF5, CPU_DATA_ANALYSIS) updated; the built site has zero Sphinx warnings and an anchor check over the rendered HTML finds no dead link. index.rst leads with the rugnux group, then acquisition, FPGA, reference and project. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	784cf87cc8	docs: put the generated python-client reference behind one landing page ... The OpenAPI Python client owned the sidebar: DefaultApi's 128 method anchors plus the 64 hidden-glob model pages were 195 of its 256 entries, because sphinx_material's globaltoc includes hidden toctrees by default. A new PYTHON_CLIENT.md landing page carries the links and a hidden glob toctree, and globaltoc_includehidden is off, so every generated page is still built and reachable (verified: 64 model pages + DefaultApi render, zero warnings) while the sidebar drops to 61 entries. docs/review/ joins exclude_patterns so a local, gitignored review report can never again be rendered into the published site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	6081b6bc43	docs: credit the L test, FFT indexing, TORO, Niggli, peakfinder8 and SparseCCL ... Six methods the pages name or describe carried no citation: Padilla & Yeates (the L test), Steller, Bolotovsky & Rossmann (the projection/FFT autoindexing MOSFLM implements), TORO (what ffbidx implements), Krivy & Gruber and the ITA lattice-character table (the reduction and Bravais assignment), Cheetah's peakfinder8 (the per-ring background statistics of the adaptive finder) and Hennequin et al.'s SparseCCL (already credited to traccc, now also to its authors). Each gets its ACKNOWLEDGEMENT.md paragraph, a References entry in CPU_DATA_ANALYSIS.md, and a one-line credit at the algorithm. The Sheriff & Hendrickson / Popov & Bourenkov entry is re-scoped so each claim sits on the paper that supports it - P&B 2003 is titled, and credited for the sigma-aware anisotropy estimation its statistic modelling contains, not for the tensor and its constraints. All DOIs verified against the publishers; the SparseCCL DOI resolves to IEEE document 9049184 (IEEE blocks content scraping, so verified by the resolved document id plus two independent sources). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	86f8edfabc	docs: the FFT axis ceiling, one-sweep inputs, exit status and _anom.ccp4 ... The longest FFT axis (500 A, no flag; -C moves it) was implied twice and never stated, and its failure mode is a plausible sub-cell rather than a refusal. One input is one sweep - said affirmatively where inputs are described instead of in an aside about pointless. A default 50 A low-resolution cut discards real reflections on a very large cell; the option row says so. Exit status is documented for scripts (0 = completed, non-zero = stopped), and the --model output list gains _anom.ccp4, which was written but undocumented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	87b1402c50	docs: hand the output onward - downstream commands and the three blocking facts ... The page named phenix, REFMAC, POINTLESS, AIMLESS, careless and SHELXC a dozen times without one command line, and left unstated the three facts those programs stop on: the free-flag convention (0=work 1=free; REFMAC needs FREE 1), the phenix label choice the double intensity array forces, and the unmerged file's header symmetry and sort order (determined group, sorted H K L M/ISYM BATCH - WriteReflections.cpp sorts it). A new 'Taking the data onward' section carries the worked lines, the careless column renames (BGVAR is a variance), and the Phaser SGALTERNATIVE keywords for the enantiomorph the report leaves open. The POINTLESS series trap now names ALLOW OUTOFSEQUENCEFILES instead of telling users to touch their data, the report-grep block warns that SPACE_GROUP_NAME carries one member of an enantiomorphic pair by convention, and the P1 cross-check's free set is declared to be its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	b1ae1ece67	docs: a default rotation run already writes real anomalous data ... The -A row and FRIEDELS_LAW=TRUE together read as 'without -A your Friedel pairs were averaged', which is false: the rotation merge always keeps the Bijvoet split and the default .mtz/.hkl carry it (WriteReflections.cpp, BuildMergedRows). Say what -A actually changes - the counting basis and the error model - and give the merged MTZ's exact column labels, which scripting against phenix needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	40cb5cd6db	docs: the quick start tells the truth about inputs, outputs and the cutoff ... A default rotation run writes seven files, not five - the two the list omitted are most of the bytes. The input is any NXmx/EIGER master or miniCBF sweep, which the page said only 350 lines later after twice implying Jungfraujoch data only. The CC1/2-0.30 resolution trim moves up to the quick start, and _image.dat's columns are finally named (ScalingResult.cpp writes a # header). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	40403d7beb	docs: the direction grid bounds the longest findable axis; say so in 5.3 ... The ranking-not-sampling observation was measured on the coplanar-shortlist rescue and is scoped to it now. The angular-resolution bound theta < d_min/(2a) means the shipped 16384-direction grid resolves axes only to roughly 120-150 A, far below the 1200 A the accepted maximum admits, and the failure mode is a plausible sub-cell, not a refusal - the reader raising fft_max_unit_cell alone deserved to know it cannot work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	32adb882eb	docs: disambiguate the reused symbols and the bandwidth definition ... sigma_bw is one physical smear written in reciprocal units in 8.2/11.1 and in pixels in 9; the two Delta-phi of the partiality formula are named; 13.5's |s| = 1/d is reconciled with the s = sin(theta)/lambda of 10.6/14.2; bandwidth is the rms spread, with the FWHM-input conversion (/2.355, BraggIntegrationEngine.cpp) stated. The ice-extinction clause now covers (104), not only (00l), and the AIMLESS <I/sigma> = 2 constant is named as AIMLESS's default rather than pointed at a criterion this project does not use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	e94548c614	docs: the twinning exemption list, and why -1, 2/m and mmm are not on it ... The stated criterion (a merohedral twin law exists) formally exempts every holohedral Laue class, but the code (TwinningAnalysis.cpp) deliberately keeps the low-symmetry ones eligible because pseudo-merohedral twinning through a special metric cannot be excluded there. Say both halves, and give the reference values of the statistics so the mmCIF numbers can be read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	9814846ebd	docs: state the q convention at every numeric q and on the q flags ... Every q in the pipeline is 2*pi/d (Definitions.h, the azint bin mapping), but the numbers in 3.3, 7.6 and 10.10 and the --azim-* flags never said so, and a reader taking q = 1/d would set --azim-q-spacing or --azim-max-q wrong by 2*pi. The 7.6 ice triplet is spelled out so 'within 0.06 of one another' reads as the adjacent-ring spacing it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	14b79b0314	docs: fix the sigma-ratio, the profile-fit term and the outlier cut as written ... The 1.109 in 9.2 is the variance ratio of the shipped 4/6/13 stencil (45/408 pixels), not the sigma ratio the sentence attached it to, and the effect is a bound attained on weak reflections, not uniform. The profile-fit background term is (sum P/v / sum P^2/v)^2 var(b), matching the code; the undefined w is gone. The 13.3 refit cut is N^2 times the model variance (RotationScaleMerge.cpp), not N*sigma^2. The sigma floor is 1 count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	f68fd3d7af	docs: pin the sign convention of the back-rotation in 7.3 ... R(phi) rotates the observations by +phi about the stored axis, which makes it the inverse of the crystal's own rotation - the reading under which the formula as written is correct, and the same convention that has the exported MTZ batch axis and the XDS echo negated. Said explicitly, so a reimplementation cannot take R(phi) for the crystal rotation and land at 2*phi. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	694e687fce	docs: name the beam parameter as the PONI and state the laboratory frame ... Section 1.1 called (x_beam, y_beam) the direct-beam position while constructing it as the point of normal incidence; on a tilted detector the two differ by D*tan(tilt), and a reader loading a header direct beam into it would be wrong by several pixels. Name it as the PONI (the system-wide convention, DETECTOR_GEOMETRY.md), say which point the construction pins, and state the laboratory frame's axes and handedness, which the rotation-sign, R-centring and Bijvoet-hand discussions all silently depended on. Two-theta is computed as a two-argument arctangent; say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	d3879d0112	docs: the change of hand is a label, only the ambiguity choice reindexes ... Both pages said in one place that adopting a model's enantiomorph moves no reflection, and in another that it is applied to the merged reflections and exchanges the Bijvoet mates. The code (ModelValidation.cpp, AdoptModelFrame) does the former: the model's group is a label on the written files, I(+)/I(-) stay as measured, and only the merohedral-ambiguity reindex touches reflection indices. Say that once, consistently, in CPU_DATA_ANALYSIS.md §14.5 and in the two RUGNUX.md passages that carried the wrong reading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	200a2d53b7	report: name the groups the data could not separate, as keys ... SPACE_GROUP_NAME is a scalar and reads like a determination, and the docs nominate a grep of it as the interface. Where several groups predict the same absences it is one of them, picked by convention: the report said so in prose and nothing machine-readable carried it, so a script recorded a coin flip as an answer. Refining against a deposited model in the wrong enantiomorph gives R = 0.549. Four keys, none of which change which group is adopted: the alternatives the data cannot separate; whether the enantiomorph was determined, given, or is undetermined - decidable from the group number alone, so it is answered even where the search did not run; and the higher point group whose promotion was refused, with its reason, which until now existed only as prose. REPORT_VERSION is 5. No existing key changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	247130a841	-S: check the metric the fixed group needs, not only its centring ... Both guards on a user-fixed space group keyed on the centring differing, so an axis permutation within the same centring passed: a run that indexed a=37.909 b=78.031 c=77.594 and was told -S P41212 - which needs a=b with the 4-fold along c - merged through operators that do not act on its own indices, and --mode scale on it reported COMPLETENESS= 195.3, an arithmetically impossible number, without complaint. MetricViolation asks the setting-independent question instead: a group's rotations must leave the cell's metric tensor invariant. The re-seating arm now runs on either failure, so a permuted cell can be reindexed into the setting the group needs rather than merely refused; the refusal arm catches what re-seating could not fix. --mode scale never reaches either arm, so it gets the same test where it fixes its cell and its group, which is the only place an impossible completeness could still be produced. The tolerance is a refusal bound, so it sits above what a correct answer reaches: over 113 corpus runs every group determined from its own cell scores under 0.032 and the permuted case scores 0.764. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	47ec939c39	merged MTZ: FreeR_flag in the convention its own column name carries ... The column was written with 1 for the test set and 0 for the working set - the phenix/CNS numbering - under FreeR_flag, which is CCP4's column name. CCP4's own freerflag writes 0..19 with 0 as the test bin, and REFMAC5's default FREE 0 reads it that way, so REFMAC5 stopped on every file we wrote: "more than half of reflections are in free R set", then "Cannot switch free R flag", exit 1. phenix auto-detects either numbering with equal confidence (measured on both, score 3 each), and rugnux's own reference-MTZ reader already takes flag 0 as the test set, so 0 = test is the numbering that works everywhere and the one the rest of the code assumes. This changes every merged .mtz we write: a script that reads FreeR_flag == 1 as the test set has to be inverted. The mmCIF's _refln.status_free is a separate item with its own convention and is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	762c093b10	unmerged MTZ: export the events the merge would keep, not every event ... The export declared its rows FULL - LDTYPE=2 in the batch header, M=0 in M/ISYM - while filtering them on --min-partiality alone (0.02), where the 3D combine also applies --min-captured-fraction (0.7 on rotation) to the same summed event. AIMLESS reads the FULL declaration, reports "all runs have only fulls" and ignores FRACTIONCALC, so an event that caught a twentieth of its rocking curve entered the merge whole, with a small sigma, and was weighted heavily. There was no cut for the reader to make: the column that would let it make one is the one the reading program has been told to ignore. Dropping those events moves AIMLESS's Rmerge at 1.8 A from 1.353 to 0.694 and CC(1/2) from 0.985 to 0.993. FRACTIONCALC itself is unchanged, values above 1 included. Each part's partiality is the erf pair the predictor computed on that frame, from that frame's own refined lattice and mosaicity, so the parts of one event do not tile the rocking curve exactly - the offset steps by the wedge to within 12% of it, and the sums land in a peak at 1.000 whose 95th percentile is 1.09. It is an honest estimate of a captured fraction, and it is not the number rugnux scales on, so clamping it would only hide the spread. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	b33cae9068	unmerged MTZ: a batch header for every image the sweep spans ... The header set was built from the batches that produced an observation, so an image that indexed nothing left a hole in the phi series. AIMLESS starts a new run at such a discontinuity: measured on a 360 degree sweep with 67 unindexed images it made 23 runs of one sweep, its scale model diverged, Rmeas overflowed to -1266 and the result no longer correlated with rugnux's own merge (Pearson 0.0018). Renumbering the batches contiguously does not help - the split is on phi, not on numbering - so the fix is a header per image over the span the observations cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	fb07263025	predict: integrate every centring node, so a fixed space group costs no data ... With -S the prediction rejected the fixed group's centring absences, so those reflections were never integrated. Two things followed, and only the second was known. The P1 cross-check was withheld on such a run, because a P1 merge missing whole centring classes is misleading rather than merely small - 50% of the nodes on an I lattice, 75% on F, 67% on R. That was the documented reason and it was right. The unknown one is that it cost intensity accuracy. Every predicted reflection marks its signal region so a neighbour's background ring can exclude it (BraggIntegrationEngineCPU, the reflection mask); an unpredicted node is an unclaimed patch of detector, and the neighbouring reflections sweep those pixels into their background and over-subtract - worst at high angle, where the background dominates. On a fixed F-centred group that is three quarters of the nodes: measured against the de-novo run of the same data, <I/sigma> 16.07 against 17.21, CC1/2 0.9862 against 0.9895, ISa 10.93 against 11.92. Predicting them costs nothing downstream, because both merges already decide absence against the group they are merging in: the run's own merge drops them again, and the P1 cross-check keeps them because P1 has none. One integration, two correct merges. The -S output becomes byte-identical to the de-novo run on the five crystals measured, which is the point - pinning a group should not change the answer - and such a run can never be slower than de novo, since it predicts the same reflections and additionally skips the space-group search. The de-novo path is untouched by construction: it already predicted in P. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:13 +02:00
leonarski_fandClaude Opus 5	ffce27ac33	rugnux: the two-pass supercell guard compares primitive cells, not settings ... The guard that catches the bistable supercell collapse between the two rotation passes took the volume of each pass's CONVENTIONAL cell. A centred conventional cell is an exact integer multiple of its primitive one - C and I twice, R three times, F four times - so two settings of the same lattice differ by exactly that factor, and the guard read a change of setting as a supercell. It then forced pass 1's result, and with it pass 1's lower symmetry, on a pass that had found the same lattice in a better one. Ten of 116 rotation datasets tripped it, every one at an exact centring multiplicity: seven at 2.00x, one at 3.06x, one at 3.99x. The last is an F-centred lattice; it was held in P1 where the second pass had found it centred orthorhombic. The two structurally identical guards inside RunPipeline already convert with ToPrimitive first, and their comments say why. This one could not: ProcessResult carried the consensus cell with no centring beside it, so at the comparison there was nothing to convert with. The centring is now carried alongside the cell, set at each of the four places the cell is - the finalized rotation lattice, the reindex into a user-fixed group's setting, the reduction of a doubled cell, and the committed higher-symmetry reindex - and the guard converts both sides before comparing. Nothing else reads it; every other consumer of the cell is unchanged. On the F-centred dataset the run goes from P1 at 2.232 A to I 2 2 2 at 2.077 A, multiplicity 1.86 to 7.07. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-01 23:20:32 +02:00
leonarski_fandClaude Opus 5	ecb0571d8d	integration: the observed centroid is the signal's, not the disk's ... The centroid was a first moment of the RAW counts over the signal disk, so it weighted signal plus background. The background is flat over a disk centred on the PREDICTION, which makes its own centroid the prediction exactly: it adds nothing to the displacement and everything to the denominator, and the measured offset comes out shrunk by I/(I + n*bkg). That factor is worst where the background dominates the signal, which is at high resolution - so the one consumer of this quantity, the geometry post-refinement, fits the beam centre and the detector distance on displacements that are systematically too small, by a factor that varies with resolution. An estimator whose bias depends on the very coordinate it is correcting. Subtracting a flat pedestal from a first moment is exact, and the background is not known until the ring has been read, so the positions of the pixels behind the intensity sum are accumulated alongside it and the correction is applied afterwards: sum(x*(px-bkg)) = sum(x*px) - bkg*sum(x). Both engines, identically. Where nothing rises above background there is no signal centroid to compute and the raw one is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 19:55:47 +02:00
leonarski_fandClaude Opus 5	4026ebc6ab	writer: the direct beam is absent where there is no geometry, not fatal ... Computing it needs a DiffractionGeometry, and that refuses a detector distance under 1 mm or a pixel size of zero. A start message can legitimately carry neither: the writer's own pre-flight check asks it to prove it can create the files for a dataset that never describes a detector, and the master write then threw where it used to succeed - Preflight_TCP fails at rc.166 and passes at rc.165. The dataset is a convenience for whoever reads the file later, so where the geometry is not there to compute it from it is simply not written. Everything else in the master is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 18:58:38 +02:00
leonarski_fandClaude Opus 5	8650ac1c59	rugnux: carry the tilt across the two-pass handoff, and post-refine about it ... Post-refinement was handed the HEADER geometry while the outcomes it refines had been integrated at the first pass's own, and only three numbers came back - beam_x, beam_y, distance. The tilt was dropped at both ends, so the pass that ships started at the header tilt with a beam that step B had moved to absorb a tilt the images do not have. On one in-house mounting the first pass ends at rot1 -0.2995 deg and the shipped value is -0.0992, against a powder calibration of -0.282: the two geometries the two passes use differ by the whole of the tilt error, inside a single run. Both ends move together, and they have to. Step B holds the tilt fixed while it fits the beam and the distance, so the beam it returns is only meaningful about the tilt it was given: handing it the refined tilt without carrying that tilt forward, or carrying the tilt forward under a beam fitted about the header's, each describe a geometry that never existed. The carrier goes from three floats to five and the header snapshot the quality guard reverts to grows with it, so a rejected second pass still returns to the geometry the file states. This is not a tilt measurement and does not make one: rot1 remains the gauge direction of a single-axis rotation experiment. It only stops the two passes of one run from working in two different geometries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 18:49:33 +02:00
leonarski_fandClaude Opus 5	9c19646f6a	refine: fit the direction of the goniometer axis, not its length ... The residual applies angle_rad * |rot_vec| and rot_vec is a free three-vector, so the first pass has been fitting a goniometer rotation SCALE nobody asked for. GoniometerAxis::Axis() then normalises it away on write-back, and RotationIndexer scores the candidate with the normalised axis - so the cell that won the fit is judged under a rotation model the fit did not use. Measured over 43 rotation datasets: the length reaches 1.2%, and the fit-vs-score disagreement a median 0.124 deg and up to 6.19 deg of goniometer angle, against rocking widths of 0.05-0.36 deg. That score picks the lattice class, which nothing later revisits. The fitted length is not a usable measurement of anything either: on synthetic data it recovers 54% of a known scale error, repeated first passes on one dataset disagree with each other in sign, 26 of 43 datasets disagree with themselves, and on the one dataset with a proven 1.3% stage fault it comes out negative. It is absorbing other systematics. The rotation scale is measured properly, once, with cross-validation and gates, in PostRefine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 18:36:32 +02:00
leonarski_f	d32c8d526b	geometry: make which tilt component is restrained two numbers rather than a shape ... Both directions of the tilt now go through the same restraint, with a budget each in shared direct-beam pixels; zero means free. Swapping which component is held, or holding both, is a change to two constants. The perpendicular budget is zero today, which is the arrangement the measurements support: it is the component whose conditioning tracks the 2theta the fit reaches, while the parallel one's does not move with 2theta at all - a conditioning number that ignores the data is the prior talking. This is a clarity change, not a precision one, and the record should not read otherwise: the standing 'freeing the tilt inflates both beam components' cost is the perpendicular component's doing, not the parallel one's, so restraining the parallel one recovers essentially none of it. What it buys is that a beam-centre error is no longer reported as an angle.	2026-09-01 15:34:06 +02:00
leonarski_f	657b6c5d1a	geometry: say that the gauge argument is local to the fit that co-refines the orientation ... The restraint is correct here because the crystal orientation is free alongside the tilt and absorbs the spindle-parallel difference. A stage that freezes the orientation has no such compensator, so whether that component is measurable THERE is a separate question; the comment no longer reads as a claim about the tilt everywhere.	2026-09-01 15:33:51 +02:00
leonarski_f	c0bb159c98	geometry: take the gauge direction in the frame each parameter lives in, and tighten the tilt's budget ... Two corrections to the restraint added in the previous commit, both measured. The beam prior compared the goniometer vector's LABORATORY components against an index into the PIXEL-frame beam centre. det_matrix is PoniRotMatrix * DetectorOrientation::Matrix(), so on a quarter turn of 1 or 3 the pixel X axis is the laboratory Y axis and the comparison picks the wrong component - pinning the determined one and freeing the gauge one, which is worse than having no prior. Two datasets in the corpus are in that state. The direction now comes from projecting the spindle onto the pixel axes' own laboratory images, which is exact for any orientation, any tilt and a spindle at any angle, and equal to the old comparison when the orientation is the identity. The tilt keeps the spindle's laboratory components, because rot1 and rot2 are laboratory rotations applied outside that orientation matrix - same physical direction, each in the frame its own parameters live in. And the tilt's restraint is three times tighter than the beam's, in shared direct-beam pixels, because the data determine the SUM of the two: at equal budgets the shift splits evenly and the reported tilt still followed the starting beam centre at 41% of one-for-one. At a one-pixel budget it follows at 8%, from 79% before this work, while the free component moves by 0.0035 deg over the same eight-pixel swing and the indexing rate does not change. The comment says plainly what the change does not do: it does not make the tilt accurate.	2026-09-01 15:33:51 +02:00
leonarski_f	e441434644	geometry: restrain the spindle-parallel tilt, the same gauge the beam prior already restrains ... The beam prior at XtalOptimizer already treats the beam component parallel to the spindle as the gauge direction of a single-axis rotation experiment and restrains it toward the value it was handed. The detector tilt is that same gauge described a second time, and it was left free with a flat +-3 deg box. The correspondence is not an observation about one beamline, it follows from the convention: PoniRotMatrix builds the detector matrix as R(-rot3,z)R(-rot2,x)R(rot1,y), which puts the direct beam at (beam_x - rot1*D/pixel, beam_y - rot2*D/pixel). rot1 IS beam_x written as an angle and rot2 IS beam_y, component for component. So on a horizontal spindle the gauge tilt is rot1 and rot2 is refined; on a vertical spindle it is the other way round. With only one end of the alias restrained, a beam-centre error the prior refuses to absorb reappears as an angle: measured elsewhere in this campaign, refined rot1 tracks the starting beam centre at 0.072 deg per pixel against a geometric 0.080, while the refined beam never leaves its anchor by more than 0.24 px. Restraining both ends, in the same direction and to the same three-pixel budget, leaves the determined component - the direct beam, and the tilt perpendicular to the spindle - alone. Both restraints now take their direction from one projection of the spindle onto the detector plane rather than from two independent snaps to whichever of X/Y dominates, so they cannot disagree, and a spindle at any angle is handled. On a spindle along a detector axis the projection is the snap.	2026-09-01 15:33:51 +02:00
leonarski_fandClaude Opus 5	5869a27fe2	docs: one line per behaviour change in the rc.166 changelog, named by program ... The section had grown to eighteen entries, several of them describing the same report or carrying detail that belongs in a commit message. Consolidated to thirteen one-liners, each naming the program it concerns - rugnux, jfjoch_writer, jfjoch_broker, jfjoch_viewer - so a reader can find what changed in the part they use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 15:20:40 +02:00
leonarski_fandClaude Opus 5	51e90f5549	report: say what the measured detector tilt is worth, and what it is not ... The report invited the reader to check REFINED_DETECTOR_TILT against a powder calibration. Measured over two detectors' worth of crystals, that check misleads: the rotation fit does one outer round, so it leaves its starting value by only a small and crystal-dependent fraction of the distance to the calibrated value, and on the worse of the two detectors the per-crystal median sits an order of magnitude further from the powder answer than either method's uncertainty. A user comparing one run against their own calibration would conclude the calibration was wrong. What survives the aliasing is the direct beam, which is already printed beside it, and the MEDIAN of the tilt over several crystals on one detector - enough to show up a placeholder or a stale value in the file, not enough to replace a calibration. Say that, and say not to feed one run's value back into the instrument. The conditioning law (VIF = 4.40 tan(2theta_95)^-1.66) was tested as a per-run gate and does NOT order the errors - the crystals whose fit stays put are found in the best-conditioned band as often as the worst - so no threshold is added here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 15:15:01 +02:00
leonarski_fandClaude Opus 5	1a25d396a8	docs: the beam centre is the PONI, the direct beam is the beam ... Document direct_beam_x/direct_beam_y in the CBOR start-message table and in the HDF5 detectorSpecific catalogue, and add the section that says what the distinction is: the PONI is the foot of the perpendicular, the direct beam is where the beam lands, they are distance*tan(tilt)/pixel apart, and XDS ORGX/ORGY wants the second. Also record that the same field is a statement about the user's geometry in a broker master and a measurement in a rugnux _process.h5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 15:15:00 +02:00
leonarski_fandClaude Opus 5	26ba019752	writer: the direct beam beside the beam centre in the NXmx master ... Write direct_beam_x/direct_beam_y, in pixels, into /entry/instrument/detector/detectorSpecific - snake_case and unsuffixed like the rest of that Dectris-style group, with a units attribute like the beam_center_x it must be read against. Computed inside Metrology from the very locals that produce the beam_center_x/y and rot1/2/3 written a few lines away - refined when the offline analysis refined them, the StartMessage values otherwise - and from the same start.detector_distance the translation vector uses. The file therefore cannot disagree with itself: one composed geometry feeds all of them. Provenance differs by producer (the broker's master states the user's geometry, a rugnux _process.h5 a measured one), the field does not. The test writes a real master in all three NXmx layouts at a non-zero tilt and reads the datasets back, and requires the tilt to have moved the point, so a writer that stored the PONI would fail it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 15:12:29 +02:00
leonarski_fandClaude Opus 5	336fafffd0	cbor: carry the direct beam on the start message ... beam_center_x/y is the PONI - the foot of the perpendicular from the sample - so on a tilted detector it is not where the beam lands, and the two are distance*tan(tilt)/pixel apart (~8 px on a real in-house setup). A consumer that wants the beam position, which is what most of them mean, had to redo the tilt arithmetic or get it wrong. Add optional direct_beam_x/direct_beam_y to StartMessage, filled from the geometry's own GetDirectBeam_pxl() so there is no second formula, and put them at the top level of the CBOR start map beside beam_center_x/y. Optional keys, so a consumer that does not know them skips them. No _pxl suffix: the stream2 neighbours (beam_center_x, pixel_size_x, detector_distance) carry none and the units are in docs/CBOR.md. The API's calibration schema spells it direct_beam_x_pxl because its neighbours there are beam_x_pxl - same quantity, each matching its own neighbourhood. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 15:12:28 +02:00
leonarski_fandClaude Opus 5	0e245d28cf	docs: name the in-house SLS 2.0 data beside the public sets ... The acknowledgement read as though the whole test corpus came from other people's beamlines. It does not - in-house data collected at SLS 2.0 is tested alongside it. One clause, at the top of the section; everything that follows still concerns the public data, which is the part that carries an obligation to cite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 09:58:40 +02:00
leonarski_fandClaude Opus 5	04d823135e	rugnux: record what the two-pass completeness arm can decide, and what three stand-ins could not ... Only under -S does the arm read a completeness with headroom. De novo the search merge is in P1 and does not count its possible reflections, so there is no number and CC1/2 is the only live term. Three ways to give the arm one back were measured on 28 rotation crystals and none lands, so say so at the guard rather than leave the next reader to rediscover them: * Count the possible reflections on the search merge too. It is affordable - 5.2 ms against a 38 s run - but a healthy crystal reads 27-43% of the P1 hemisphere, so the 100.5% bound is never approached and no decision changes. It would only add a figure to the user's report that reads as the dataset's completeness while being a fraction of a different group's asymmetric unit. * Observation count. The two search merges are built in the same terms, so pass 2 retaining under 90% of pass 1's observations looks like the signal the completeness ratio stood in for. It never fires - bit-identical on all 28, including the crystal whose pass 2 predicted 44% fewer partials. Predicted partials and observations that survive into the merge are not the same population. * Completeness ratio. Same idea one level up, and it does fire - on the wrong crystals. This merge is deliberately never resolution-cut, so the possible list grows with whatever range the pass reached and a pass that predicts FINER scores as one that lost the sweep. At a 0.90 bound it reverted two crystals indexing at 100%, taking one of them from 0.07% to 0.29% cell deviation and ISa 19.5 to 17.1, and rescued nothing. Bringing the wrong-cell detector back for de-novo data means putting it on the final merge or retiring the arm and saying so; neither is done here. No test covers the decision line: Rugnux_Rotation does not reach this guard - the two passes in its log are the indexer's, not RunAllPasses' - and there is no fixture for building a RotationScaleMerge, so an assertion placed there would never execute. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 09:48:39 +02:00
leonarski_fandClaude Opus 5	8935a59721	rugnux: the two-pass guard says when it has no completeness, instead of printing 0.0% ... The guard that judges the post-refined pass against the header-geometry pass reads completeness off each pass's space-group search merge. That merge does not count possible reflections, so the quotient is 0.0 on both sides and every de-novo rotation report has been printing completeness 0.0% vs 0.0%, CC1/2 before corrections 0.994 vs 0.993 as though a comparison had happened. On the 28-dataset in-house battery that is all 28 runs. The number is not just uninformative, it is a test that did not run: "completeness above 100% means the cell is wrong" is the guard's wrong-cell arm, and it cannot fire against a constant zero, so only the CC1/2 arm decides - which is computed on whatever data the pass kept, so a pass that discards a large part of the sweep can post an equal CC1/2 and win. Carry a measured flag beside the number. The arm is skipped and the log and PASS_DECISION say "completeness not measured" where there is nothing to read. No behaviour changes: the arm could not fire before either. The next commit gives the merge the count so it can. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 09:48:25 +02:00
leonarski_fandClaude Opus 5	5ce7eb3014	docs: one changelog line for the geometry reporting work ... Three lines were appended over the course of the work - the tilt and direct beam in the report, the measured tilt beside them, and the post-refine beam bound. They are one change to a user: the report now describes one geometry and says which point is which, and the bound that governs it is measured against the run's own measurement rather than against the file it is meant to correct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 07:39:23 +02:00
leonarski_fandClaude Opus 5	f98442077e	post-refine: bound the beam move on the measurement, not on the header ... PostRefine step B refuses a fitted beam centre more than 15 px from the geometry the run started from. Two things were wrong with that and they compounded. The 15 px was BOTH the Ceres search box and the acceptance threshold, so a fit that wanted to move further was pinned at the box face, landed at exactly hypot >= 15.0, and was then refused for being there. The gate never saw the fit it was judging. Measured on one crystal at three detector distances, the fit came back at 15.128, 15.133 and 15.130 px - the box corner, three times. And it was measured from the header, which is the value most worth correcting exactly when it is most wrong. The run already knows better: the default beam-centre check fits the centre from the isotropy of the scattered background on every rotation run, reported it, and then discarded it. The bound is now measured from whichever of the nominal centre and that measurement is nearer - accept a move that is small relative to something independent, rather than small relative to the header alone - and the search box is widened to match, so the gate decides rather than the box. The measurement is used only where it was precise enough to be adopted as a centre in its own right; a run without one is bounded exactly as before. Step B also now says WHICH of its four tests refused a fit, since all four leave the same geometry behind. Measured over 113 rotation datasets against the same corpus, one binary per arm, with a control arm that reproduced the stored battery on 112 of 112 reports: the bound fired and refused a real correction 1 6ukf the bound fired and caught a runaway 0 the bound never fired 112 On 6ukf the header centre is 22.1 px from what the background fit measures. The old fit, pinned at the box, improved the held-out positional residual 9-fold and was refused; the new fit lands 0.46 px from the measurement and improves it 418-fold. ISa 4.42 -> 6.71, R_meas 18.5 -> 12.8 %, CC1/2 0.987 -> 0.995, d_min 1.119 -> 0.973 A; at matched resolution, 1.68 A goes I/sigma 8.9 -> 15.1 and R_meas 28.8 -> 13.4 %. The refined cell moves from 0.4 % off the deposited one to under 0.1 %. It survives perturbation, and inverts it: over +-0.5 mm of detector distance the corrected run is flat (ISa 6.62 / 6.71 / 6.94) while the uncorrected one is bistable (4.02 / 4.42 / 6.90). The change removes an instability rather than exploiting one. No space group anywhere in the corpus moves, and no other dataset's report changes at all - 112 of 113 are identical. The distance half of the gate is untouched: it is separately argued, and it is what legitimately refuses the one dataset whose distance wants to move 1.05 %. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 20:19:36 +02:00
leonarski_fandClaude Opus 5	efca844bce	calibration: --no-refine-tilt holds the header tilt, it does not zero it ... GuessInitialGeometry resets rot1/rot2/rot3 to zero before seeding, and the branch that puts the header's tilt back was guarded by `refine_tilt && !tilt_refined`. With --no-refine-tilt the tilt is never free, so tilt_refined is false, so the guard is false, so the restore never ran - and the fit reported a zero tilt while the CLI printed "held at the header value". The guard only needs !tilt_refined: a tilt that was declined and a tilt nobody asked to refine both want the header put back and the beam centre and distance refitted around it. That is what the branch already does. Measured on a LaB6 sweep with a tilt imposed on the command line and asked to be held, --calibration spots: before Rot1= +0.0000 deg (+0.001400 rad from the header) after Rot1= -0.0802 deg (+0.000000 rad from the header) The rings path reaches this only when the spot-derived start wins, which is why it does not show on a file whose profile start is chosen; the spots path always did. A calibration whose header tilt is already zero is unaffected - checked, byte-equal distance either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 19:47:02 +02:00
leonarski_fandClaude Opus 5	dcfcff5bc4	report: the measured tilt, and one geometry instead of two ... Adds REFINED_DETECTOR_TILT - rot1/rot2 in degrees as rotation indexing measured them - so the value can be checked against a powder calibration, which measures the tilt from ring shape and is the better instrument for it. rot3 is not carried: a rotation about the beam is an exact null of a single-axis rotation experiment, so the fit cannot move it and printing it would suggest otherwise. Also repairs two defects in the commit that added DETECTOR_TILT, found in review. The geometry was taken from the caller's copy of the experiment, which is the state BEFORE the run, while BEAM_CENTRE and DETECTOR_DISTANCE come from result.used_*, which is the state after - so on a rotation two-pass the report mixed a post-refined centre with the file's rotations and DIRECT_BEAM was off by the whole post-refinement shift. JFJOCH_DATASET_SETTINGS carried the same mixture, which is a geometry no pass ever ran at. The tilt and the direct beam are now taken from experiment_ alongside the other three, so the block describes one geometry. And the prose claimed the tilt "is not refined here". It is: rotation indexing refines rot1/rot2 on every rotation run. What is true is that the result is never written back onto the geometry, which is why it needs a key of its own. M_PI -> PI (common/JFJochMath.h). rugnux is in the forced-Windows viewer-only set and M_PI is not portable to MSVC; the file did not include <cmath> either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 19:35:27 +02:00
leonarski_fandClaude Opus 5	7c00731992	docs: publish the powder calibration schemas in the python client docs ... docs/python_client is tracked and is what readthedocs publishes, but it is generated - update_version.sh copies it out of python-client after regenerating the client - so adding schemas to the spec left it four files behind. The README's model list did not name them and their pages did not exist. Copied the way update_version.sh copies them. The four new pages and four lines in the README, nothing else: the generator's README is built from the spec, so the stale Calibration* pages it had left on disk under the old names are absent from it and were removed before copying rather than published alongside their replacements. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 19:02:12 +02:00
leonarski_fandClaude Opus 5	bd6e933f04	api: name the calibration schemas powder_, not calibration_ ... "Calibration" already means something else in this broker: the JUNGFRAU pedestal and gain measurement, which owns /statistics/calibration, calibration_statistics and JFJochState::Calibration. Four schemas called calibration_* would have sat beside it meaning a completely different procedure, and the confusion is cheapest to remove now, before an endpoint returns them. calibration_output, calibration_quality, calibration_fit_sigma and calibration_spot_check become powder_calibration_output, powder_calibration_quality, powder_calibration_fit_sigma and powder_calibration_spot_check. All four, not only the outer one - the collision is in the word, and half a rename would read as though the inner three belonged to the other kind of calibration. Rename only. Every client regenerated from the spec: the C++ server model, the TypeScript frontend client, redoc-static.html, and the python client, which is gitignored but was checked to still decode the file rugnux writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:58:24 +02:00
leonarski_fandClaude Opus 5	6fe3f30ab4	api: define the powder calibration result as calibration_output ... The JSON a calibration writes was a shape invented at its writer, described only by the comments around it. That is enough for a file somebody reads with jq and not enough for anything else: a client cannot type it, and an endpoint returning it later would have to declare the shape a second time and keep the two in step by hand. So declare it where every other shape in this system is declared. calibration_output holds dataset_settings and a calibration member; the latter is calibration_quality, which nests calibration_fit_sigma and calibration_spot_check. The descriptions carry what a reader has to know to use the numbers rather than only what they are named - that beam_x_pxl is the PONI and the direct beam is elsewhere, that the rotations travel together because a body omitting them states a flat detector, that a tilt below about three sigma was declined and pinned, and that the two correlations approach 1 as the tilt stops being separable from the beam centre. Nothing references it yet. It is declared now because /powder_calibration will return exactly this, and because the file rugnux already writes is decodable today: jfjoch_client's CalibrationOutput.from_dict reads it as it stands, with o.calibration.fit_sigma.correlation_beam_x_rot1 and the rest typed. Generated clients regenerated from the spec, as the spec requires: the C++ server model (four new pairs under broker/gen/model), the TypeScript frontend client, and broker/redoc-static.html. Both regenerations are purely additive - no existing generated file changed except to export the new names. The python client regenerates from the same spec and is gitignored. The test now validates the WHOLE file against the generated Calibration_output rather than only its geometry member against Dataset_settings, so the quality block is under the same contract: a field renamed or newly required in jfjoch_api.yaml fails here rather than at a client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:54:09 +02:00
leonarski_fandClaude Opus 5	8691bf4876	calibration: write the result as JSON, not only as a PONI file ... A calibration run produced one file, and it was a pyFAI PONI - which pyFAI and its neighbours read and nothing in this system does. Carrying the answer back into the instrument meant a person reading numbers off a printed report and retyping them into a dataset_settings body, and the report is where the two points a "beam centre" can mean are easiest to confuse. So write <prefix>.json beside it. Its "dataset_settings" member holds the geometry under the property names broker/jfjoch_api.yaml gives them and holds nothing else, so it is a valid dataset_settings body as it stands: curl -X POST -H 'Content-Type: application/json' \ -d "$(jq -c .dataset_settings det.json)" http://broker:5232/start beam_x_pxl is the PONI, as everywhere here. The three poni_rot*_rad ride along whenever any is non-zero and are left out when all are zero: a body without them does not leave the tilt unstated, it states a FLAT detector, so they travel together or not at all - the same rule the report's JFJOCH_DATASET_SETTINGS block already follows. The "calibration" member holds what the run knows about that geometry: the residual, the fit's own sigmas and the correlation between the tilt and the beam centre, whether the tilt cleared its significance test or was declined and pinned, where the direct beam lands, and where the spots independently put the beam. A calibration that has gone wrong looks exactly like one that has not until those are read, and a machine-readable file that carried only the geometry would be the easiest possible way to feed a bad one into an instrument. Tested against the model generated from the spec rather than against a list of field names written out by hand: the file's dataset_settings member is parsed into org::openapitools::server::model::Dataset_settings and validated, so a field renamed or newly required in jfjoch_api.yaml fails here rather than at someone's POST. The tilted and untilted branches are both covered, and the PONI/direct-beam distinction is asserted rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:47:51 +02:00
leonarski_fandClaude Opus 5	16b368739e	calibration: re-integrate the images about the geometry that was fitted ... Everything the ring fit reads was binned with the geometry the run started from, and a calibration is run precisely because that geometry is in doubt. Binning is not something a later fit can undo: which pixel landed in which (q, phi) bin was decided when the images were read. Get the distance wrong and the radial sampling is the wrong scale - a run recovering 110 mm from a 250 mm header ended at rms 1.07 px where the same data from a right header gives 0.42. Get the centre wrong and every ring is smeared across its own sectors, and past about a hundred pixels the fit leaves rms 4.8 px even when it is started from the exact answer, because there is nothing left in the profile to fit. So integrate the images a second time, binned about what was fitted, and fit that. A calibration run is a handful of images and the answer is worth far more than the extra read. Re-binning about the FIRST fit is not enough on its own. Where the beam centre was badly out that fit is itself wrong, and binning about it digs deeper - rms 3.75 -> 5.66 on an exposure 200 px off. The spots' geometry is the one that does not degrade there, having never read the header, so both are tried where they differ and whichever comes back better is kept. On that exposure the first candidate gives 288 points at rms 5.66 and the second 369 at 0.81. The second pass is taken only if it is actually better, by the same rule that ranks everything else here - at least half as many ring points and a smaller residual - and it is skipped altogether when the fitted geometry moves a ring by less than the radial width of one profile bin, since re-binning would then put every pixel back where it already is. On the 110 mm exposure with its own header that is 1.77 px against a 1.84 px bin, so the run is untouched and its answer bit-identical. Measured on the 110 mm LaB6 exposure, true PONI x 765.90 at 110.03 mm. The calibration is now independent of the header it was given: beam-x out by +20 +40 +100 +200 +400 px -> 765.9-766.6, 110.02-110.05 mm 250 mm header, beam-x 780 -> 765.998 at 110.037, rms 0.394 250 mm header, beam-x 867 -> 765.723 at 110.038, rms 0.413 900 mm header, beam-x 1167 -> 766.496 at 110.038, rms 0.804 30 mm header, beam-x 967 -> 766.486 at 110.039, rms 0.807 All of those failed before this, most of them catastrophically. The residual inflation is gone with them: the 250 mm case now leaves 0.394 px, better than the same data from its own correct header. The four other datasets are unchanged where the re-bin fires, and slightly better where it does: rms 0.433 -> 0.423 at 150 mm and 0.555 -> 0.550 at 200 mm, with the geometry moving under a tenth of a pixel. A run that needs the second pass costs about twice one that does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:17:17 +02:00
leonarski_fandClaude Opus 5	49dd01331b	calibration: let the spots vote on the geometry in ring mode too ... The rings and the spots fail in different regimes, which is the whole reason to carry both. The header is right on a well-configured instrument and is the thing a calibration is run to check. The profile's own estimates are exact while the error stays small and stop meaning anything beyond that - past about half a ring spacing each ring reaches the azimuthally averaged profile as two horns rather than one peak, and the distance search reads a list of horns as a list of rings. The circle through the spots reads nothing from the header at all: measured on a 110 mm LaB6 exposure, --calibration spots returns the same geometry from a header 400 px out in the centre AND eight times out in distance. So ring mode now finds spots as well - half a second - and offers what they make of the geometry as one more starting hypothesis, fitted like the others with the residual left to choose. It is added whole rather than as a centre alone: GuessGeometry votes for the circle centre, clusters the radii into rings and takes the distance from the innermost, and those two belong together. Taking only its centre would not have helped, because the distance candidates come from a profile averaged about the header's centre, and a ring smeared over hundreds of pixels cannot be un-smeared by reading its bins differently. Measured on the 110 mm exposure, whose true PONI is 765.90 at 110.03 mm. A header 40 px out in the centre now lands within 0.6 px (it landed 41 px away before). The cases with BOTH wrong, which failed before this and equally before the beam centre work, now come out: 250 mm / 780 px gives 110.041 mm and 766.10 px against 42.3 mm and 782.7; 250 mm / 867 px gives 110.065 and 765.48. All five datasets are unchanged from their correct headers and the distance still recovers from any header between 25 and 1200 mm. Past about a hundred pixels nothing rescues ring mode, and the reason is the profile rather than the seeding: binned about a centre that far out it shows each ring smeared across its own sectors, so even started from the exact answer the fit leaves rms 4.8 px and drifts. Re-binning would fix it and would need the images read a second time; --calibration spots, which never touches the profile, already covers it. That regime is now visible rather than silent. The spots' beam centre is printed beside the fitted one as a cross-check - two methods sharing no assumption, so a reader can see at a glance whether they agree. It costs nothing, the spots having been found already, and it separates cleanly: 0.3-0.4 px on the good runs against 201.4 px on the exposure whose profile could not be fitted at all. Reported as a fact and not gated on, since at long distance both methods weaken together and the honest thing is to show the number (5.2 px at 300 mm, 12.6 at 500). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:58:26 +02:00
leonarski_fandClaude Opus 5	c84b91be8a	calibration: take the beam centre from the rings too ... The header's beam centre was the last input the ring fit had to be roughly right about. Each ring is looked for in a window a few pixels of radius wide, and a centre wrong by (dx, dy) puts a ring at a different q in every sector, so past about ten pixels the ring leaves that window over much of the turn - and the fit then reads its cos(phi) signal off whichever sectors are left, which are the ones where the signal is weakest. A 20 px error ended 31 px wrong. The rings answer this without a calibrant and without a distance. A powder ring is a conic centred on the beam, so a wrong centre makes EVERY ring's radius oscillate once per turn by the same amount: r(phi) = R + dx cos(phi) + dy sin(phi), solved directly and pooled over every ring the profile shows, with each ring searched about its own measured radius rather than about where a standard says it should be. Using it needs the extraction to follow the rings sector by sector, which is what ProfileRingTrack now does - exactly, and in all five parameters at once, by walking the ring in the geometry believed true and asking the binned geometry what q and azimuth it would have given each point. That replaces the flat-detector distance correction it grew out of. Following the rings is not free, and the reason is worth stating: a window that moves with phi makes every systematic of the peak finder - where the background line is taken, how the centroid sits in the window - vary with phi as well, and phi is exactly the axis the beam centre is read off. Measured, it costs rms 0.415 -> 0.525 px on a good 110 mm fit, and 0.831 when the window follows the fitted tilt too. So a second measurement is taken with a window that is the same in every sector - the binned geometry with only its DISTANCE replaced, which is phi-independent by construction - and both are offered to the same rule that ranks everything else here. Acquire by following, measure by holding still. The seeded centre is likewise a hypothesis and not a belief. It reads a once-per-turn wobble, and a tilt puts a term of that shape there too - one that grows as the radius squared, where a centre error does not - so pooling the rings absorbs part of the tilt into the centre. Believed outright it made a good 110 mm fit worse; offered as an alternative start it costs one more fit and needs no rule about when it applies. It is skipped entirely below a pixel, where it is not a different hypothesis at all, which keeps a well-headed run at 0.71 s. Measured on the 110 mm LaB6 exposure, whose true PONI is 765.90: a header centre 20 px out now lands within 0.5 px, where before it landed 31 px away. All five datasets are unchanged from their correct headers, and the distance still recovers from any header between 25 and 1200 mm. The limit is now understood rather than merely reached. Past a few pixels the azimuthally averaged profile stops showing rings: a ring tracing r(phi) piles up density where that turns round, so it averages into the two HORNS of the sinusoid, at R-|d| and R+|d|. The radius finder reports two rings where there is one, and the gap between them is 2|d| - the search window shrinks to exactly the offset it was meant to span. That caps recovery at roughly half the ring spacing, about 20 px here and failing by 40. Beyond it nothing is left in an azimuthally binned profile, and --calibration spots, which works from the spot positions themselves, is the method that still can. One pre-existing limit measured and NOT introduced here: a wrong distance together with a centre more than about 5 px out fails, because the centre error splits the radius list the distance search reads. The committed code before this change fails identically on those cases. Also fixed: fit_from now takes a whole geometry rather than a distance, and the declined-tilt refit was inheriting rot1/rot2 from it - pinning the tilt at exactly the unvalidated value the gate had just rejected. Same fault the gate exists to catch, one level up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:38:36 +02:00
leonarski_fandClaude Opus 5	9267bd67de	report: the detector tilt, and the two points the beam centre is not ... BEAM_CENTRE is the PONI - the foot of the perpendicular from the sample - so on a tilted detector it is not where the direct beam lands. The report printed only that one number and never mentioned the tilt at all, so the two points were indistinguishable to a reader. On a detector at 85 mm with a 0.22 degree tilt they are 8.3 px apart. Print DETECTOR_TILT (rot1/rot2/rot3, degrees, as the run used them - nothing is refined here) and DIRECT_BEAM beside BEAM_CENTRE, and say which is which. JFJOCH_DATASET_SETTINGS was the more serious half: it exists to carry a geometry back into the instrument, and dataset_settings carries poni_rot1/2/3_rad, but the block omitted them - so it described a FLAT detector, silently, on every tilted setup. The rotations now ride with it when they are non-zero; they stay out when they are zero, because the API's own default is 0.0 and the shorter block means the same thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 17:35:03 +02:00
leonarski_fandClaude Opus 5	01ad1e1743	calibration: only report a tilt the fit actually measured ... The detector tilt is refined by default, and on a pattern that cannot separate it from the beam centre the fit returns one anyway - there was nothing to stop it. Both displace a ring's radius as cos(phi), and only how that amplitude grows with the ring's radius tells them apart, which takes two well-sampled rings. At 500 mm on the LaB6 series only two rings reach the detector and the outer one is barely there: the tilt came out at the opposite sign to every shorter distance, dragged the PONI 28 px, and bought a residual of 0.960 px against 0.962 pinned. The covariance says so plainly - 0.1 sigma, and a beam centre quoted to +-180 px. So ask it. A tilt is kept only where the fit had it free AND it stands at least three times its own uncertainty; otherwise rot1/rot2 go back to the header's values and the beam centre and distance are refitted around them. Over the series the tilt stands at 50, 33, 15 and 8 sigma at 110 to 300 mm and 0.1 at 500 mm, so any threshold between 2 and 5 gives the same verdict on all five - this says which regime a fit is in, not where a line was drawn. The declined 500 mm fit lands on a direct beam of 773.53 px, against 773.56 for the pinned fit measured independently. It is a rejection criterion and nothing more. Clearing it does not certify a tilt: that estimator is limited by systematics rather than by this sigma, and a coherent half-pixel error in the ring positions fakes a tilt of the usual size while leaving sigma small. The report says "refined", never "verified". Writing the gate turned up a related fault in the pass loop. RingOptimizer pins the tilt by itself when every point it is given lies on one ring, and on a barely-sampled pattern a later pass lands in exactly that state - which froze the tilt at whatever the FIRST pass had produced and returned it with sigma zero, an unmeasured tilt wearing the appearance of a fixed one. The gate reads that as "not measured" and refits pinned, which is why it is stated over the geometry that gets reported rather than over what the last fit happened to do. Both paths are covered, profile and spots; the spots path was reporting a refined tilt as declined for the same reason. Two things measured and NOT taken: A robust loss. A Cauchy loss scaled to the previous pass's median residual changed nothing on the series - rms 0.415 to 0.421 at 110 mm, no case improved, every direct beam within 0.06 px. Ring points are per-sector peaks that already had to stand 3 sigma clear of their own background, so there are no gross outliers left to reject. Recorded at the call site rather than left as an unused option. A quality gate that refuses a bad calibration. Three candidate signals, all measured against naming the wrong standard on LaB6 data: sigma(PONI) does not see it at all (0.52-0.65 px, indistinguishable from healthy); the residual only half sees it (3.2-3.6 px wrong against 0.4-1.0 right, but a correct run from a wrong header sits at 1.0-2.4 and would be caught too); and the seed's match score is dominated by how many rings the calibrant lists, scoring 0.29 for a perfect LaB6 fit against 0.21 for a wrongly named silicon. None of the three separates, so no gate is shipped. What the run does say is the recovered distance against the header, and a wrong standard moves that to 446 mm on a 110 mm exposure - unmissable, and the operator's call rather than a threshold's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:07:37 +02:00
leonarski_fandClaude Opus 5	a5f416fcdc	calibration: take the detector distance from the rings, not from the header ... A powder calibration is run because nobody is sure the header is right, and the header's distance was the one number the fit could not survive being wrong about. The ring search is local - each ring is looked for inside a window a few pixels of radius wide - so a distance more than a percent or two out puts every ring outside its own window, and the fit then converges on whatever background fluctuation each window contains. It does not fail: a 110 mm exposure told the detector was at 150 mm reported 149.8 mm, with 146 ring points and exit 0. Only its residual said anything, 5.3 px against 0.4 px, and nothing read it. Measure the distance from the rings instead. The peaks of the azimuthally averaged profile give ring RADII, and a radius does not depend on the assumed distance at all - bin i holds the pixels at one particular radius whatever q that radius was called - so the radii are a property of the image. Against the calibrant's d-spacings, r = D tan(2 asin(lambda/2d)) then has one unknown. It is scanned rather than solved because the pairing of observed rings to d-spacings is unknown too, and the winning basin is solved in closed form. Nothing here reads the header distance except to bin the profile; it needs only the wavelength, the pixel size and the detector's extent. A powder pattern has genuine distance aliases, so one answer is not enough. A cubic primitive standard puts its rings at radii proportional to sqrt(N), and scaling the distance by sqrt(2) maps ring N onto ring 2N - most of the comb still lands on peaks. Measured: the 110 mm exposure with a 115 mm header scored its best at 156.5 mm, which is 110*sqrt(2). No adjustment of the score removes an alias the lattice really has, so the scan hands back the few best distances and each is fitted, the header among them as one hypothesis of several. The residual then separates them - 0.4 px against 5.2 px on that case - subject to an attempt explaining a comparable share of the pattern first, because a start so wrong that one ring point survives leaves a residual of exactly zero. Each attempt re-extracts at the geometry it converged to and fits again. The seed is measured from blended peaks and is good to about a per cent, close enough to converge from but far enough to sit every search window a few pixels off its ring, and an off-centre window takes its background off the ring's own flank. Nothing is re-read from disk, so the loop is free. Measured on the LaB6 distance series. A 110 mm dataset now recovers 110.03-110.17 mm from any header between 25 and 1200 mm, against +-2 mm before. All five datasets recover their own distance from a fixed wrong 250 mm header. With correct headers, four of the five are bit-identical to before and the 500 mm one moves by a single ring point - the two-ring fit whose tilt is 0.1 sigma anyway. Run time is unchanged at 0.62 s. The residual is larger on a run whose header was wrong (1.1 px against 0.4 px on the 110 mm case), because the profile was still binned at the wrong distance and its radial sampling is correspondingly coarse. The geometry is right; only the scatter about it is inflated. Re-running with the recovered distance recovers the residual too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:48:32 +02:00
leonarski_fandClaude Opus 5	d275bbcae7	calibration: -C overrides the calibrant, with absences from -S ... The five named standards are a convenience, not the limit of what a powder calibration can be run against. A unit cell given with -C now IS the standard in --mode calibration, and its rings are enumerated from that cell. A cell alone does not give a ring list, though: the centring and any glide decide which hkl the lattice actually diffracts into, and the fit pairs the innermost OBSERVED ring with the innermost LISTED one - so a list opening with a reflection that is not there scales the whole calibration by the ratio between them. Where -S is given, the absences come from the space group itself via gemmi, which covers centring, glides and screws in one mechanism rather than the three hand-written conditions the built-in table uses. The two agree exactly on LaB6 and CeO2, which is the cross-check that says the gemmi route is safe to hand a user's cell. They do NOT agree on silicon, and the test now pins that: Fd-3m's symmetry absences are only the F centring, while silicon's 222 and its relatives are extinguished by its two-atom basis - a structure-factor absence, not a symmetry one, so no symmetry handler can know it. gemmi offers 24 rings where the diamond condition gives 18. The extra rings do not move the first one, so the distance is not scaled, but they are rings with no intensity offered to the matcher - which is why --calibrant si stays, and why the usage text says -S supplies symmetry absences only. Without -S the cell is taken as primitive and the log says "assumed primitive". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:23:37 +02:00
leonarski_fandClaude Opus 5	5a80d2df53	calibration: fix four ways the powder fit quietly loses its input ... None of these changes the answer on a well-separated cubic standard - the LaB6 distance series is bit-identical by both methods - but each one is a case where input is dropped or mis-assigned without saying so. The circumcentre vote grid was a fixed 4000x4000 box, and the caller never passed anything else. That allocated 128 MB whatever the detector, and on a detector larger than 4000 px in either direction it put the beam centre outside the grid, so every vote was discarded and the guess failed with "Beam center not found". Span the spots' own bounding box instead: a powder ring encloses its centre, so that is where the answer has to be. uint32 votes while there - the most any bin can take is C(500,3). Spots were assigned to the FIRST calibrant ring within a fixed 0.1 1/A, not the nearest. Silver behenate's orders sit 0.108 1/A apart and hexagonal ice has three rings inside 0.06, so for those two standards the window reaches the neighbour and every point lands on the lower-q ring of the pair, biasing the distance. Take the nearest ring, and clamp the window to half the gap to the neighbour - which is what the profile path already did inline, now shared as RingMatchWindow and covered by a test that checks it actually narrows on the crowded standards and not on LaB6. A profile bin no pixel fell in is NaN. SectorPeakQ dropped such a sector by accident, through NaN comparisons falling false; check the four background bins and return explicitly. Ice-ring handling is switched off in calibration mode. Flagged spots are sorted last by the spot budget and so discarded first, which for --calibrant ice throws away exactly what is being calibrated on. The two per-ring std::cout lines in GuessGeometry are gone: a library has no business writing to a terminal, and constructing a Logger to keep them would emit a version banner from inside a fit. What matched belongs in the result struct, which the quality gating still to come needs anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:18:27 +02:00
leonarski_fandClaude Opus 5	fba5435c38	calibration: report what the ring fit knows about its own answer ... The ring fit reported the SCATTER of its measurements (rms, and the beam-centre standard error that follows from it) but nothing about how well the fit pinned each parameter. Those two part company exactly where a calibration is worth doubting: as the rings run out, the tilt and the beam centre stop being separable - both displace a ring's radius as cos(phi) and only the way that amplitude scales with radius tells them apart - so the fit can sit tightly on the few points it has while being free to spend tens of pixels of beam centre on a tilt the data do not support. Take the covariance of the converged problem from Ceres and report it. Measured on a LaB6 distance series, the fitted tilt is 50 sigma at 110 mm and 0.1 sigma at 500 mm, where only two rings reach the detector; at 500 mm the fit quotes its own beam centre to +-180 px and its tilt to +-2.9 deg on a 0.35 deg value, and the correlation between them is 1.000. Nothing acts on this yet - it is printed so the next change can gate on it. Ceres returns the bare (J'J)^-1 of an unweighted problem, so it is scaled by chi2 per degree of freedom; that leaves the sigmas in pixels, mm and radians whatever unit the residual is stated in. Cost is one 5x5 SVD per run, below the noise of the surrounding I/O. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:11:18 +02:00
leonarski_fandClaude Opus 5	f34d9fe62b	calibration: move the powder fit into image_analysis/geom_refinement ... CalibrateFromProfile/CalibrateFromSpots/WritePoniFile sat in rugnux/, so the only way to reach them was to link the Rugnux library - which drags in JFJochWriter and gemmi. Nothing in them needs either: the includes are all common/ and image_analysis/geom_refinement/, next to the RingOptimizer and RingsFromProfile they call. Moving them to image_analysis/geom_refinement/PowderCalibration.{h,cpp} puts the powder fit beside the rest of the geometry refinement and makes it reachable from anything that already links JFJochImageAnalysis - the receiver and so the broker included, which is what an online geometry calibration would need. Pure move: the file contents differ from their previous form only in the include paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:09:42 +02:00
leonarski_fandClaude Opus 5	ba24946c0f	twinning: do not report a twin the L-test rules out ... The two indicators can only move one way under twinning - <|L|> down from 0.500 toward 0.375, the second moment down from 2.0 toward 1.5 - but they were combined with an OR, so a narrow intensity distribution could report a twin on its own while the L-test said the opposite. That is not a marginal disagreement. It fires on small-cell rotation data, where a twin fraction of 0.50 was reported for a crystal whose <|L|> was 0.63: a value no twin can produce, and one that points at the opposite situation - a structure whose intensities behave centric. Suppress the flag when <|L|> sits at or above its untwinned value; the two indicators are otherwise combined exactly as before. Measured over 113 stored reports: two flip, both of them wrong today, and nothing else moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:34:16 +02:00
leonarski_fandClaude Opus 5	71cca91ec1	rugnux: rank screw candidates without the control count ... A screw's control class is the complement of its absent class on one axial row, so the two move together: a candidate that predicts more of the row absent leaves fewer reflections to be judged against. The Beta tail AbsenceEvidence computes grows with that control count, so the stricter candidate was charged for the very reflections it correctly called extinct, and a group whose predicted-absent class is a strict superset of another's - with the extra reflections equally dead - could score LOWER than the group that explains only part of the row. Two mechanisms, one dataset each. * The control-count minimum gated the row MEAN, which is the scale the zone's evidence is stated in, on the same count as the row MEDIAN, which is a violation threshold. A candidate could therefore forfeit a whole zone by being right: on a tetragonal 4_1/4_3 wedge, thirteen 00l reflections measured at 0.1% of the two l = 4n beside them scored ZERO because only two control reflections were left, while the nine of them a 4_2 also predicts absent scored 38.7. The mean now stands on two, the median still on three. * The Beta tail is replaced, for a SCREW zone only, by its b -> infinity limit - the same statistic with the control count dropped. A p-value computed against each candidate's own null is not one scale across candidates; what is left is the likelihood ratio of the absent class against Wilson at the row's own mean, which is a sum over reflections and therefore comparable. Asymptotically it is n_absent * (log(1/ubar) - 1), so an equally dead superset can no longer score lower. Measured on a tetragonal 4_1/4_3 crystal: 29 dead 00l against 8 control read 47.7 nats where a subset of 19 of them against 18 control read 55.5. The centring statistic is untouched: its control is the whole present population, not the complement of a claim on one row, so neither mechanism applies to it. Measured. Analytic superset monotonicity, on a 176368-point grid with no calibration and no corpus: 1.29% violations -> 0.00%, exactly monotone wherever both candidates keep a control class. Over 112 stored merges (open, in-house and private arms) exactly ONE decision changes and it is a gain, a 4_2 -> 4_1/4_3 swap on a deposited 4_3; the screw column over the deposited arm goes 33/41 -> 34/41 with zero screw over-calls before and after, and the four centring outcomes are bit-identical. An eight-wedge reproducer whose 00l row a strict candidate eats goes from two mis-calls to none, end to end. The refused side moves the safe way: 103 zones the old statistic scored negative go lower, none turns positive, and the minimum evidence of an adopted screw rises 20.4 -> 23.8 against an unchanged bound of 20. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:33:29 +02:00
leonarski_fandClaude Opus 5	c25418319a	report: the twinning measured before the search, and the strong-direction limit ... Three additions to the report, none of which changes a number the run computes. The twinning statistics printed today are measured in the Laue class the run ADOPTED, so where the search promoted the point group they can say no more than "no twin law exists inside the class it chose" - and a twin is precisely what would have caused that promotion. The uncontaminated pair was already computed on the subgroup merge the search was given and reached only a log line; it is now printed as L_TEST_MEAN_ABS_L_BEFORE_SEARCH and SECOND_MOMENT_I_BEFORE_SEARCH, beside the post-adoption pair rather than instead of it, since the two answer different questions. ANISOTROPY_D_MIN_BEST names the finest of the three principal limits so it can be grepped, and a paragraph distinguishes the three resolutions the report now carries: what was written, where the isotropic fit crossed, and how far the crystal reaches where it reaches furthest. It is read off the fitted tensor rather than off the cut, so it can land either side of the other two - measured on one dataset at 2.01 A against a 1.81 A cut - and the text points at ANISOTROPY_D_MIN_CENSORED for when it is the edge of the data instead. A warning when CC1/2 falls and then climbs again, which means the fall-off the cut is read off does not describe the data. A climb counts only above CC1/2 0.30 and only if it exceeds 0.05: past that the number is oscillating in noise beyond any cut this run would take. It advises no re-cut - the generous cut is deliberate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 14:06:42 +02:00
leonarski_fandClaude Opus 5	6491345fea	comment: say what the refined geometry in the end message actually is ... Two things the field names hide, both verified in code and both able to mislead a reader into publishing a wrong geometry. The beam centre written here is the PONI - the foot of the perpendicular from the sample to the detector plane - so on a tilted detector it is not where the direct beam lands. That point is D*tan(rot)/pixel away and is GetDirectBeam_pxl(). And the tilt is not written only for the record, which the old comment implied by mentioning nothing but the _process.h5: IndexAndRefine puts it on outcome.experiment, which is the geometry Bragg prediction and integration then run at. It is fitted with the distance frozen and nothing checks the covariance, so the split between the PONI and the tilt is not reproducible between passes even though their sum - the direct beam - is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 14:06:03 +02:00
leonarski_fandClaude Opus 5	d189faeec7	viewer: draw grid scan cells in the proportion of the scan steps ... A grid scan with, say, a 20 um step in x and a 5 um step in y was drawn as a square grid, so the picture had nothing to do with the shape of the area that was scanned. The cells now carry that proportion. The anisotropy lives only in the view transform: JFJochImage gains a pixel_aspect_ (the drawn height of one pixel in units of its width, 1.0 for a detector image), the initial fit fits the image as if it were that much taller and then puts the factor back into the transform, and the grid view sets it from |step_y / step_x|. Scene coordinates stay one unit per cell, so the mouse mapping, the selected-image box, the pixel labels and the ROI code need no change and the uniform wheel zoom keeps the proportion. Measured on an 8x4 grid driven offscreen: m11/m22 comes out 4.000 for steps 20/5 um, 0.250 for 5/20, 12.333 for 37/3 and exactly 1.000 - the old behaviour - for a square step. Every cell centre still maps back through mapToScene to its own cell in all four cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hcoh6VNrmSswjfeMeDQqWP	2026-08-31 13:38:51 +02:00
leonarski_fandClaude Opus 5	756c3a22bb	docs: confirmed cells for three small-molecule sets; drop the unprocessable CCD one ... The four Diamond I19 small-molecule datasets had no reference of any kind, so a run on them could not be scored at all. Three now have one, each from a published structure with its DOI verified against Crossref; the fourth has a published space group but no numeric cell anywhere, which is recorded as such. Also removes the "obtained but not in the table" section. That dataset is stored as CCD TIFFs the reader does not support, so it is not processed here and adds nothing to a page about what the test corpus contains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 11:02:38 +02:00
leonarski_fandClaude Opus 5	6d37b31948	report: give the resolution the CC1/2 fit reached, not only the one it was cut at ... The automatic cutoff fits the CC1/2 fall-off, takes the crossing of the target (0.30 by default) and then deliberately keeps one more shell. Only the extended limit was reported, as INCLUDE_RESOLUTION_RANGE, and that is the number a reader takes for "the resolution of this dataset" - so the report quoted a limit that is generous on purpose as if it were the measurement. The generosity itself is right and stays: a shell that is included can still be downweighted or dropped by refinement, while one that was truncated cannot be put back. What was missing is the other number. ComputeCCHalfLogisticCutoff now also returns the crossing, ApplyResolutionCutoff passes it out, and the report prints it as FITTED_RESOLUTION beside the range the reflections were written to, with a sentence saying which is which. The log line names both as well. Nothing about the cut, the merge or the written reflections changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 11:02:37 +02:00
leonarski_fandClaude Opus 5	55d9e2536f	release: 1.0.0-rc.166 ... The changelog entry for this release is six lines, not the fifty-four the development log had accumulated: what a user of rc.166 gets is native miniCBF input, masters from other facilities opening, the beam centre measured on every run, a 2theta-swung detector placed where the file says, the unmerged MTZ and a P1 merge written by default, and a substantially reworked symmetry determination. The per-change detail is in the commits. update_version.sh regenerates the three clients from broker/jfjoch_api.yaml, so this also carries the one API description that had drifted from the generated code - fft_high_resolution_A, which sizes the FFT's projection histogram and does not filter spots by resolution. Everything else in the regenerated tree is the version string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 09:11:22 +02:00
leonarski_fandClaude Opus 5	3a30dc8ab5	rugnux: a space group that names twice the cell is a signal to halve it ... C 1 1 2 and C 1 1 2_1 are the only two monoclinic settings whose change of basis from the reference has determinant 1/2, and the reason is physical: with unique axis c the C-centring vector lies wholly in the oblique a-b plane and reduces away. Such a group is never a distinct Bravais centring - it is a primitive lattice written on twice the cell - and it wins the candidate table only because the table is scored on that doubled cell, where half the reflection positions are absent by construction. Refusing to enumerate those settings was measured and is worse: the volume-preserving alternative keeps the by-construction absences as real observations and merges no better than P1. So keep the enumeration as it is and act on the win instead: halve the cell, re-classify in the group's own system, carry the reflections across (the invented positions map to non-integral indices and are dropped, which is what the centred candidate was already doing) and search again there. Measured on one rotation crystal: C 1 1 2_1 on 68.6 73.0 170.3 90 90 90 becomes P 1 2_1 1 on 50.1 170.3 50.1 90 93.5 90, exactly the deposited setting, with every merge statistic unchanged (R_meas 0.1209, CC1/2 0.9979, d_min 1.238) and ISa 18.27 -> 18.31. Stable across +/-0.5 mm of detector distance. Eight other crystals are bit-identical, and across 112 stored battery reports exactly one adopts a volume-doubling setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:19:38 +02:00
leonarski_fandClaude Opus 5	5e3a7d113c	rugnux: keep a rotation-axis sign the first pass corrected ... The two-pass post-refinement snapshots the goniometer before pass 1 and restores it after, to undo the start/stride shift a sub-range run applies inside the pass. But the rotation-axis sign rescue also runs inside the pass, and adopts the opposite axis when the file's own indexes almost nothing - so the restore threw that away, handed pass 2 the sign that had already failed, and applied pass 1's fitted rotation scale to the axis it was not fitted on. It self-heals: pass 2's rescue fires again, and a genuinely wrong sign indexes almost no frames, so the run reaches the same answer having paid for one redundant first pass. But the correctness of that rests entirely on "a wrong sign always scores below half the validation frames", which nothing states and nothing tests. The restore now keeps the axis direction and reverts only the angles, which is what it was for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:41 +02:00
leonarski_fandClaude Opus 5	7c3ab72fed	reader: refuse a CBF that is not a detector image instead of misreading it ... Three ways a miniCBF opened silently wrong. A byte-offset CBF with no PILATUS header at all was accepted: Count_cutoff then defaulted to 0, SaturationLimitFromValue(0) is 1, and every pixel at or above one count was flagged saturated - the integration accept gate drops the whole reflection, so the run comes out empty for a reason nothing reports. The pixel size defaulted to 0 with no validation anywhere downstream, which collapses every resolution, every scattering vector and the beam centre in millimetres. XDS writes its correction files in exactly this shape, so this is not hypothetical. A pixel size is now required to claim the file at all, and a missing Count_cutoff leaves the saturation limit unset - falling back to the container's own overflow, which can only fail to call a pixel saturated - with a warning saying so. And the header captures are character classes, not number grammars: "[\d.eE+-]+" matches a bare "." and "(\d+)" matches a digit string too long for int64. std::stod and std::stoll answer both with a raw std:: exception, which escaped the format probe - CanRead catches JFJochException only - so merely LOOKING at a corrupt file threw out of the viewer's open path. A header field that does not parse is now reported as a malformed header. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:40 +02:00
leonarski_fandClaude Opus 5	8d719c1383	symmetry: a zone whose absences were never measurable must not outrank a dead one ... sum_u is a sum of max(0, E^2) over a zone's predicted-absent reflections, in units of their control class's mean, so it is EXACTLY zero when every one of them merged non-positive. The Beta tail then diverges and the clamp that caught it made each such reflection worth about 690 nats. That was harmless while the value only had to clear a bound of 20. Since the absence evidence became the primary sort key, summed across zones with no minimum count per zone, it has ranked the candidates - and it ranked them backwards. A zone of two absences that were never measurable scores 1378 nats where a genuine zone of six absences at 1% of its own row scores 22, so the candidate claiming a screw on an UNMEASURED row beat the one whose rows are actually dead, by sixty times. The comment on the ranking claimed an extra condition "LOSES the zone's evidence when the row is not dead"; an unmeasurable row could not lose, it won outright. No measurement places a merged intensity at exactly zero, so sum_u is floored at a thousandth of the control mean per absent reflection. The constant is an order of magnitude below the precision any real merge reaches - a thousandth of the row mean needs I/sigma ~ 1000 against that row, where ISa tops out near 40 - so it can only remove the singularity, never suppress evidence a measurement could have produced. Measured across the realistic range it changes no genuine zone at all (22.0, 34.7 and 65.4 nats, unchanged to four figures) and takes the unmeasurable ones to 13 and 36, below and around the bound respectively. AbsenceEvidence moves to the header, where the comments already named it, so the ordering it has to satisfy can be tested directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:26 +02:00
leonarski_f	e0ff51e2a0	lattice: transpose the change of basis to a primitive cell, as everywhere else ... gemmi states centred_to_primitive as an operator on COORDINATES and CrystalLattice::Multiply combines BASIS VECTORS, so the matrix has to be transposed on the way in - as it already is at both of the other places a gemmi Op::Rot reaches Multiply, one of them in this same file. A, B, C, I and F are symmetric matrices, so for them the transpose is a no-op and the omission never showed. R and H are not. Measured on an R-centred hexagonal lattice, ToPrimitive('R') returned 59.5 81.7 43.3 / 145.6 124.5 46.7 where the rhombohedral primitive cell is 49.3 49.3 49.3 / 60.9 60.9 60.9. What hid it is that a determinant is unchanged by transposition, so the VOLUME came out right - and most callers only take the volume. It is not only cosmetic: the result feeds the re-seating path that puts a lattice into a space group the user fixed by hand, so an R-centred lattice was handed the classifier a "primitive" cell that is not that lattice - broken for exactly the centring whose setting most needs re-seating. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 6ca00e927d664e870515c04164defa81d8a18725)	2026-08-31 07:17:00 +02:00
leonarski_f	527a5187f4	lattice: restore the character that names a centred monoclinic mI reduced form ... ITA character 43 was absent from the table. It is the type-II reduced form of a centred monoclinic lattice with no length equality - mC, mI, mA and mF are one Bravais lattice in four settings, and this row names the one whose conventional cell comes out I-centred. With the row missing such a cell reaches character 44 and is reported as triclinic, losing its centring outright. It accounted for 30 of the 31 demotions left after the two fixes before this one, and for 4.1% of random centred-monoclinic lattices. Both of its conditions are equalities on scalar products - International Tables gives them as 2|D+E+F| = A+B and |2D+F| = B - so the three angle tests are vacuous for it and the second equality is what selects it. cond_2DF was declared in the character struct and never tested against anything, because until now no row used it. Audited over 14000 exact lattices, the row fires 35 times: 32 are centred monoclinic lattices it recovers correctly and 3 are triclinic cells it promotes, all three of which Le Page promotes at the same tolerance. No over-call is attributable to it. The row is taken from International Tables rather than derived - no single integer matrix of determinant 2 covers more than 68% of these cells - and the test checks the conventional cell it produces has two right angles and twice the primitive volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 4c2baf0cc3b2177196a9448df06e24b5446de96a)	2026-08-31 07:17:00 +02:00
leonarski_f	931d7acc7c	lattice: judge a structurally-zero scalar product against the size of the cell ... The Niggli type of a reduced cell is the sign of its three scalar products, and gemmi's reduction was asked to decide those signs against an absolute tolerance of 1e-9 while the products themselves are 10^3 to 10^5 A^2 and the cell is held in float. A product that is structurally zero therefore arrives carrying about 1e-4 A^2 of rounding and is read as definitely signed. The reduction lands on the wrong side of the type-I/type-II boundary, the character written for the other side matches nothing, and the lattice comes back with less symmetry than it has. It is not a corner case. Take an exactly body-centred tetragonal lattice with c > a*sqrt(2) and merely ROTATE IT IN SPACE: 38 of 60 rotations lose the 4-fold, and it comes back C-centred. Every lattice this code classifies is a refined, rotated one; the two tetragonal-I cases already in the tests are axis-aligned, which is the one corner where the rounding vanishes. The tolerance is now scaled by the cell's own magnitude. Over 14845 exact lattices that puts it on a plateau three decades wide - demotions 461 -> 31 - with the over-call count unchanged at every point of it, and with the three fixes that follow the plateau is four decades wide and the demotions are zero. The constant is 100x the one Grosse-Kunstleve et al. give because theirs is calibrated for a double-precision cell and this one is float: measured, their value recovers 6% of these lattices and this one 93%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit b82f217719450e1e46e94be5b5a43ed8ffbf2ca4)	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	b31fb0a55c	symmetry: take the point group from whichever merge found it, the absences from all observations ... The search runs on two merges - one holding every observation, one holding only the well-measured ones - and the second was allowed to report but never to decide. That cost real symmetry: over the corpus the two arms disagree on eight crystals and the arm that found the LARGER point group is right on all eight. A promotion is refused by the most damning statistic it can be shown, so only one of the two arms has to be starved for a genuine operator to be thrown away, and which arm that is depends on the crystal. But the two halves of the answer do not come from the same place. Absences live in the weak reflections the filter discards, so adopting the filtered arm whole recovers the point group and loses the screw - a P6_5 crystal comes back as P6. So the arms are split rather than ranked: the point group is taken from whichever supports the larger one, and its absences are then judged here, on every observation. SearchSpaceGroupOptions::fixed_point_group was written for exactly this and had no caller. Measured on the deposited arm: a trigonal-3 answer becomes P 6_1 against a deposited P 6_5 (an enantiomorph pair, so correct), and a monoclinic answer becomes P 2 2 2 against a deposited P 2_1 2_1 2. The screw survives the split, which it did not when the filtered arm was adopted whole. No crystal that was already right moved, and the over-call column is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	d35e8f680f	symmetry: say when the cell metric hosts more symmetry than the group adopted ... Every symmetry under-call in the corpus has the same signature: a lattice whose metric carries rotations the adopted group does not, with nothing in the run saying so. The user is left to notice that a P1 answer sits on a cell whose axes are equal and whose angles are 60 degrees. So a run that determines its own group now compares the two and says what it sees. It decides nothing - no threshold, no promotion, no demotion, no reprocessing - and the message says as much, because a pseudo-symmetric metric is ordinary and only the intensities can settle whether the extra rotations are real. Measured over 83 crystals it fires on 19 and covers 6 of the 7 point-group under-calls; on the crystal whose deposited group is F 4 3 2 and which rugnux reports as P 1, the line reads "the cell metric is cubic - it admits 24 rotations where P 1 has 1". The metric symmetry is Le Page's, taken through GEMMI's implementation, at an obliquity of 1 degree - the middle of the band over which the answer is stable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	493be2cf19	symmetry: enumerate the settings the cell can host, not only the reference ones ... The space-group search offered a candidate only if gemmi calls it the reference setting. A setting is a statement about direction, so that restricted the search to the axes the convention chose: a crystal whose 2-fold lies on c had no rung between P1 and 222 and fell to P1, and one whose screws lie on b and c was reported as the group with a single screw on c - the wrong group, not a lower one, because the candidate that predicts a subset of the real absences and nothing else wins on no evidence at all. Both stages now enumerate more, under refusals rather than thresholds. Stage B offers the non-reference settings of the chosen point group. Their rotation set is equal to the chosen one, not merely contained in it, so this cannot raise the symmetry; what it adds is a screw or a centring on the axis the data show it on. A candidate is offered only if the cell's own metric admits the rotations its setting names, and one predicting exactly the absences another candidate already predicts is dropped as the same hypothesis under a second name. A non-reference candidate whose centring class this merge does not contain is refused outright: the reference path may adopt an untested centring because the caller's centred-lattice re-test backs it, and a non-reference setting has no such backing. Stage A offers the rotation sets no reference setting carries - the a-unique and c-unique monoclinic 2-folds, and the two rhombohedral-axes trigonal groups - and only those, so every point group reachable before is still reached by the same group in the same setting. A rung reached only that way may be ADOPTED but does not judge anything else: it is skipped when the reference chi^2 is formed and when a higher promotion's parents are collected. Without that it made higher promotions strictly harder - a promotion answers to the most damning of its parents, and offering two more order-2 subgroups of 222 refused 222 and 432 on crystals that had them. The run report gains SPACE_GROUP_NAME beside SPACE_GROUP_NUMBER, since the number alone does not say which axes a group's symmetry lies on, and a group that is not a reference setting is printed as its extended Hermann-Mauguin name. The two-arm reconciliation compares point groups by that name rather than by number for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_f	6ed4ea541e	reader: a link to a file that is not there is not a dataset that exists ... Every DECTRIS Eiger master links saturation_value, pixel_mask, bit_depth_readout and serial_number into a companion <prefix>_meta.h5, and that file is routinely not kept when a dataset is archived or deposited. The existence test asked only whether the LINK was written, which it is, so every optional-field guard in the reader answered yes and the read that followed threw. A deposited Eiger 16M set could not be opened at all, over values the reader was perfectly prepared to do without. Exists() now asks the second question too - whether the object the link names can be reached - so an orphaned link reads as absent and the fallbacks behind it do their job. The saturation value is then allowed to be missing outright, because on such a file it is: neither the NXmx name nor the DECTRIS one is readable, and there is no third place to look. Left unset, GetSaturationLimit() falls back to the container's own overflow. That is the safe direction - it can only fail to call a pixel saturated, where too LOW a value drops the whole reflection and silently removes the strongest data - and the run says out loud that nothing will be called saturated. The set that could not be opened now processes to 2.17 A against a deposited 2.40 A, in the deposited space group, with a cell agreeing to 0.08%. Output is byte-identical on datasets that already opened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 0637979f6d95b446406ab70d1f3982841d195b36)	2026-08-30 21:43:54 +02:00
leonarski_fandClaude Opus 5	fb18457e6f	docs: bring the changelog and the method notes up to rc.166 ... The rc.166 changelog was missing fourteen user-visible changes and carried rationale and measurements that belong here instead. Added: the native miniCBF sweep reader, the third-party and firmware-1.x NXmx masters, the plain LZ4 filter, the image orientation taken from the file's module direction vectors, the two beam-centre flags and their rescues, and the FFT reach past 500 A. Trimmed the rest to one line each, moving the numbers out of the user-facing file. RUGNUX.md described the input as a single Jungfraujoch master file, which it has not been since this branch; it now covers the foreign and legacy masters, the accepted compression filters and the miniCBF sweep, including how a sweep is collected from one named frame. Six options existed with no entry in the table - --beam-center-check, --beam-center-search, --fft-min-unit-cell, --min-indexed-spots, --rot3 and --no-p1-crosscheck - and -C now moves both FFT cell bounds, which was not written down anywhere. CPU_DATA_ANALYSIS.md carried two statements this branch made false: 7.5 still said pass 2 reuses pass 1's space group, and 13.1 still said centrings are ranked by net absence count. Both now describe what the code does - the group is determined after pass 2, and centrings are ranked by the same Beta-tail likelihood the screw test uses. Also documents the per-zone screw scoring, the coplanarity volume-fraction guard, the plane-normal transform, the FFT cell bounds and the twelve refined candidates. The miniCBF reader implements the x-CBF_BYTE_OFFSET scheme and reads the imgCIF axis table from the specification alone. No CBF code is vendored or linked, so there is no licence obligation, but reimplementing a published specification carries one of credit: ACKNOWLEDGEMENT.md gains a section and the two algorithms carry a one-line reference each. Both DOIs were resolved before being written. Rottger's initial was wrong where this branch first cited it - K, not A. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 21:15:31 +02:00
leonarski_fandClaude Opus 5	9924dd9fc3	docs: the open test battery, updated - 82 datasets, and not only non-SLS ... The public data the pipeline is exercised on has grown from 59 datasets to 82, 77 of them with a released PDB entry and released structure factors, so the page that credits the depositors and carries the DOI to cite for each is brought up to date with what is actually run. Renamed from NON_SLS_TEST_DATA to EXTERNAL_TEST_DATA, because the old title stopped being true: a few of the sets were collected at SLS beamlines, where the data are still written by someone else's detector and someone else's acquisition system. What the battery tests is foreign files, not a foreign facility. Also rewritten from the current archives rather than the earlier sample: - Multi-collection archives: eleven are not a single continuous rotation, not four. Seven IRRMC archives hold more than one collection; one sweep is kept in six of them, and both are kept in the one whose two sweeps are at different wavelengths. A repository project page is not a reliable guide here - one describes a 900-frame sweep its own tarball does not contain. - Detector labels: 76 rows can be compared against the PDB entry. Seven genuinely conflict, and one of those the file settles outright - pixel count, pixel size, sensor thickness and firmware string agree with an EIGER2 9M against the entry's PILATUS4 4M. A further 29 differ only in how much they state, which is not a conflict. - One archive ships 30 placeholder files named like images that are 64-byte text; named on the page so a reader that globs the directory is not surprised by them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:50:10 +02:00
leonarski_fandClaude Opus 5	03456dcddd	calibration: an option to hold the detector tilt fixed ... --mode calibration fits five parameters - beam centre, distance and the two PONI tilts - and a program that cannot express a tilted detector has nowhere to put the last two. Dropping them after the fact is worse than never fitting them: the centre and the distance of a tilted fit have already absorbed the tilt, so the flattened geometry is right nowhere. rugnux --no-refine-tilt, the "Refine detector tilt" tick box on the viewer's Calib page and RingOptimizer's refine_tilt argument hold rot1/rot2 at the value the geometry came in with and fit the remaining three. That is the best flat-detector answer, and the one such a program would refine to itself. Measured on a five-distance calibrant series. At short distance the tilt is real and reproducible - three independent fits agreeing to 0.01 deg, radial rms 1.4 -> 0.4 px - and its direct beam agrees with the background beam-centre estimator to 0.05 px, so the tilted model is the physically right one. The pinned fit then displaces the centre 2.6 px to absorb the tilt and lands within 0.03 px of the same place at every distance. Past ~300 mm, where only two rings reach the detector, the tilt is instead under-determined: it comes out with the opposite sign to every short-distance fit and drags the PONI 28 px while the rms does not move (0.960 against 0.962). The existing degeneracy guard only fires on a strictly single ring, so it does not catch that; declining a tilt that does not pay for itself in rms is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:44:12 +02:00
leonarski_f	d5818713cb	rugnux: also merge in P1, so a wrong space group is recoverable ... A rotation run that determines its own space group now also writes <prefix>_P1.mtz: the same observations, the same scaling, merged in P1. If the group was wrong there is no route back today except processing the images again, and the P1 data settle it - re-merge, re-solve or re-refine in any subgroup. Written whether or not the search adopted P1, because a file whose presence depends on what the pipeline decided cannot be harvested by a script. A run given -S writes nothing: its group's centring absences were never integrated, so the P1 merge would be missing whole classes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md # rugnux/rugnux_cli.cpp	2026-08-30 14:15:20 +02:00
leonarski_f	123dc22be2	rugnux: write the unmerged MTZ by default ... A run that does not ask for it still has to hand its data to aimless, pointless or careless eventually, and the file it needs is one a user had to know a flag to get. It is the largest a run produces, but it is 3.4 to 5.8 per cent of the images it replaces - and the better it is, the less anyone needs to keep those. The batteries pass --no-export-unmerged: they process the whole corpus and would write a few gigabytes nobody reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f # Conflicts: # docs/CHANGELOG.md	2026-08-30 14:14:28 +02:00
leonarski_fandClaude Opus 5	f75cf72d1a	beam centre: measure it on every run, and try it when the header fails ... The header beam centre is wrong by more than the geometry absorbs on two thirds of foreign depositions, and nothing measured it. The solvent ring already gives it away: the background projection the beam-stop pre-scan builds is enough to fit the centre, so the measurement is a fit over an array the run has already paid for. It is reported on every run and committed on none. The first pass runs again at the measured centre and the two lattices are compared; the measured centre is taken only where the header indexes nothing, and a disagreement is reported rather than resolved, because the only arbiter available at that stage is the frame count and it is inverted. Recovers two depositions whose header is the geometric detector centre. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:07:13 +02:00
leonarski_fandClaude Opus 5	65dfdb1324	reader: read the detector arm and the mounting each file states ... A detector swung out on a 2theta arm was read as if it stood square on, and a miniCBF written with a vertical spindle or a quarter-turned image was read with the standard mounting assumed. Both are stated in the file and both were ignored: the NXmx depends_on chain was never followed, and Detector_2theta was parsed into a field nothing read. Recovers three datasets that produced no usable lattice, and the small-molecule sweeps at 30 and 55 degrees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:07:04 +02:00
leonarski_fandClaude Opus 5	7528bd8761	docs: write down the three directions this work is heading in ... Three preferences that have been decided in conversation and applied to several changes already, but were nowhere a new reader would find them. They are directional, not absolute. Spending compute to buy quality is now affordable where it once was not, so the question to ask of a slower design is what it buys. Deciding late beats deciding at a threshold, because a gate calibrated on one population refuses another - measured repeatedly in the lattice and symmetry code. And the target is a bare invocation with no flags, because a capability nobody knows to switch on has not solved the problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:06:54 +02:00
leonarski_fandClaude Opus 5	a27c4cf26f	reader: take a miniCBF's mounting from the imgCIF axis table its header states ... A miniCBF header states three things about how the instrument is put together that the reader was assuming instead: which laboratory direction the image's columns run along, which its rows run along, and which the spindle turns about. Some beamlines append a CBF template block holding the full imgCIF axis table, which says all three outright. Two instruments in the corpus are not what was assumed, in two different ways. One mounts its detector a quarter turn round, so the image's columns run vertically. Another turns its spindle about the VERTICAL, with the image mounted the usual way; its table says so, and its "# Oscillation_axis" line says so a second way, by naming the image direction the spindle runs along rather than a vector. Either error leaves the spindle 90 degrees from the image. That is not a sign, so the run's axis-sign rescue cannot reach it, and no refinement recovers it: all three affected sweeps indexed nothing usable. So the table is read. The element axes give the image orientation, matched against the eight discrete mountings exactly as the NXmx module directions already are - the match itself moves to DetectorOrientation, so both readers share one definition rather than two copies. The goniometer axis with no parent gives the spindle DIRECTION; its sign stays the rescue's business, which is the part a convention can legitimately differ on. The detector axis with no parent gives the 2theta arm, replacing the assumption that the arm shares the spindle's axis - the one header stating both states them with the same vector, so this changes no answer, only what it rests on. imgCIF's frame differs from the internal one by a half turn about x, a rotation and not a mirror, as writer/HDF5NXmx.cpp already records from the other side. Where a header carries no table, a "+SLOW" on the Oscillation_axis line still says the spindle runs along the image's slow direction. That is the only thing one of the three affected sets says about it. The axis NAME on that line stays unusable - the header that carries both says "X.CW" where its own table says Y - but the direction token is not: where both are present they agree, which is what makes reading it evidence rather than a guess. Also: naming a frame with no directory at all now finds its sweep. parent_path() of a bare filename is empty and iterating an empty path finds nothing, so running from inside the data directory reported that no images were found. Measured, with nothing on the command line. The vertical-spindle protein set goes from no usable lattice to 100% indexed, P 6(3) 2 2 with a cell 0.43% from deposited, 87846 reflections at 86.3% completeness and CC(1/2) 0.995. Its companion from the same detector, which has no table and only the +SLOW token, goes from a spurious monoclinic cell at 2.3% completeness and I/sigma 0.21 to the right orthorhombic lattice, 97.7% indexed, 59.7% complete, CC(1/2) 0.996. The quarter-turned set's three sweeps, at three arm positions, now all index without the hand-passed quarter turn they needed and agree on one cell to 0.03 A. Six miniCBF sets that state no table and no +SLOW - including one whose Oscillation_axis line names an axis in a third dialect - are byte-identical in .hkl, .mtz, .cif and the image statistics, as are two NXmx sets, which is the shared orientation matcher moving nothing on that path either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 14:03:32 +02:00
leonarski_fandClaude Opus 5	dc71cb7299	rugnux: keep the P1 merge, so a wrong space group is recoverable ... A de-novo rotation run merges in P1 to search for the symmetry, adopts a group, and then overwrites that merge with the in-symmetry one. If the adopted group is wrong the user has no route back: every file the run wrote, and every statistic in them, is computed in the group that was assumed, so nothing in the output says the choice was wrong and the only way to a different answer is to process the images again. Merge the same integration once more in P1 after the run's own files are written, and put it beside them as <prefix>_P1.mtz. From it the space group can be re-determined and the data re-merged, re-solved or re-refined in any subgroup - measured end to end on three crystals: POINTLESS reads the file, recovers the group, and AIMLESS re-merges it, reproducing rugnux's own merged intensities at CC 0.9965 where the two agree. On one of the three it recovered the deposited/XDS group where this run had under-called the screw axis. The merge is the full one - correction surfaces fitted, ice rings and near-tangential observations kept, whole resolution range - not the deliberately degraded merge the space-group search itself runs on, and it is what `rugnux --mode scale -S P1` produces from a _process.h5. That route already existed but needs a _process.h5, which a merging run does not write, so it only helped a user who had foreseen the problem. Every de-novo rotation run writes the file, including one whose search concluded P1 and where it therefore repeats the merged output byte for byte. Whether a file exists must not depend on what the pipeline decided: a script harvesting results would otherwise have to reproduce the search's decision to know whether to expect it, and a missing file would not separate "the run chose P1" from "the run failed". A user-fixed -S writes nothing, and that condition is not a pipeline decision. With a group fixed, prediction rejects that group's centring absences (IndexAndRefine.cpp:499-506), so those reflections are never integrated; a P1 merge built from such a run would be missing whole centring classes and would mislead rather than merely be smaller. Cost, median of five paired runs read off the log timestamps (the box is shared, so end-to-end wall time is noise): +0.32 s of 4.2 s, +2.18 s of 31.4 s, +0.19 s of 17.9 s, +0.66 s of 15.9 s - 1 to 8% of a run. The file is 1.6x the merged MTZ on a monoclinic crystal and 30x on a cubic one, where the merged MTZ is tiny; it is well under the unmerged export in every case measured, and 0.02 to 0.6% of the raw dataset. Rotation only for now: the stills merge re-fits per-image scales and per-reflection partialities onto the integration outcomes, and _unmerged.mtz is written from those afterwards, so on stills this extra merge alters a file that is the run's own output. Fixing that means running the cross-check below the unmerged export, which needs the merge lambda hoisted out of its block; deferred, since rotation is what the online pipeline processes. On rotation the merged .mtz, .cif, .hkl, _image.dat, _unmerged.mtz and _unmerged_partials.mtz are all byte-identical with and without this change, including on a run whose search returns P1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:49 +02:00
leonarski_fandClaude Opus 5	5c44e544dc	reader: place a detector swung out on a 2theta arm where the file says it stands ... Chemical crystallography reaches high angle by swinging the detector out on a 2theta arm. Both readers had the number and neither used it: the miniCBF header's Detector_2theta was parsed into a struct member nothing ever read, and on the NXmx side the rotation was in the depends_on chain, which was not followed at all. A sweep taken at 30 degrees was therefore processed with its detector plane 30 degrees from where it stood, and nothing indexed. The geometry could already express it, and needed no change: the arm turns the detector about the sample, so the distance is still measured along the detector normal and the beam centre is still the point of normal incidence - which is exactly the PONI convention, and a swung detector is one PONI rotation. What moves is the direct beam, by distance*tan(2theta), off the beam centre and often off the detector. NXmx is the harder half, because the swing has no field of its own: it is one rotation in the chain the detector's position depends on, and "two_theta" is only one beamline's name for that dataset. So the chain is followed and its rotations composed, rather than a field of one name being looked for - each transformation states its vector in the frame of the one it depends on, which is why the product is the whole placement. Translations are skipped; they are the distance and the beam centre, which the file states separately in the square-on frame. Vectors come from McStas through the same 180-degree turn about z the module directions already use, a proper rotation, so an axis carried through it turns the same way. The three rotations a file this system writes ARE that chain, and are also read as the PONI angles - so those three paths are skipped, or every tilted file we have ever written would come back tilted twice. That is the one way this change could have broken existing data, and the test for it writes a tilted file and reads it back. For miniCBF the arm turns about the base spindle axis: on the four-circle geometry those headers describe the two are one axis, and the imgCIF axis table such a header carries states them with the same vector. Both now come from one constant, so a later correction to the frame moves them together. Measured. On a swung NXmx sweep the chain gives rot2 = -0.34907 rad for the 20 degrees it states, and the sweep goes from "nothing was integrated" to 25000 reflections at 82.2% completeness and CC(1/2) 0.9993, in the same space group and the same cell to 0.03 A as the square-on sweep of that crystal; the opposite sign indexes nothing. A miniCBF sweep at 30 degrees goes the same way, to 0.585 A, and a second sweep of that crystal at 55 degrees reaches 0.476 A and reproduces the cell again - with a low-resolution limit of 2.36 A rather than 13 A, which is what a detector swung that far records. On all of them post-refinement recovers the header's own beam centre and distance, and the beam stop shadow sits within four pixels of where the swung geometry puts the direct beam, 417 and 537 pixels from where the unswung one does. Seven sets whose detector is square to the beam, three of them carrying a chain whose 2theta is zero, are byte-identical in .hkl, .mtz, .cif and the image statistics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 12:36:46 +02:00
leonarski_fandClaude Opus 5	5f47d73cc1	rugnux: write the unmerged MTZ by default ... The unmerged export is how a run's observations reach the rest of the field. It is what aimless, pointless and careless read, it is what a head-to-head against another program's answer runs through, and handing several of these files to pointless is the only way to merge sweeps rugnux does not combine itself. Behind a flag it reached only the people who already knew the flag existed, which is the shape of defect the "rugnux <file> with nothing else" direction asks to design out. --no-export-unmerged turns it off, and the regression batteries now pass it: the objection to defaulting it on was their disk and time cost, not the product's. Measured on three rotation crystals of 900, 1800 and 3600 frames: the file is 21, 86 and 26 MB - two to five times the merged .mtz, .cif and .hkl put together - and the write costs 0.24, 2.29 and 1.11 s, 5.5 %, 12.2 % and 3.1 % of each run's own wall time. The cost is linear in the number of observations and independent of how long processing took, so it is the largest fraction of the runs rugnux finishes fastest, not of the longest sweeps. A run's .hkl, .mtz and .cif are byte-identical with the export on and off. --export-unmerged-partials stays off: it is a second, larger artefact and a separate question. The viewer's reprocessing dialog ties the export to its merged-output switch, so it appears beside the merged files rather than beside a job that only asked for the per-image _process.h5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 10:51:25 +02:00
leonarski_fandClaude Opus 5	bf41b1fab1	rugnux: index both beam centres always, without a significance gate first ... The second first pass was skipped where the measured centre sat within three times the fit's own sigma of the file's - a quarter of the non-SLS corpus, eight of thirty-two runs. That is a decision taken at a threshold before the evidence is in, and the evidence here costs a median 0.55 s: the spots are already found and cached, and their positions do not depend on the centre, so a second first pass is a median 20 % of what the first one costs. Worse, the gate removed exactly the case worth asking about. A centre error ALONG the spindle does not fail - it holds 96-100 % of frames indexed and quietly returns an axis harmonic - and the scale it turns on is a fraction of a pixel: measured on real data, 0.12 px of centre is the whole difference between the deposited cell and a halved axis. No sigma small enough to gate on would notice that, so the gate could only ever hide the question, never answer it. The three-sigma test stays where it belongs, in the report: it says whether the difference is a measurement or noise, which is what a reader wants to know. It no longer decides whether to look. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 10:28:35 +02:00
leonarski_fandClaude Opus 5	d8816169e1	rugnux: ask the rotation-axis sign at the measured centre, and say when only the symmetry differs ... Two corrections to the check, both from measuring it on the non-SLS corpus. Where the file's centre indexes nothing and the measured one does not either, the rotation-axis sign is now asked again AT the measured centre. The two unknowns are discrete and coupled: on the two public depositions in the corpus whose header beam centre is the geometric centre of the detector, the file is 73 px out AND its axis sign is the opposite of the one that indexes. The sign rescue therefore asks its question at a centre 73 px wrong (0/60) and puts the sign back, and the centre is then asked at the wrong sign (0/60). Each error hides the other and the run produces nothing. Asking the pair takes both runs to 100 % indexed on the deposited lattice - 199.40 67.11 against a deposited 199.54 67.15, and 208.77 67.20 against 208.77 67.22 - at the cost of one more first pass on a run that has already failed twice. Asked here rather than by moving the check ahead of the sign rescue, which was tried: that also works, but it moves the beam centre of three datasets whose only fault is the axis sign, for no benefit. Where nothing works the file's centre is put back, so a run that fails for another reason fails at the geometry it was given - 6yqf, whose spindle is along the detector's slow axis, and 7atg both come out exactly as they did before. And the beam centre decides the primitive VOLUME, not the Bravais class: a centre error along the spindle makes the FFT take an axis harmonic, which changes the volume by an integer factor or by sqrt(3), while a class differs for a reason that has nothing to do with the centre. The two are now reported separately. Both "disagreements" in the corpus are of the second kind - primitive volumes agreeing to 0.16 % and 0.57 % while one pass reads R-centred trigonal or P orthorhombic and the other stays triclinic - and in both the file's centre is the one that finds the symmetry and the run's cell matches the deposited one to 0.23 % or better. The cell was never in question, and the strong warning now fires zero times in 32 datasets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 09:59:41 +02:00
leonarski_fandClaude Opus 5	60f6a46b25	rugnux: index the measured beam centre too, and say when the two answers differ ... The estimate the previous commit makes is free, and a first pass re-uses spots it has already found, so running the first pass a second time at the measured centre costs about what one rung of --beam-center-search costs. That buys the one comparison an on-failure trigger structurally cannot make. A centre wrong ACROSS the spindle announces itself - the indexed fraction collapses - but a centre wrong ALONG it does not: the run holds 96-100 % of frames indexed and quietly returns a 2x, 3x or sqrt(3) axis harmonic. Nothing fails, so nothing fires. Indexing both centres and comparing the two lattices is what can see that at all. What must not arbitrate the two is the indexed frame count. Acceptance is a fractional-Miller test, so a cell twice as long has to place every spot twice as accurately to score the same; measured on real data, two centres 0.12 px apart gave the deposited cell at 99.23 % and a halved axis at 100.00 %, and the wrong answer indexed better. A rule of the form "take the centre that indexes more" picks wrong in exactly the case the comparison exists for. So only what needs no arbiter is decided. The file's centre indexing nothing where the measured one indexes a majority is not a comparison, it is a run that produced nothing and now does: take the measured centre. The two agreeing on the lattice is reported and nothing else - it is a free statement that the header is good enough for this crystal, which is most runs and is worth saying. A disagreement between two passes that both worked is reported with both cells, their primitive volumes and the ratio, flagged when that ratio is an axis harmonic, and left undecided. Runs before the blind ladder, so a measured hypothesis is tried before any grid, and only where the move exceeds three times the fit's own sigma - below that the two centres are one measurement twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 09:00:55 +02:00
leonarski_fandClaude Opus 5	0644404286	rugnux: say where the scattered background puts the beam, on every run ... A beam centre in a file is the metadata field everyone jokes about and nobody measures. rugnux already has an estimator that can measure it - the isotropy of the scattered background, whose leverage is the curvature of the water ring - and it already builds, on every run, the projection that estimator needs: --detect-beam-stop makes one to find the shadow. So the measurement is a fit over a mean image that already exists, no frames of its own, and it can simply be done always. --beam-center-check, on by default, does that and reports it: what the file claims, what the background says, how far apart the two are, how well the fit knows its own answer, and how right this particular geometry needs the centre to be. That last term is what makes the line a verdict rather than a pixel count - 3 px is nothing at 100 mm and a lost lattice at 500 mm. On the two public depositions in the test corpus whose header is the geometric centre of the detector, the line reads 73.49 px and 70.96 px against a 1.57 px and 1.82 px tolerance, on runs that until now said nothing at all about it. Nothing is committed. The run keeps the centre it was given, and where the fit cannot place the centre - a flat background has nothing to separate a radial shift from an amplitude - it says so rather than guessing. Measured on a 0.3 Mpx detector the fit takes 8 ms. The fall-through in --estimate-beam-center now reads this same result instead of repeating the fit; it is the same call on the same projection with the same mask, so that flag's answer is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:53:05 +02:00
leonarski_fandClaude Opus 5	a2f451cf12	rugnux: try the beam centre as an indexing hypothesis after a failed first pass ... A beam-centre error is not repairable downstream. It is fixed in the lab frame, so accumulating a sweep smears every reciprocal-lattice point around a circle and the FFT amplitude at an axis of length a is multiplied by J0(2 pi delta p a/(D lambda)); past the first zero the true axis is gone and its harmonic wins. But it is decidable from the data, on exactly the count the scheme choice and the axis-sign rescue already use: the right centre indexes and the wrong one does not. So after a first pass that indexes fewer than half the validation frames, step the centre a pixel at a time and keep the first rung that clears the same majority the other guards test. It runs only after a pass that has already failed, so a run that works never pays for it, and spot finding is not repeated - the cache is keyed by image and the spot positions do not depend on the centre. On the prototype this rescued 2 of 12 failing non-SLS datasets and declined cleanly on the other 10. BOTH detector directions are searched, and that is the part to keep. Measured by injection on two crystals: across the spindle the cell stays right and the indexed fraction collapses, 99 % to 25 %, so that direction announces itself; along the spindle - the one the J0 derivation calls free, because the FFT amplitude is translation-invariant - the run holds 96-100 % indexed and quietly adopts a 2x, 3x or sqrt(3) supercell. Searching only the perpendicular direction would search the failure that already shows. The step is one pixel, flat. Deriving it from the J0 law was tried and is wrong: the only cell available at that point is the one the FAILED pass returned, and on a dataset whose failed cell was a small spurious sub-cell the formula asked for a 6 px step, which steps over the lobe it is looking for. Taking any improvement rather than requiring a majority was also tried and is wrong: where no centre works, the ladder wandered to the far end of its range on 10/60 against 5/60. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	4d2bb98a13	rugnux: name the beam centre when two candidate cells are axis harmonics ... The first pass already has the evidence and says nothing about what it means. When two schemes come back with primitive volumes related by a small integer - or by sqrt(3), which is the hexagonal harmonic and is not a whole number - one of them is the other's axis harmonic, and what decides between them is the beam centre to a fraction of a pixel. The mechanism is that the J0 law is about the FFT AMPLITUDE, which is translation-invariant. A centre error along the spindle translates the derotated cloud rigidly, so the transform cannot see it, the peaks stay sharp, and the lattice fit that follows - whose origin is the beam, and which is not translation-invariant - commits with confidence to a sub-multiple. Measured on a deposited dataset: 0.12 px of centre, 0.09 px of it across the spindle against a 2.78 px need, is the whole difference between the deposited cell and a halved axis, and the halved one is the answer that indexed MORE frames - 100.00 % against 99.23 %. So the message is deliberately not conditioned on any indexing-quality signal, and it is taken over the whole pair rather than only over the tie: on this failure the frame count points the wrong way, which is exactly why it cannot arbitrate. Nothing is decided differently; the integer-subcell tie-break below is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	52e9e9da2b	beam centre: gate on what the geometry needs, and on the move being significant ... The accept bound was one flat 1.0 px constant applied at three points. It was calibrated on a 244 mm / 0.95 A / 0.15 mm geometry, and the displacement the J0 smearing law actually allows runs from 0.6 px to 9 px over the in-house and non-SLS corpora - a fifteen-fold spread - so on a loose geometry the constant is nine times too tight and throws away answers that are perfectly usable. The bound is now the LARGER of the constant and what the geometry asks, which is one-sided by construction: nothing today's gate accepts can be lost. What the geometry asks is printed and never tested against. It says how wrong the header may be; the estimator's sigma says how well the estimator knows its own answer, and gating one on the other rejects a centre correct to 0.03 px because its sigma was 0.98. It is not a relevance floor either - measured on real data, a 0.12 px change of centre, 0.03x of what the law asks across the spindle, is the difference between the deposited cell and a halved axis - so the printed line says so. Added alongside: the move has to be worth making. Under three times the estimator's own sigma it is not a measurement of anything, and a centre that is not moved cannot push the two-pass loop off its fixed point. On the 39 rotation regression crystals this adopts 38 and keeps the header on one. Two things that were happening silently now say so: the move resolved across the spindle, which is the only component the law is about, and the fall-through itself. Below about 220 deg of sweep the spot symmetry never clears the bound and the background answers instead - correctly, to within 0.23 px of the full-sweep centre at every span from 120 to 220 deg - and until now the log did not say the spot arm had even been tried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	d1818b195e	rugnux: say what the file's beam centre looks like, and when no beam stop is found ... Four things a run can say about the beam centre before it reads a single frame. All of them are free, so none of them is behind a flag. Two are worth acting on. A centre that lands on a masked pixel is wrong about something - a real beam does not sit on a dead pixel or in a module gap - and over the 29 non-SLS datasets whose outcome is on record it fires on 4, of which 3 went wrong, against 2 false alarms in 42 in-house masters. And a beam-stop pre-scan that finds exactly zero pixels is not a beamline without a beam stop: the flood starts from seeds within 4 px of the ASSUMED centre, so a centre far enough out puts every seed in a module gap. That one fires on 3 of 29 foreign datasets, all 3 of which had a bad outcome, with no false alarm in 39 in-house sweeps. The other two are provenance and are said rather than warned about: a centre equal to the geometric detector centre, or a whole number of pixels in both coordinates, was typed rather than measured. They are true of 29 of 42 in-house masters and those are out by a median 3.5 px, but as predictors of harm they sit at or below the base rate, so they gate nothing. They earn their line because the spot-symmetry search reaches only about +-55 px: a placeholder tens of pixels out is unreachable by construction, all five of its searches agree, and the sigma comes back small. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	97f57a430d	beam centre: run the four consistency restarts at once ... FindBeamCenterFromSpotSymmetry calls Estimate() five times - once for the answer and once from each of four starts 25 px away - and three quarters of each of those is an 861-point brute-force grid over the spindle. So making the uncertainty gate live was paid for by multiplying the estimator by five, which is the whole of the pre-scan's cost. The four restarts share nothing: each takes its own copy of the geometry and only reads the spots. What is wanted from them is a max, which is order-independent, so running them concurrently gives the same number. Measured on three rotation crystals, the committed centre, the reported sigma and the fitted spindle angles are unchanged to every printed digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	3ddfbb2da9	tests: give both beam-centre estimators a case with the detector tilted ... No test anywhere set a detector tilt, so PoniRot1/2 were zero in every one of them and the PONI and the direct beam sat on top of each other. That matters because the conversion between the two is used three times in the spot estimator - to centre the vote, to start each tooth's refinement, and to turn the answer back out of the spindle frame - and with the two centres coincident it is the identity, so its sign was unobservable. Verified by flipping it: with DirectBeamOffset negated, the ten pre-existing beam-centre cases all still pass and only the new one fails. The tilt used here puts the direct beam about 12 px from the PONI, twenty-four times the tolerance asserted, and the case also pins that the tilt is not read as a spindle azimuth - what the fit sees of the detector belongs to the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	8e9ca1f6d2	tests: open a third-party NXmx master, in the shape a Diamond-written one takes ... Covers all five ways such a master differed from a DECTRIS one - lengths in millimetres, no detectorSpecific, the distance one level up in NXinstrument, a pixel_mask linked into a file not holding it, and per-file links naming a plain /data - and the reader's refusal to report a master whose data files cannot be read as a dataset with no images. Verified to fail on the parent commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	c9ca7e424a	reader: read a third-party NXmx master, and stop a broken one reading as empty ... A valid NXmx master written outside the DECTRIS toolchain could not be opened. Measured on a Diamond-written master of a 360 deg EIGER 16M sweep, where the images and the meta file are pure DECTRIS and only the master is third-party - which is why the two sides disagree on units at all. Five independent things, of which two were silent: * The image size came from detectorSpecific/x_pixels_in_detector, a DECTRIS extension rather than NXmx, so a third-party writer has no reason to emit it. It now comes from the image array's own shape, as it already did for a VDS master. * Lengths were assumed to be metres and the units attribute was never read. A pixel size, sensor thickness or distance stated in millimetres - correct NXmx - was silently a factor of a thousand out. The unit is now read; an undeclared one still means metres, an unknown one is refused. * The detector distance can sit in NXinstrument rather than in NXdetector; that is now the last fallback after the NXmx and the firmware-1.x spellings. * A pixel mask that is an external link into a file not holding it passed the Exists() check and then threw on the open. Whether the array is there is now decided by opening it. * Each data file was re-opened and searched for /entry/data/data, ignoring the path the master's own link names. A master linking to a plain /data therefore found no images at all - and that was a warning and exit code 0 over a sweep sitting right there, not an error. The link is now taken at its word, and a master that links to data files but yields no images is an error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:27:53 +02:00
leonarski_fandClaude Opus 5	bec2a10a3a	docs: add the eight new public datasets to the non-SLS test list ... Three IUCrData Raw Data Letters (Zenodo, CC-BY-4.0) and five SBGrid Data Bank depositions (CC0) have been added to the data the pipeline is exercised on. Same rules as the rest of the page: the source is the repository and its own citable DOI, every one of which was resolved before it was written down; beamline, resolution, space group and cell are the values deposited with the PDB entry; the detector is read out of the image files. None of the new detectors disagree with their PDB entry. Two of them do not fit the page's one-row-per-sweep shape, so the shape is described rather than flattened. The 6R72 Zenodo record holds two complete 360-degree collections on one crystal - a helical one that produced the deposited structure and a low-dose one that has no PDB entry - and both are listed, sharing a DOI, with the second in the no-PDB-entry table. The three CHESS depositions are 4-11 wedges of 50 degrees per crystal plus a rotation taken with the crystal translated out of the beam, tabulated in a new section. One SBGrid deposition is named but not in the table: its images are 1995 CCD TIFFs, a format the reader does not support, so it is not processed here and saying so is more useful than leaving it out. ACKNOWLEDGEMENT gains the new per-repository counts and a pointer to the Raw Data Letter citations; TESTS points at the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-29 23:40:10 +02:00
leonarski_fandClaude Opus 5	08f57235d6	reader: take the image orientation from the module directions the NXmx file states ... NXmx says how the stored image sits in the detector plane, in the NXdetector_module fast_pixel_direction and slow_pixel_direction vectors. rugnux read neither, and assumed every detector was mounted the way this system mounts its own. A facility that bolts its detector a quarter turn round therefore produced a geometry that was wrong by 90 degrees, which no amount of refinement recovers: the run indexed nothing and reported that it could find no lattice. The two vectors are read, taken from McStas into the internal frame, and matched against the eight discrete image orientations. An exact match is adopted; anything else is left alone, because an orientation that is not discrete is a continuous rotation of the detector in its own plane and cannot be told apart from the tilt by looking at the module. Every file examined here is exactly discrete. Measured over the 94-dataset battery: 78 masters state no module vectors at all and 12 state the standard ones, so the change can reach exactly 4. Three of those go from "Nothing was integrated" to a complete merged result with no flags - 43/93/92% indexed, CC(1/2) 0.957/0.984/0.998 - and are the sets that until now needed a hand-passed --rot3 pi/2. The fourth states a half turn, which the existing rotation-axis sign rescue already absorbed, and is unchanged to every digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	9d3c2787f8	docs: credit the public diffraction data the pipeline is tested against ... Jungfraujoch is developed at one facility, so the reader and the reduction pipeline are exercised on data collected elsewhere - other detectors, other file formats, other conventions. That data was collected and published by other people, and until now nothing in the repository said so. NON_SLS_TEST_DATA lists all 51 datasets: the DOI to cite for each, the repository it came from, and the experiment as deposited. Beamline, resolution, space group and cell are the values deposited with the corresponding PDB entry, read from the RCSB data API - not results measured here; no quantity produced by this software appears on the page. The detector is read out of the image file instead, because the detector named in a PDB entry is often only approximate, and the twelve cases where the two disagree are listed rather than silently reconciled. ACKNOWLEDGEMENT gains a section for the repositories themselves, with the IRRMC, SBGrid and Zenodo citations and the PDB citation for the metadata. Every DOI on both pages was resolved before it was written down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:59:49 +02:00
leonarski_fandClaude Opus 5	84efb0666c	viewer: open a miniCBF sweep, the same way rugnux already does ... The native miniCBF reader added for rugnux is a plain JFJochReader, so the viewer only needed to be told which one to open a file with. JFJochImageReadingWorker held a concrete JFJochHDF5Reader; it now holds both readers and a JFJochReader* pointing at whichever the open file needs, chosen by JFJochCBFReader::CanRead. Naming any frame opens the whole sweep - the reader's own template matcher decides which frames belong to it, so a directory holding two sweeps or XDS auxiliary files is not spliced together. The previous file is closed before the new one is opened, so switching format between HDF5 and CBF in one session leaves nothing behind. Everything the viewer draws already goes through JFJochReader and JFJochReaderDataset, so nothing else had to move. The two HDF5-only features stay on the HDF5 reader: calibration images, and the reprocessing snapshots, which are metadata read back over the same images the reader is serving and have nothing to attach to for a directory of CBFs. A raw sweep therefore shows geometry and pictures with an empty run list, which is the state a live HTTP stream is already in. Reprocessing jobs run on a CBF sweep too - JFJochProcessController opens its own reader the same way, and Rugnux has handled a CBF source since the reader landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 21:49:09 +02:00
leonarski_fandClaude Opus 5	172a845cbb	rotation indexing: refine twelve candidate lattices instead of four ... The first pass refined only min(candidates, 4). The comment at the selection already conceded that the pre-refinement indexed fraction is an unreliable discriminator, and on one dataset degenerate cells occupied three of those four slots - so the right cell, ranked fifth, was never refined. The preceding commit's volume guard frees those slots but does not widen them. This is a one-for-one trade, measured by rebuilding with the old value to confirm the attribution. It gains one dataset: R_meas 27.2% to 23.8%, with mean I over sigma better in EVERY shell at identical shell edges, so it is not a resolution ramp. It costs another: a clean abort becomes a wrong four-fold supercell in P1 at 36% completeness. That is the same shape of trade already accepted for the plane-normal cap search in the preceding commit - an honest failure becoming a wrong answer - on a different dataset. It is committed separately so it can be reverted on its own if the maintainer weighs this instance differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:35 +02:00
leonarski_fandClaude Opus 5	4e8db41785	indexing: refuse a coplanar candidate, and search the plane normal when the shortlist is flat ... Three related changes to the FFT candidate path, batteried together because they touch the same function. A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only - the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41 of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around 1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a not-a-number Jacobian and wrote several hundred lines of solver output per failed solve. VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below the flattest real candidate observed and a thousand times above where float loses the sign. It is enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is now guarded at the two remaining sites that share the pattern. A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal. That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to 3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further transform is spent with the same direction count inside a three-degree cap about the normal, so the plan and buffers are untouched. More directions cannot substitute: at the exact true direction the long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same answer to three decimal places and takes a run from 2.5 to 8 GB of device memory. fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling maximum did fire twice, so the path is live and correctly inert. Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13 failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of 93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap fires on 4 of 93, none of them in the in-house or private arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:20 +02:00
leonarski_fandClaude Opus 5	9a66a21a10	rugnux: expose the indexed-spot gate and the third detector rotation ... --min-indexed-spots sets how many spots must lie on a candidate lattice for a frame to count as indexed. It is what the rotation first pass scores candidate lattices on, and it was reachable only by editing the source. The default of 9 is unchanged; on a sparse pattern, lowering it to 6 takes the first pass from 31 of 60 validation frames to 38. --rot3 completes the set beside --rot1 and --rot2. rugnux --mode calibration WRITES a .poni file containing all three, and without this flag a refined third rotation could not be given back - it had to be transcribed by hand and its convention undone. With it, a detector mounted rotated in its own plane is expressible from the command line. Also corrects the description of fft_high_resolution_A in the API, which claimed to be the highest resolution of spots USED. It filters nothing - every spot is projected whatever its resolution. It sizes the projection histogram, and with it the transform. Implementing the documented behaviour was tried separately and makes its own motivating crystal strictly worse, so the code is right and the description was wrong. (The generated Python client doc carries the same text and will pick this up at the next regeneration.) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:59:03 +02:00
leonarski_fandClaude Opus 5	367eba55b1	reader: take a miniCBF rotation axis from the goniometer the header states ... Every miniCBF sweep was handed the same hardcoded axis regardless of what its header said, and Chi/Kappa/Phi/Omega were not parsed at all - there were no members for them. A sweep collected on a tilted chi cradle therefore ran with an axis that is 54.7 degrees wrong. The header angles and their increments are now read, the scanned axis is identified from the non-zero increment (the name is consulted only when no increment is stated, which is what absorbs the five different spellings the corpus contains, including one file that states no axis name at all), and a phi scan composes the head chain. An omega scan returns the base axis untouched, because a fixed chi cannot tilt the axis it hangs from. -9999 is a sentinel meaning "not set", not an angle. It is treated as absent, so it can never reach the geometry. The direction and sense are not invented: these files append an imgCIF _axis loop stating their own vectors, and SOURCE with GRAVITY fix the imgCIF-to-internal transform, which independently reproduces the transform this repository already documents for NXmx. Under it the file's own stated phi axis is exactly the composed one, to four decimals. Driving the real reader over all 39 corpus sweeps, 37 return the previous axis bit-identically - including every sweep carrying a large fixed chi, every sentinel header and every axis-name spelling. Only the two genuine phi scans move, and an unrelated rotation dataset is unchanged end to end. This is necessary but not sufficient for the one dataset that motivates it: with the axis corrected it still does not index, because that detector is also mounted rotated 90 degrees in its own plane, which the reader does not yet read. Compensating both takes its phi sweep from no indexed validation frames to 90.89% indexed and a complete merge, which is what shows this half is load-bearing. The detector mount and the two-theta swing belong to the detector-frame work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:50 +02:00
leonarski_fandClaude Opus 5	60c15d6931	reader: open a master whose beamline writes standards-correct NXmx differently ... Four defects, hit in sequence, that between them stopped eight masters from one beamline before any geometry question was reached. The files are correct NeXus; the reader was assuming one writer's conventions. ReadScalar demanded rank 0. NXmx puts no rank on distance, saturation_value, two_theta or det_z, and these files write them as shape (1,). Any dataspace holding exactly one element is now accepted; a genuine vector is still refused. frame_time was read unconditionally and NXmx does not require it. A virtual-dataset source filename of "." was resolved as a relative path, giving <dir>/. - but "." is HDF5's spelling for THIS file, and these masters compose /entry/data/data as a virtual dataset over datasets in themselves that are external links to the data files. The virtual source's DATASET PATH was parsed and then ignored in favour of a hardcoded /entry/data/data, while these data files keep their images at the root. Fixing that exposed a fifth: the positional-read fast path used the master's own path, but a dataset reached through an external link lives in another file and a chunk address is an offset into THAT file. An audit of all 38 masters in the non-SLS corpus finds exactly these eight need the change and the other 30 need nothing. Corpus A/B over 73 dataset pairs, base and patched back to back: 64 byte-identical results, 9 identical failures, none differing. The blast radius is bounded by construction - JFJochReader is linked only by rugnux and jfjoch_viewer, and the one shared header this touches only ever accepts more, so nothing that opened before can read differently. With it the eight files read completely: 12850 images, no decode errors, seven of them spanning two source datasets through the master's own external links. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:33 +02:00
leonarski_fandClaude Opus 5	4ef6bd9952	rugnux: decide the space group after the second pass, not the first ... Pass 1 searched for the space group on the geometry the run started with, stashed the answer, and reinstated it before pass 2's merge - so the decision was made on the worse of the two passes and then fed forward as a constraint. Four guards existed to reconcile it with what pass 2 later found. Now pass 1 does not search at all. It keeps its first merge, which is where mosaicity smoothing and geometry post-refinement already happen, and skips the in-symmetry re-merge that existed only to feed the search. prepass_merge_sg_, prepass_promoted_point_group_, the reinstatement, the primitive-to-conventional reindex it needed, and lattice_conflicts_with_prepass_sg all go. The obstacle was the pass-adoption guard, which compared completeness and CC1/2 across what could now be two different space groups. It compares each pass's FIRST merge instead - P1 on both sides, full range, before correction surfaces - which pass 2 produces anyway. One guard had to be replaced rather than removed. The index-time veto is re-keyed from pass 1's GROUP to its LATTICE and made one-directional: a centred pass-1 lattice against a primitive pass-2 one. Without it a C2 crystal fell to P1, and the obvious narrower fix - exempting a pass-1 P1 - would not have caught it, because that crystal's pass-2 lattice is monoclinic-P rather than triclinic. A centred lattice and its primitive sub-cell share a primitive volume, so the supercell arm cannot see that demotion. Corpus of 93 datasets, both arms at -N 6, every log validated: 78 comparable, 65 with a byte-identical p.hkl. Two crystals gain their reference point group, one gains its screw axis, one is lost. Per shell, over the 72 unchanged crystals, the median move is +0.25 points of R_meas and zero in <I/sigma>, CC1/2 and completeness. Load-matched compute is +0.4%, and the header-geometry fallback fires three times in the old arm against once in the new. What is lost is one crystal whose 222 the operator correlations still confirm: the merge chi-squared ratio moves 3% across a fixed bound at the refined geometry. A second crystal in the XDS harness crosses the H bound the same way. Both bounds are in SearchSpaceGroup and are recorded elsewhere as miscalibrated; recalibrating them is deliberately left to its own change. The comment at the long-axis rescue is corrected while here: the resolution setting only sizes the FFT histogram, and coarsening moves the true axis DOWN the direction ranking - 1996 to 16361 of 16384 - rather than making it more robust as the comment claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:58:17 +02:00
leonarski_fandClaude Opus 5	8634e9a72b	symmetry: score screw absences per zone, and choose centring by likelihood ... Two places where the absence tests threw away information they already had. The screw evidence was POOLED across axial rows, so one unmeasured row vetoed a confirmed one. A crystal reading h: +18/+24 and k: -6/+8 answered a silent P222 rather than "2_1 along a confirmed, b undetermined". Scoring each zone separately, and letting a zone abstain, is what POINTLESS has always done; the conditions are independent, so their log-likelihoods add. Centring was decided on a COUNT of net absences while screws used a Beta-tail likelihood. On one crystal that made F222 (1662 absent, 356 violations, absent class at 34% of present) beat C222_1 (1098 absent, 15 violations, 0.4%) purely because 1306 > 1083. Giving centring the same likelihood separates them by 1717 nats - F222 -357.9, C222_1 +1359.3, I222 -787.6 - and needs no bound: min_absent_observed and max_absent_present_ratio are untouched, and nothing was tuned to this case. The deposited structure for that crystal is C222_1. Corpus of 94 datasets, both arms at -N 6: 78 comparable, 73 bit-identical, point group correct on 68 against 66, space-group number on 54 against 51, nothing regressed. The 73 bit-identical results are also the determinism control - two binaries cannot agree byte-for-byte that often by chance. An earlier ranking key summed each group's single gating number and demoted a P2_12_12_1 to P2_12_12, because pooling three genuine screws reads weaker than two when the third row is shallower. Summing the per-zone log-likelihoods is what fixes that, and the whole corpus was re-run on the corrected binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:57 +02:00
leonarski_fandClaude Opus 5	10873eee7a	lattice search: reject an impossible character, and supplement every angle ... Two defects in the Bravais character walk, both of which silently cost symmetry. An impossible character MATCHED. The walk computes acos(cond_F/sqrt(A*B)); when the character is geometrically impossible the argument leaves [-1,1], acos returns NaN, and the acceptance test fabs(NaN - actual) > tol is FALSE - so the character is taken. One corpus crystal matched a monoclinic-C character whose implied cos(gamma) is 1.086. The type-boundary retry covered beta only. All three angles carry the Niggli boundary, and negating two basis vectors supplements the OTHER two, so each angle needs its own flip. A crystal whose reduced gamma sits at 89.900 degrees needs the alpha/beta flip and never got it. Measured on 84369 exact lattices spanning all 14 Bravais classes: nothing is lost in any class, and orthorhombic-I recovery rises from 60.3% to 87.2%. Over 440000 random and perturbed cells the first change only ever demotes a monoclinic-C match to triclinic and the second only ever promotes out of triclinic - it is a strict superset of the beta-only retry. On the 94-dataset corpus, two crystals gain their correct point group and none regresses. Both regression tests fail without the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:41 +02:00
leonarski_fandClaude Opus 5	1ea6ba25be	reader: open a master that states its goniometer axes in one place and their angles in another ... A hybrid master carries an NXmx /entry/sample/transformations group holding one EMPTY SUBGROUP per axis - the direction as a vector attribute, no NX_class, no units, no angles - beside a legacy /entry/sample/goniometer group holding all the angles. Opening it failed outright with "Cannot open HDF5 dataset /entry/sample/transformations/omega": the existing legacy fallback keys off Exists("/entry/sample/transformations"), which is true here, so it never fired, and the axis stub was then opened as if it were the angle dataset. Present is not the same as usable. GoniometerGroup now takes transformations only if it holds at least one DATASET, and ReadAxis asks IsDataSet rather than Exists, so a member that is not a dataset can no longer be read as one. HDF5Object gains that predicate, in the style of the neighbouring Exists. The stub is the load-bearing half, not merely the thing that crashed. In the legacy branch, before falling back to the assumed (-1,0,0), the reader now looks for the NXmx stub and takes its stated vector. With it the axis is (0,-1,0) and the run indexes 60/60 validation frames; with the stub deleted the assumption applies and the same file indexes 0/60 on both schemes and both signs, and the run stops with no lattice. So without this half the fix would have turned "cannot open" into "found no lattice" - a differently shaped failure, not a success. The direction stated here is 90 degrees from the assumption, not merely its negation, which the rotation first pass could have recovered on its own. The vector size check moved out of the first branch so it now covers every path that produces one. Verified: the file processes end to end, 100% of frames indexed, cubic cell 105.87 against a deposited 105.88 (0.009%), space group reported as I23 or I213 - correctly refusing to choose, since the reflections that separate them are extinguished by the I-centring and were never measured. A both-layout master reprocesses unchanged (100% indexed, P212121). [HDF5] passes: 2194 assertions in 91 cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:47:41 +02:00
leonarski_fandClaude Opus 5	6319d5c600	indexing: tell an indexer that failed apart from one that found nothing ... IndexerThreadPool collapsed two different outcomes into the same empty reply. A worker that threw set result = nullptr, and so did a dispatch that never found a free worker; both then became a default-constructed IndexerResult, indistinguishable downstream from an indexer that ran and found no lattice. The first says nothing about the frame at all - the indexer never looked, and it will fail again on the next one - while the second is a real negative result about the crystal. The visible cost was the advice a failed run gave. With the card full, rugnux printed "Indexer thread 0 failed: CUDA (GPU) error" and then ended with "Two-pass rotation indexing found no lattice. Check the beam centre (--beam-x / --beam-y), raise --max-spots ..." - sending the operator to look at geometry that was never wrong, for a machine that was simply out of memory. None of those remedies can help when the frames were not examined. IndexerResult gains an optional error, set by the worker and by the pool's own catch; the remaining nullptr path keeps its meaning of "not attempted" and deliberately carries no error. RotationIndexer records it and exposes GetIndexerError(), and rugnux's first pass branches on it at the throw site, naming the resource failure instead. The error travels as DATA through image_analysis/ rather than as an exception, because RotationIndexer::RunIndexing() is on the online path - IndexAndRefine calls it on a schedule from the broker and the receiver, where dropping a frame is the right failure and killing a live acquisition is not. Only rugnux, which owns the "this run is over" decision, turns it into one. The failure result is byte-identical to the default-constructed one it replaces, so any_executed and the online frame-drop behaviour are unchanged. The new IndexerError category exists because the category is only the display prefix on what() - Category() is read nowhere - and the old line read "Processing failed: Input parameter invalid" for a GPU fault, which is the same defect one layer up. SpotFinderError is the precedent. msg.indexing_result is deliberately left alone: of its three states, "not attempted" is the honest one for a frame the indexer never examined, and asserting false would be the same collapse again. Verified on a squeezed card (246 MB free, -N 4): exit 1, zero dropped frames, the new message, and no mention of --beam-x. On a free card, against the previous binary at -N 6 on the same input, the logs differ only in paths and timing - same space group, cell and merge statistics. Targeted Catch2 cases pass, including three that drive the pool through the online path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:30:28 +02:00
leonarski_fandClaude Opus 5	92ad304f15	rugnux: fail on an exhausted GPU instead of dropping the images it could not process ... A per-image worker caught every std::exception and continued to the next image. That is right for one undecodable image and wrong for a resource fault: a CUDA out-of-memory says nothing about the image that happened to be in flight and everything about the machine, and it recurs on the next one. The run therefore logged an error per image, skipped each of them, and exited 0 with a dataset that was silently short - the merged numbers all moved, and nothing in the exit code or the summary said so. Observed with several processes sharing one card: two arms of the same comparison lost 21 and 15 images, by different amounts, so the arms no longer saw the same data. IsFatalResourceError rethrows for std::bad_alloc and for a JFJochException of category GPUCUDAError or MemAllocFailed, and every per-image catch consults it first. There are ten of them, not the three the obvious grep finds: besides the analyze/integrate/load sites in the two image loops, the first-pass spot read swallows the same way at Warning severity, the pre-scan swallows it in three places, and one geometry-refinement site caught it with no log line at all. The first-pass one matters most - those frames build the reciprocal-space cloud the indexer runs on, so losing them thins the input that decides whether the crystal indexes, and it does not change IMAGES_PROCESSED, so a frame-count audit cannot see it. Every one of these loops joins with future::get() or ParallelFor's RunTasks, both of which hold the first exception and rethrow it at the join, so the throw propagates and nothing terminates. JFJochException gained a Category() accessor for the test. Verified by squeezing the card to 804 MB free and running at -N 4: the run now exits 1 with zero dropped-image lines, where the same test on the previous binary produced the silent skips above. The broker and receiver paths are deliberately untouched - dropping a frame is the better failure there than killing a live acquisition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 14:58:04 +02:00
leonarski_fandClaude Opus 5	dce8d877d0	Scaling: make Run() idempotent, and re-calibrate the promotion it was hiding ... RotationScaleMerge::Run() resumed its alternating-least-squares loop from wherever the previous call left off instead of restarting from the ingested data. A de-novo run calls it four times, so pass N was scaled at 3N iterations, not 3. Two identical calls on the same object produced merges correlating at 0.847 with ZERO bit-identical intensities; they are now 68010 of 68010 identical. Where that mattered is narrower than it sounds, and worth stating exactly. In a determined point group the fit is stationary from iteration 2 - over the battery the collapsed-scale guard drops 377 frames at three iterations and 397 at twelve, and 30 of 39 crystals drop nothing at any count - so production merges were barely touched, 38 of 39 rows character-identical. But the SEARCH merges in P1, where a reflection has two or three observations, and there the fit never reaches a fixed point at all: the same guard drops 417 frames at one iteration and 1101 at three. A factor 1.05 against a factor 2.6, same code. So the defect lived in the one arm whose output the user cannot recover from downstream - the arm that picks the space group. Fixing it moves the statistics the promotion gates are calibrated on. The population as a whole does not shift (median change 0.9-2.7% over 280 matched candidates) but the tails do, p90 by 8-14%, and that is exactly where a promotion is decided. On the two crystals that decide, the H statistic's two populations SWAP ORDER: a genuine tetragonal 4->422 goes 1.685 -> 1.819 while a trigonal 3->32 twin law goes 1.906 -> 1.665. No H bound exists any more that keeps the genuine case and refuses the twin - the noise the loop was adding had been acting as a brake. Of the four statistics only b and R_meas still order the two correctly. Set max_systematic_b_ratio 1.90 -> 1.78, by a rule written down before the population statistic existed: take the statistic with the largest spurious_min/genuine_max on the population its gate actually sees, put the bound at the geometric midpoint. That is sqrt(1.711 * 1.853) = 1.78. Battery 36/39 space groups matching XDS, the same as before the scaling fix and one better than the scaling fix alone; the three mismatches are the known references where XDS is wrong. The refusal now reaches the user by name. The bound's margin is 4% either side - SMALLER than the p90 shift this very commit produced in that statistic. It separates the two crystals that exist and is not robust to another input shift, which is written into the header rather than left to be discovered. Two hazards go with it: re-measuring it must hold the scaling correction surfaces fixed, since changing the iteration count re-rolls their cross-validated on/off decision on 7 of 39 crystals for ~0.3 in ISa, all-or-nothing and non-monotone; and --scaling-iterations stays at 3 on both arms, confirmed rather than assumed (production 2->3 moves <I/sigma> +1.38% and 3->6 only +0.13%; the search gives the identical space group on all 39 crystals at 2 and at 3, and loses one at 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 22:44:21 +02:00
leonarski_fandClaude Opus 5	fa4aa30a1e	reader: match the legacy companion suffixes exactly, not any underscore ... The legacy DECTRIS layout stores five companion scalars beside each goniometer axis - AXIS_end, _start, _increment, _range_average, _range_total - and only the bare name is the axis itself. The first cut rejected _end and _range, which left _start and _increment to be offered to ReadAxis as axes in their own right. Matching the full suffix set, anchored at the end of the name, also keeps a genuine two_theta axis readable, which a "contains an underscore" test would not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	df3696e6e2	reader: fall back to the pre-NXmx key names, so an Eiger 1.x master opens ... Firmware 1.x writes the same three values under different names. Try the NXmx spelling first and the old one only if it is absent: detector/distance <- detector/detector_distance detector/saturation_value <- detectorSpecific/countrate_correction_count_cutoff sample/transformations <- sample/goniometer This cannot change what a current file reads: every modern Eiger master carries BOTH spellings. Measured on thirteen masters from ten facilities, firmware release-2020.2.1 through release-2024.1.1 - all of them write detector_distance and countrate_correction_count_cutoff beside the NXmx names, and a goniometer group beside the transformations one. The goniometer is the one that matters. A goniometer is only ever set from that one group, so a file whose axes are somewhere else was not an error - it was read as STILLS, silently, and the run completed with the wrong answer. The old layout also tags no axis with transformation_type and gives no vector, both of which ReadAxis required, so absence now means two different things by layout: in a transformations group it still means "not an axis" (that is how AXISNAME_end and the width scalars are skipped), while in the legacy group every leaf IS an axis and the companions are recognised by name instead. A missing direction defaults to the one every DECTRIS master since has written and says so in a warning rather than assuming it silently; a wrong guess there does not index, so it is visible, and the rotation first pass tries the opposite sign anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	a7615c8778	compression: read plain LZ4 (HDF5 filter 32004), and name the filter when one is unsupported ... DECTRIS Eiger firmware 1.x compressed its images with the HDF Group's plain LZ4 filter rather than bitshuffle, and files from that era are still what a repository hands you. Read-only: nothing here produces it, and the new enumerator goes last so the existing values do not move, the enum being part of the CBOR stream. The framing is the same as bitshuffle's - a 64-bit big-endian total size, a 32-bit big-endian block size in bytes, then each block prefixed by its 32-bit big-endian compressed size - so only the unshuffle step differs. Verified on a real chunk: the declared total matched width*height*4 exactly and the block walk consumed the chunk to the byte. It cannot reuse the bitshuffle path, which requires the block to be a multiple of BSHUF_BLOCKED_MULT elements: these files put the whole image in ONE block, which is not. The unsupported-filter message now names the filter it found and the ones it knows. Before, an Eiger 1.x file did not report a codec problem at all - it reached spot finding with nothing decoded and failed as "0 spots from 60 images" then "found no lattice", which reads as a crystallography failure rather than a format one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 21:56:12 +02:00
leonarski_fandClaude Opus 5	84d14696b1	rugnux: accept a CBF sweep, decide the rotation axis sign, and search as far as a given cell needs ... Three changes to the rotation path, all in the same first-pass block. Input. Rugnux now holds a JFJochReader rather than the concrete HDF5 reader, so a sweep can come from a directory of miniCBF as well as from a master file; the CLI picks by looking at the file. Only the _process.h5 image links need the concrete reader and they ask for it by dynamic_cast - there is nothing for a virtual dataset to point at in a directory of CBFs, so that file writes its images instead of linking to them. --mode scale refuses a CBF sweep, there being no stored reflections to re-scale. Search bound. A reference cell longer than the FFT bound cannot be represented by the search meant to find it, so the run reported "found no lattice" - strictly worse than not giving the cell at all. Raise the bound to cover a given cell (margin 1.3, as DIALS uses when a cell is known), and raise it in the long-axis rescue too, where it costs nothing: that pass already coarsens the resolution to 3.5 A, which shrinks the transform by 1.75x, so spending it back on reach leaves the rescue at 1.14x the standard pass. The rescue's constrained re-index gets a pool wide enough to hold what the coarse pass recovered, which it did not before. ROTATION ONLY - stills fire the FFT once per image across every worker and routinely run with a known cell, so widening the transform there would cost the whole serial run for nothing. Clamped rather than throwing: the rescue can recover an implausible axis, and a bad candidate must not take the run down with it. Axis sign. The rotation-axis sign is a convention the input often cannot settle - a miniCBF header names the axis but gives no direction, and an NXmx vector is only meaningful together with the detector orientation, which nothing here reads. It is decidable from the data, though: the wrong sign does not index at all (measured on one sweep, changing nothing else: every frame indexed one way, none the other). So after a poor first pass, try the other sign and keep whichever indexes more validation frames - the same count the scheme choice already uses, so no new metric and no new threshold. It flips the AXIS, not the angles, because prediction reads the axis too; the decision then holds for the rest of the run. Runs before the long-axis rescue, since with the sign wrong every candidate lattice is wrong. Only after a poor pass, so a correctly-signed file costs nothing, and spot finding is not repeated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:41 +02:00
leonarski_fandClaude Opus 5	dc16a00271	reader: read a PILATUS miniCBF sweep natively, without libcbf ... Most facilities still archive rotation data as a directory of miniCBF frames, which until now had to be converted to HDF5 before rugnux could see it. Nothing in that format needs a CIF parser or a library: it is an ASCII header, four separator bytes, then one byte-offset compressed image, and every value the reader wants sits on a "# " comment line or a MIME line. MiniCBF holds the format itself - header parse and the byte-offset decoder, which is a running value with deltas stored smallest-container-first. Verified byte-exact against dxtbx on PILATUS 6M, 6M-F, 300K, silicon and CdTe sensors, and three sensor thicknesses. JFJochCBFReader is a sibling of JFJochHDF5Reader under the JFJochReader base. NAMING ANY FRAME READS ITS WHOLE SWEEP: the sweep is identified by the template (prefix + digit count) the named frame belongs to, not by "every .cbf in the directory", so a directory holding two sweeps does not splice two crystals together. Naming a directory takes the sweep with the most frames in it. Images decode on demand, one per call, so any number of workers can read at once - there is no global lock as there is on the HDF5 path, HDF5 not being thread-safe. A raw CBF carries no analysis results, so the dataset it builds is the geometry, the mask and nothing else, exactly as a plain DECTRIS file with no /entry/MX gives. Two header quirks are handled because real files have them: the sensor material is written "Silicon" where the rest of the code compares against "CdTe", and the thickness unit is sometimes omitted. Headers are not a fixed size either - one set carries 6335 bytes - so the parse runs to the binary separator rather than over a fixed prefix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00
leonarski_fandClaude Opus 5	e78fbd55b7	Indexing: let the FFT search reach past 500 A, and stop clipping the peak background ... fft_max_unit_cell_A was both the default and an enforced check_max, so 500 A was the longest basis vector the FFT could ever return: FFTIndexer sizes its projected histogram from that value and the transform's last usable bin IS that length. Of the PDB's 206950 X-ray entries, 1091 (0.53%) have an axis longer than that and were unindexable by construction. The accepted range now goes to 1200 A, which leaves 8. The DEFAULT is unchanged at 500 - the histogram is sized from the value in use, so nothing pays for the wider range unless a caller asks for it. The peak picker's running-mean background was truncated at the ends of the spectrum rather than slid inward, so a peak within bg_half (~15 A) of either end - which is exactly where the longest cells sit - was judged on a one-sided background, biasing its prominence by however much the spectrum sloped there. Keep the window a constant width and slide it. Both bounds stay monotonically non-decreasing in j, so the GPU kernel's running sum is still valid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00
leonarski_fandClaude Opus 5	4951538368	rugnux: measure the spot budget instead of taking it as given ... --max-spots was a fixed 1000, and on a rotation sweep it sets the DENOMINATOR of the per-frame acceptance gate, which admits a frame when at least a fifth of its spots index. Detections are not all reflections: on a strongly diffracting crystal with heavy solvent background there are 2372 a frame, 1416 of them on ice bands, and only 178 index - so a budget that takes essentially the whole list puts the entire frame population on the gate (median indexed fraction 0.271, tenth percentile 0.218) and the run integrates 83% of its images. The same run with a smaller budget gains 211 frames and loses none, and the frames it gains are the ones with the MOST detections. That is how a larger budget integrates fewer images. So measure it: over the first pass's validation frames, with the sweep's lattice known, tally each spot by rank as +1 if it lies on the lattice and take the budget at argmax over N of n_indexed(N) - 0.20 * n_counted(N) which rises exactly while spots at that depth index better than the gate's own floor and falls after. The 0.20 is that floor, not a new constant - it is lifted out of the function-local it already lived in. It means: as deep into the intensity-ordered list as the image is still showing reflections of THIS crystal. The argmax alone would not do. Under the null that spots index at the same rate at every depth the tally is a driftless random walk, whose maximum is positive whatever the data, so a bare argmax shortens every dataset. The budget therefore has to clear the walk's own noise: the quantity it acts on is the fall from the peak to the end of the list, which is that walk read backwards, and the reflection principle gives its null law in closed form - P(fall > z*sqrt(g(1-g)T)) = 2(1-Phi(z)). That already pays for the search over ranks, so nothing further is owed to multiple comparisons. One false cut in a thousand measurements - a twelfth of one over a 39-crystal two-pass corpus - fixes z at 3.29. The level was chosen before the rule was written and was not revisited afterwards. Battery, same build, one changed default: 36/39 space groups in both arms, none lost, and 36 of the 39 crystals BIT-IDENTICAL. Two move materially - the strong crystal by +17.4% observations and +35.9 points of CC1/2 at 1.58 A, its indexing rate 83.4 -> 95.1%, and another by +93.5% observations with completeness 76.6 -> 98.4%. The significance requirement is what makes that list clean: it removed the one crystal the unguarded rule regressed, and with it two of the five gains, whose peaks do not clear 3.29 sigma on 60 frames. The lever for those is the sample and not the threshold - power grows as the root of the frame count while the bar stays where it is. Rotation only, and the first-pass lattice search always sees the full list: a fixed --max-spots 250 would have been much cheaper to write and fails a battery crystal outright, starving the de-novo FFT into a wrong cell that indexes 7 of 60 frames. Stills never reach the code path - verified bit-identical merged reflections on a serial set - and --max-spots N still pins it, as does the library default the broker and the FPGA use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 19:02:02 +02:00
leonarski_fandClaude Opus 5	df98bb8789	Space-group search: ignore the frames the crystal barely diffracted on ... A crystal that repeatedly leaves the beam over a full turn was read as P2(1) where it is P4(3)2(1)2, on data that merge at CC1/2 96-98% once that symmetry is assumed. The cell was right, the indexing was right - 83% of images - and every 422 operator correlation still came out between 0.08 and 0.38 against a gate of 0.30. The per-frame scale enters the intensities as 1/G. On this crystal the fitted scale spans 356x between its 5th and 95th percentiles and 15% of frames sit below a tenth of the run median, so those frames arrive amplified ten- to a hundredfold - and their sigmas are amplified by exactly the same factor, so nothing weighted by sigma can see it. What saves the PRODUCTION merge is multiplicity: in the determined symmetry a reflection is measured ten or twenty times and --reject-outliers throws the amplified observation out. The search merges in P1, where a reflection has two or three observations and no majority exists to call any of them an outlier, so the bad value lands in the merged intensity at full weight and the operator correlations pay for it. So drop those frames from the search merge only, beside the |zeta| filter and through the same snapshot that undoes it before the production merge, which keeps every frame. A tenth of the median because that is where the two populations sit: over the 39-crystal battery 33 crystals have NOT ONE frame below it, so the filter is inert on them by construction; of the six that do, five spend 0.1-3.5% of their frames there against this crystal's 15%. A twentieth leaves it under the gate; a fifth starts costing frames the battery says are real, the smallest legitimate min(G)/median(G) measured being 0.070. Battery 35/39 -> 36/39 space groups matching XDS, with 38 of the 39 rows character-identical: the only crystal that moves is this one. On it the operators go to 0.37-0.62, and every one improves in every resolution shell, most at low resolution - the signature of a scale error removed rather than noise removed. The run then merges to 1.41 A instead of 1.67, at CC1/2 97.7% and completeness 99.7%. Four other explanations were measured and refuted first: the search resolution cut (the correlations are flat in resolution, and a battery crystal with a finer detector and no cut at all gets 422 without trouble), the space group being inherited from the pass-1 geometry (handing the refined geometry in from the start changes nothing), the ordering of the corr snapshot around the collapsed-scale guard (0.02 in correlation, nothing in space group), and the ice-ring flagging (removing it entirely leaves the answer at P2(1)). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 17:20:36 +02:00
leonarski_fandClaude Opus 5	9f2696c1ac	Adaptive spot finding: the ring threshold replaces the floor, not the local test ... The self-calibrating finder was meant to replace the classic finder's FIXED PHOTON FLOOR with a per-resolution-ring threshold read off the image's own noise. As written it replaced the local-box SNR test as well, and that is the defect: a whole-ring threshold is an ABSOLUTE contour with no feedback from a pixel's own surroundings, so the area a spot puts above it grows as sigma^2*ln(peak/threshold) and never saturates. Measured on a strongly diffracting rotation set, the detected footprint grows by +8.05 pixels per e-fold of peak, so the brightest reflections came out as 100-500 pixel blobs and were then discarded for exceeding the size bound - every one of the ten strongest on an image. Intersecting with the local box gives -0.24 pixels per e-fold, the classic finder's own number to two decimals. WHY the local box is the right partner, rather than merely the incumbent: it is a prominence rule whose reference level is a 961-pixel mean. A spot inflates the box's own variance and the peak divides out of the acceptance test, so it cuts at a fixed FRACTION of the spot's own height. Referring that level to fewer pixels makes it inherit their shot noise - at FIXED footprint, estimating the level from 961 pixels, from 25, and from the single maximum gives centroid residuals of 0.524, 0.539 and 0.656 - so flat growth and a stable centroid turn out to be two ends of one dial. A contour on the bare maximum has the flattest growth of anything tried (+0.1) and merges worst. The two arms bind in different regimes, which is why intersecting beats choosing: on serial stills the ring threshold is 0.6x the classic floor, on this rotation sweep 2.3-6.0x. Stills are a strict no-op - 175 components against 175, identical per frame - so the +40% in stills indexing that the adaptive threshold was introduced for is untouched. What it buys, stated as one fact rather than two. Across five geometry pins spanning 1.1 mm it indexes the most frames of any arm tried, 0.831 against 0.803, and integrates 3.04 to 5.76% more observations - but those are the SAME number: regressing observation count on indexing rate over four arms leaves residuals of +/-0.7 percentage points against swings of -7 to +4.5%, so the extra observations ARE the extra indexed frames, not better data per frame. CC1/2, the only statistic here carrying per-observation quality, is +0.66 at one pin and -0.06 at the other: not harmed, not improved. <I/sigma>, ISa and R_meas cannot arbitrate on this data - across those pins each crosses zero as a monotone function of the pin. WHY an absolute contour indexes fewer frames, when its spot list is equal or better on every axis measured - recall, top-1000 recall, centroid, ice fraction, component count - is the interesting part, and it is not a detection effect at all: ITS OWN SIZE BOUND DELETES THE BRIGHTEST REFLECTIONS ON THE FRAME. A component is discarded because it grew past 200 px, and it grew past 200 px because it was bright, so the deletions are drawn from the head of the indexing budget rather than uniformly from it: they are 11x enriched in the top 250 of the thousand spots handed to the indexer, and the bound's own real deletions sit at MEDIAN RANK 12. Turning the bound off recovers 66% and 50% of the deficit at the two pins, against a bar registered at 33% before the run. Three of us dismissed this for most of a day on the grounds that the gates delete only ~4% of what is detected. That arithmetic was right and the denominator was wrong - a rate is not an impact when the thing being lost is selected for the property that makes it matter. Reworking the bound instead was measured and rejected: it recovers half the deficit, and it cannot be done without re-admitting what the bound is for - 68 components past 200 px, of which 8 are real and 60 are junk, where the intersect gets the 8 without the 60. The residual once the bound is off, +1.08%/+1.70%, is the contour itself. Component merging is ruled out separately: geometrically impossible here, 33.9 px minimum reflection separation against components spanning 10 px. So is a ranking effect - the intersect's lead runs +0.06% at --max-spots 250, +3.46% at 1000 and +14.26% at 2000, which is backwards for a selection artefact. Costs 0.48 ms per image in the finder, and 0.044 px of bright-spot centroid precision - measured convention-free, by fitting a line to a reflection's own centroid across five frames, after an XDS-referenced figure proved to be four fifths aperture convention. It also makes the compactness gate above it safe. On the absolute contour that gate is net damage, deleting 37 genuine reflections per ten frames; once the footprint stops growing nothing reaches its threshold at all. Also fixes a real but unexercised defect in PoissonThreshold, where the exact tail handed over to a normal approximation with a step. It changes nothing here: the clipped ring sigma is over-dispersed 1.2-4.9x against sqrt(mu) because it still contains diffraction, so the Gaussian arm wins every ring above mu=50 and none of the 522 thresholds move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:30:39 +02:00
leonarski_fandClaude Opus 5	844b461232	Spot finding: say honestly that the compactness gate is inert ... The commit that added it claimed the shape test was what made raising the size bound safe - "with a flat 200 alone, two battery crystals moved; with the shape test both are unchanged to three decimals". That is wrong. Those two runs came from different builds, and the comparison never isolated the gate at all. Isolated properly, gate on against gate off in ONE build: byte-identical .hkl on three rotation crystals, the strongly diffracting one included. Not merely close - identical, every digit of every merged reflection. On that crystal the gate rejects 5 to 22 of the 110 to 440 oversize components a frame, so essentially all of what that commit bought, it bought by raising the bound from 50 to 200. Keep it, but for the reason that survives rather than the one that did not. The raised bound admits components up to 200 pixels, and on data carrying ice arcs, cosmic-ray tracks or a lit detector row those are what arrives; none of the crystals measured here carry that population in quantity, which is exactly why the gate reads inert on all of them. So it bounds the SHAPE of what the larger size bound now lets through, and no experiment on these crystals can validate it - an inert gate scores identically whatever its threshold. The comment also carried a "keeps 97% of the genuine wide reflections and none of the ice arcs" that came from a simulation over component lists, not from the pipeline. Removed rather than restated: the pipeline measurement says no reflection anywhere changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:04:05 +02:00
leonarski_fandClaude Opus 5	0af109b838	Spot finding: bound how large a spot may be, not how bright ... A component of more than 50 connected pixels was discarded. Under the self-calibrating threshold that is an INTENSITY CEILING, not a size bound: the threshold is an absolute per-ring contour, so a component's area above it grows as sigma^2*ln(A/T), without bound in the peak amplitude. Measured on the strong rotation set, footprints run 3 px at 30-100 counts to 50 px above 10000 - a slope of 4.4 px per ln(peak) against the fixed local-box test's 0.8, which saturates at 13 px and never reaches the bound at all. So the brighter a reflection, the more certainly it was thrown away: 59% of the box finder's d>3 A spots were missing from the adaptive finder's list, including every one of its ten strongest, at an intensity ratio of 1.085 for those that did match. Indexed spots per image collapsed from 220 to 7. Raise the bound to 200 - CrystFEL peakfinder8's --max-pix-count, the only directly comparable number in the field; XDS has no such parameter and guards on shape instead - and ask a component above 50 pixels to be COMPACT: it must fill a fifth of the square its bounding box fits inside. A Bragg reflection is round and fills about half of that square however bright it is; an ice arc, a cosmic-ray track or a lit detector row fills a fifth or less, and those are what an upper bound was ever protecting against. Below 50 nothing is asked of the shape, so every component accepted before still is. Integer arithmetic on both sides, and on the GPU the bounding side fits in what was padding, so the device struct does not grow. The shape test is what makes the raise safe. With a flat 200 alone, two battery crystals moved: one lost a little I/sigma, and the other's de-novo lattice was NOT MONOTONE in the bound - correct at 50, 100 and 200, wrong at 150 and at 250 and above - so 200 was partly luck. With the shape test both are unchanged to three decimals. De novo at defaults on the strong set: indexing rate 0.574 -> 0.804, completeness 97.3 -> 99.6%, <I/sigma> 3.42 -> 6.07, R_meas 0.299 -> 0.249, CC1/2 0.947 -> 0.961, ISa 3.29 -> 4.13. The full 37-crystal battery, scored per shell, is still owed. Also documents what StrongPixelSet::AddStrongPixel has required since the component search became linear - pixels in raster order - and puts the existing test's insertion order into it. Both callers scan a bitmap in ascending flat index and always satisfied it; the test did not, and was the only thing that did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:49:49 +02:00
leonarski_fandClaude Opus 5	6516bc96af	Ice rings: carry the list past 1.5 A, where ice does not stop ... The eleven measured bands end at 1.522 A because their source says so in its own words - "pure hexagonal ice has 11 diffraction rings between 4 and 1.5 A resolution" - and its subject was detecting ice in deposited data, not masking it. On a detector that reaches further, the rings it does not list are the ones left in the data: on the strong rotation set just added to the battery, 44% of every image's spots sit in ice bands, and beyond 1.5 A the spot list is ice and nothing else, which is why the resolution estimate read the ice rather than the crystal. There is nothing measured to copy below 1.522 A, so the eight added bands are calculated. Enumerating hkl is not enough and the code already said so: ice Ih is P6_3/mmc with O on 4f, and most of what enumeration emits is extinguished by the OXYGEN SUBLATTICE rather than by the space group - which is why (004) at 1.830 A and (104) at 1.657 A are missing from the measured list although they sit inside its range and its reflection conditions allow them (for (00l) the structure factor goes as cos(2*pi*l*z), and z ~ 1/16 kills l = 4). So compute structure factors - oxygen only, the hydrogens being half-occupancy disordered and weak to X-rays - and keep the lines reaching 3% of the strongest. That rule REPRODUCES THE MEASURED ELEVEN EXACTLY and every line it drops inside their range computes to zero, which is what makes it trustworthy below 1.522 A. It stops at 1.170 A: below that the real lines fall to 2-3% while the extinct ones rise to about 1%, and an oxygen-only calculation cannot separate them honestly. Every added band was independently confirmed in the data - the spot-count histogram of the strong set peaks at each of them and is empty between - and every line the rule calls extinct is absent there too. Costs, measured. The bands are inert above 1.6 A: on 38 of 39 battery sets the profile ice score does not move at all, and the one that appeared to (a jet set, 1.25 -> 2.60) does not on the peak-excluded profile the score actually uses - that was Bragg peaks in the plain profile, which is what the peak exclusion is for. Where a detector does reach past 1.5 A the bands cover more of reciprocal space: unchanged at 1.6 A, +7.4 points at 1.4 A, +16.5 at 1.18 A. On the strong set that is 17% -> 27% of reflections held out of the scale fit, and it shows: the spot resolution estimate improves from 1.33 to 1.46 A against a truth near 1.42, while CC1/2 falls 98.5 -> 97.6% and ISa 3.58 -> 3.37. Ice handling only runs at all on a run that trips the ice gate, so a clean crystal pays nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 11:27:28 +02:00
leonarski_fandClaude Opus 5	fe1f3ba96e	rugnux: report the geometry in the form the broker takes it in ... A rotation run post-refines the detector distance and the beam centre, and that is usually the best measurement of them anyone has - but the only way back into the instrument was to read two numbers off the report by eye and retype them into a collection. Write them once more, as the object jfjoch_broker takes: the four required properties of dataset_settings in broker/jfjoch_api.yaml, spelled the way the API spells them, on one line of valid JSON that a script can lift with a grep and POST. The existing DETECTOR_DISTANCE and BEAM_CENTRE keys are unchanged and stay the ones a person reads. REPORT_VERSION is not bumped: a new key breaks no consumer of the old ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:49:18 +02:00
leonarski_fandClaude Opus 5	42add7c0e2	Report the strong-pixel count off the FPGA path too ... DataMessage::strong_pixel_count is written all the way to CBOR, to /entry/MX/strongPixels, to the receiver plots and to the frontend - and only the FPGA receiver ever set it. Every EIGER dataset, and every rugnux run, stored zeros. It is the one number that distinguishes an image the spot finder gave up on from an image that did not diffract, and its absence is why a defect that cost 767 of 1800 images their spots looked like a crystal that stopped diffracting for two blocks of the sweep. Fill it on the CPU/GPU path as well: the finders know the count, the GPU extractor already had it on the device, and it rides back with the spot count on the frame's one synchronisation, so it costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:46:47 +02:00
leonarski_fandClaude Opus 5	1388b16d7a	Resolution estimate: predict past the edge of the detector ... The spot-finding resolution estimate was clamped so it could never beat the detector corner. On a crystal that diffracts past the corner that reports where the DETECTOR stops, which is the one thing this number is not for - it is meant to say how far a merge of data like these would reach, a property of the crystal and the exposure. The clamp also hid the interesting case: an estimate finer than what the run actually merged is the statement "this run was detector-limited", and there was no way to make it. The statistic already extrapolates. Its quantile sits in the middle of the fall-off, well inside what the detector records, so it goes on measuring the crystal's own decay when the detector cuts that decay short. Measured by truncating the spot lists of 31 battery crystals at an artificial detector edge and scoring the unclamped answer against each crystal's own measured CC1/2 = 0.30 crossing, it holds its 8-9% floor out to about 1.7x past the cut and only then drifts pessimistic, which is the safe direction. Every genuinely detector-limited crystal in the battery needs between 1.10x and 1.63x. Against a truth corrected for censoring - the six crystals whose merge is cut off by their own detector cannot have a measured crossing, so theirs is extrapolated from multiplicity-corrected <I/sigma> and anchored on the 25 where both exist: symmetric-log RMS 13.5 -> 9.4% over 37 crystals, 25 -> 28 within 0.2 A. On the six detector-limited ones 26.1 -> 9.8% and the bias goes +19 -> -3%; on the 31 that are not, 9.23 -> 9.32%, i.e. it costs them nothing. The 0.30 tail fraction and the 2.25 reach were refit by leave-one-out against that truth and did not move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:41:34 +02:00
leonarski_fandClaude Opus 5	ac202a55a1	Spot finding: the strong-pixel limit follows the detector ... An image with 65535 or more strong pixels was given up on and reported ZERO spots - silently, no log line, indistinguishable from a frame that did not diffract. 65535 is one pixel in 64 of the JUNGFRAU 4M the number was written for; left fixed while the detectors grew it became one in 276 of an 18-megapixel EIGER, which a strongly diffracting crystal passes on its best frames. On the strong rotation set just added to the battery it cost 767 of 1800 images: peakCountUnfiltered 0 and resolutionEstimate NaN across two blocks of the sweep, the two where the crystal diffracts hardest. Make the bar one pixel in 64 everywhere, and never below the value that stood here, so no smaller detector loses ground. It lived in three places - the host extractor, StrongPixelSet, and SpotExtractorGPU's buffer capacity - now one function. The bar was there for a reason and raising it alone would not have been safe. sparseccl walks a sliding window of the last two lines and tests every pixel in it, which is quadratic in how many strong pixels a line pair holds: a handful for the silicon-tracker hits upstream wrote it for, four thousand for a lit detector line, and 76 seconds for a fully lit frame. But the pixels arrive in raster order, so the window need not be walked at all - a pixel's earlier 8-neighbours are the one to its left and the at most three above it, which is what the GPU extractor already finds by binary search. Keeping the previous line's range and a forward-only cursor gives the same edge set and the same unions in the same order, so the labels are identical, and the fully lit frame now takes 0.16 s. Verified bit-identical on real frames, on fully dense frames, across occupancy 1e-5 to 5e-2, and on 4000 randomised images including ones with blank lines; SpotExtractorGPU's host-vs-device parity test passes untouched. ImagePreprocessorBufferGPU's gather staging was sized to the old constant, with a comment tying it to the caller's give-up. Raising that give-up without it would have run the gather off the end of the device buffer, so it follows the same limit now. Byte-identical .hkl on three battery crystals that never reach the bar. On the strong set, with symmetry, cell and geometry pinned so only the spot list moves: <I/sigma> better in every resolution shell, CC1/2 97.8 -> 98.5%, R_meas 30.5 -> 28.4%, ISa 3.36 -> 3.58, indexing rate 0.772 -> 0.824. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:38:10 +02:00
leonarski_fandClaude Opus 5	a6fbd1d195	Viewer: Alt and the wheel step through the images ... The wheel already zooms, and with Ctrl or Shift it moves the foreground; Alt now moves through the dataset, one image per notch, wheel up forward - the direction QAbstractSlider's own wheel handling uses, which is what the toolbar's scrub slider follows. The view does not know which image it is showing, so it emits the step and the navigation toolbar applies it: the same loadImage() every other control ends in, with the same clamp and the same Sum setting, so nothing about loading is duplicated. An empty dataset is left alone, because loadImage(-1) does not mean "before the first image" but "the latest one" when the viewer is following a running collection over HTTP. The signal is on JFJochImage, so the other views emit it too; only the diffraction view is connected, which leaves them as they were. Note for a desktop where Alt+wheel does nothing: many window managers grab Alt-modified mouse events before the application sees them, and that is a window-manager setting, not something the viewer can take back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:18:19 +02:00
leonarski_fandClaude Opus 5	8999287ba4	Test: a failed calibration now leaves the machine in Error, not Inactive ... JFJochStateMachine_CalibrationFailure was written in rc.164 against the behaviour of the time. rc.165 moved every calibration FAILURE path - CalibrateDetector's catch, TakeDarkMaskInternal's mask-size check and the three ImportPedestal checks - from Inactive to Error, so that /wait_till_done and /wait_until_running answer with the reason instead of the bodiless 502 that a deliberate Deactivate() also leaves; only a CANCELLED calibration still ends in Inactive. The test kept asserting the old state and started failing. Assert Error. The rest of the case is unaffected: the message and severity are what the new SetState writes, and Start() still throws from Error because it requires Idle, so "a data collection is refused on it" keeps its meaning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 10:01:49 +02:00