208 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	7bb74a9dc8	review fixes: two silent wrong outputs, and one rule for a quantity nobody measured ... From four whole-branch code reviews of rc166. Reviewing the net diff rather than the commits found what per-commit review structurally cannot: a later commit leaving an earlier one's claim standing, and two cases of a later commit quietly undoing an earlier one. THE FITTED RESOLUTION WAS THE P1 CROSS-CHECK'S. The block that merges in P1 to write <prefix>_P1.mtz saves and restores the error model around itself, because that merge is not the run's answer. A later commit taught the same function to report the CC1/2 resolution fit and did not extend the list, so every de-novo rotation run in a non-P1 group has been quoting FITTED_RESOLUTION - the number the report itself calls the one to quote - from a merge with n_ops times the unique reflections at a fraction of the multiplicity. AND IT ROUND-TRIPPED THE SPACE GROUP THROUGH ITS NUMBER. A number names only the reference setting, which stopped being enough when the search learned to adopt P 1 1 2(1) or I 1 2 1. Everything written after that block - the unmerged MTZ included - therefore carried the reference setting while the merged file carried the adopted one: two files describing one dataset in two different settings. It now carries the group. A PONI CANNOT STATE A MIRRORED OR QUARTER-TURNED DETECTOR, and the fits became orientation-aware on this branch while the writer did not. It wrote five numbers that silently described a different geometry from the one measured; it now refuses, and says the JSON beside it has the full one. A REFUSED FIT'S ERROR BARS COULD BE HANDED BACK AS AN ACCEPTED GEOMETRY'S. RingOptimizer::Run writes its uncertainty only where the solve is usable, and CalibrateFromSpots runs it twice - tilt free, then tilt pinned. A failed second fit kept the first's sigmas, valid flag and all. It is cleared on the way in. ONE RULE FOR NOT MEASURED. The report had four conventions for it and printed the same missing quantity two ways on adjacent lines: SIGANO as the literal "nan" and CC_ANOM by absence, for a Friedel-merged run that split no Bijvoet pair - which is the default. A quantity a run did not measure now writes no key, and the shell table's dash follows the same rule rather than a 0.0% that reads as a measured total failure. ANISOTROPY_D_MIN_BEST also stops printing nan when only its first principal direction is unmeasured. Four claims that a later commit made false are corrected where they stand: the merge header promising an order-independence the balancing rule gave up, the reference page arguing against CCanom 28 minutes before it shipped, the screw threshold whose "three dead reflections clear it" the evidence floor caps at 19.2 nats, and a shell comment calling equal width in 1/d^2 equal volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 17:27:56 +02:00
leonarski_fandClaude Opus 5	8661193712	merge: balanced half-sets by rank, and CCanom read off them ... Two statistics and the machinery they share. THE SPLIT. CC1/2 correlates two half-set means, so a reflection whose observations all land in one half has no second mean and contributes nothing. The half was a hash of the image index alone, which does that to 2^(1-n) of the reflections at multiplicity n: half of the doubly measured ones, a quarter of the triples. Measured on in-house rotation data that is 0% of a 26-fold redundant sweep but 6-30% of a 3-to-5-fold one, and 35-60% of its outer shell - the shell the automatic resolution cutoff is read from. An observation's half is now the PARITY OF ITS RANK among its reflection's observations, ordered by a key built from the raw Miller index and the peak frame. That is exactly balanced - floor(n/2) against ceil(n/2), the split cctbx's compute_cc_one_half uses and phenix.merging_statistics through it - and, because a rank is a property of the set rather than of the order it is walked in, it is the same on every path. A sequential "put it wherever the counts are more even" rule is balanced too but not that: the device holds the fulls in emit order and the host in frame order, and on three test crystals that alone moved CC_HALF between a CUDA and a JFJOCH_USE_CUDA=OFF build by up to 0.0077, with 4 to 12 shell rows differing. Both now agree bit for bit, on the whole scaling-and-merging section, while the radiation-damage B in the same report still differs between the builds - so the agreement is a property of the split, not of the two pipelines being identical. It is also stable across repeated runs and across -N 1, 4 and 12. Ranking is quadratic in a reflection's multiplicity - tens - so it is one multiplicity-weighted sweep of the observations, assigned once per pass and read by both the host loop and MergeAccumKernel. The kernel no longer decides anything, which is why the two cannot drift apart. CCANOM. SigAno was the only anomalous quality statistic reported, and it is a ratio against the error model: an optimistic sigma raises it, and it cannot separate real anomalous signal from an underestimated sigma. CCanom carries no sigma. For each acentric reflection the anomalous difference is formed twice, once per half, and the two are correlated over every pair where both hands split into two non-empty halves - per shell, and overall as one correlation rather than a mean of shells. The halves for it are balanced within each MATE, not within the reflection: a mate left entirely in one half loses the whole pair, and balancing per pair instead measured 0.223 where per mate gives 0.260. It hangs on the anomalous accumulation that already existed for the I(+)/I(-) export, which runs whether or not the merge is Friedel-averaged - so CCanom is reported on a default run too, and -A only changes the counting basis. The whole-mate sums SigAno and the reflection files read are the two halves added back together, so neither moves. Against external programs on the same observations: AIMLESS 0.264 and cctbx 0.266 where this gives 0.260, per shell within about a point. XDS's CORRECT.LP has a column named `Anomal Corr` which is NOT this quantity - two to three times larger in the low shells, with a total below every one of its own shells - so the report says so where it describes the key. On data with no anomalous signal and around two observations per mate the statistic is unstable and goes strongly negative; phenix reads -0.83/-0.45/-0.42 where this reads +0.12/-0.33/-0.41 on the same file, so that is the statistic, not this implementation, and it is reported as measured. REPORTING. A shell prints `-` where it could not measure a quantity instead of `nan` - which every run printed for CCref, and a multiplicity-1 shell printed for most of the row - and CC_ANOM is omitted from the report rather than written as a placeholder when no pair could be split in both hands. A dash and an absent key are the same statement; a measured value always prints, including a negative one. Reported CC1/2 moves, so baselines keyed on CC_HALF need regenerating, and the automatic resolution cutoff reads the same statistic and can move a shell edge with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 17:07:32 +02:00
leonarski_fandClaude Opus 5	0c3e1462ef	merge: the completeness denominator is the declared range, not the surviving one ... CalcPossibleReflections was handed d_min/d_max derived from the reflections that came out of the merge, so a loss at either extreme took the numerator and the denominator with it. At the high end that is right: d_min is the finest d reached anywhere and the denominator is the full sphere down to it, so anisotropic loss shows. At the low end it was a tautology - d_max was the coarsest reflection that happened to survive, so anything the beam-stop shadow mask (on by default), a detector mask or the low-resolution limit itself removed left the denominator along with the data and could not be reported as missing. Both statistics paths now bin, and count, between the DECLARED low-resolution limit and the finest d reached: MergeOnTheFly::MergeStats (stills) and RotationScaleMerge::MergeAndStats (rotation). The grid and the denominator keep sharing their bounds, so no possible reflection falls outside a shell. An undeclared low limit is the whole sphere - 1/d^2 down to 0 - spelled as an infinite d_max, which ResolutionShells already handles and which gemmi's for_all_reflections special-cases; the change therefore reads correctly whether or not the 50 A default stays. The innermost shell keeps a finite d_max label, falling back to the coarsest reflection measured when the bound is infinite. This makes the shell boundaries the ones the integration document already claims: XDS lays its nine 1/d^2 bins between INCLUDE_RESOLUTION_RANGE's two values, not between the extremes of the surviving data, and counts POSSIBLE against the declared low limit - which is why its innermost shell reports the beam stop's loss. Verified against a CORRECT.LP: all nine boundaries reproduce to the printed precision from the declared 50 A, and not from the coarsest observed reflection. Measured on stored merges of seven rotation datasets, small-molecule and protein, re-scaled with --mode scale: the overall denominator moves by 0 to 2 reflections out of 70,000-100,000, because on every one of them the coarsest reflection the declared limit allows was itself measured - the corpus has no dataset whose stop eats a whole low-resolution class. What does move is the shell grid: the innermost boundary shifts by 0.1-0.4% in d (e.g. 7.21 -> 7.22 A), which changes the innermost shell's counts by up to a few per cent and its R_meas by around 0.1 percentage points. Stored battery baselines for rmeas_lo must therefore be regenerated, not compared across this commit. Two decisions read merged completeness (the two-pass wrong-cell guard, which only fires above 100.5% and only under -S); a larger denominator can only lower the figure, so the guard can fire less often, never more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 16:39:40 +02:00
leonarski_fandClaude Opus 5	d5cf3a6e58	docs: worked Phaser and SHELX routes, and the MTZ base dataset they need ... Both programs were run on merged rugnux output and the commands are the ones that worked, error messages included. Phaser takes myrun.mtz with no LABIN, picking IMEAN/SIGIMEAN and the cell, group and resolution out of the file; MODE MR_AUTO already searches both hands of an enantiomorphic pair and returned the hand opposite the one in the header on a tetragonal test. SGALTERNATIVE SELECT ALL covers the screw variants the report lists as indistinguishable, MODE CCA prints the list without searching, and a wrong point group has to be re-merged instead. mmCIF is refused by 2.8.3 in both the CCP4 and the phenix build; gemmi cif2mtz gives the same solution. SHELXC reads only myrun.hkl - it refuses an MTZ and exits 0 while doing it - and needs CELL and SPAG repeated, since HKLF 4 carries no metadata. A default rotation merge already writes the Bijvoet split, so no flag is needed and --no-export-unmerged hides nothing. On a 5 keV cubic sweep SHELXD separated a space-group pair the merged intensities could not (CFOM 69.4 against 52.0) and SHELXE solved it, 42.9 % against 15.3 % autotrace CC between the two hands; a tetragonal sweep at lower completeness and multiplicity did not separate at all, and that is recorded too. Verifying it exposed one defect. The merged MTZ was written with its data dataset at id 0, the id MTZ reserves for HKL_base, so mtzlib dropped the wavelength and mtzinfo reported the 1.54187 A default - the wrong edge for anything taking f'/f'' from the file. Writing HKL_base first, as the unmerged writer already does, puts the real wavelength back; Phaser's solution is unchanged either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 13:07:03 +02:00
leonarski_fandClaude Opus 5	7b1caa6ce5	mmcif: the free set is _refln.status = f, not a column of our own ... The merged mmCIF carried the R-free flag in `_refln.status_free` as 1/0 and wrote `o` into `_refln.status` on every row. `_refln.status_free` is not in the PDBx/mmCIF dictionary - it is absent from mmcif_pdbx v4.0, v5.0, v5.288 and v5.362, from CCP4's and phenix's shipped copies, and its wwPDB item page is a 404 - and no deposited structure-factor file uses it. `_refln.status` is the item that carries the free set, with `f` for a test reflection and `o` for a working one; on a deposition that also carries `_refln.pdbx_r_free_flag` the two agree exactly. Measured on a real merged file this run wrote: CCP4 cif2mtz refuses the file outright - "Unexpected context type for category REFLN" from its dictionary-validating parser, exit 1, a 12-byte truncated MTZ. Dropping the non-dictionary column is what fixes it: the same file without it converts. gemmi converts, but its cif2mtz spec knows only `status` and `pdbx_r_free_flag`, so FreeR_flag comes out 1 everywhere and the free set is silently lost - R-free would then be computed on the working set. phenix worked, but only by a filename heuristic matching the words "status" and "free". Writing `f` while keeping the extra column is worse than either, because phenix then finds two candidate free-set arrays and refuses the file, so the column goes in the same change. After it, all three read the same 5% test set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 10:53:31 +02:00
leonarski_fandClaude Opus 5	ef5da29319	calibration: a fit that hands back the file's own tilt is not a calibration ... A powder calibration is run because the file's geometry is in doubt, so a fit that quietly returns part of that file has answered nothing - and it is indistinguishable from one that worked, down to the residual and the sigmas arranged around it. On one of four LaB6 exposures of one detector the tilt came out at 2.93x its own sigma, a hundredth under the significance gate, so it was declined and pinned - at the master's hardcoded rot1 -0.08, rot2 -0.22 deg. That is eight times the tilt just refused, on no evidence, and worth 10 px of PONI at 190 mm. rugnux printed it to four decimal places, wrote the .poni, and exited 0. Judge the result on provenance instead of on any residual: a geometry is a measurement only if every parameter in it came from this data. Two ways out of the fits do not qualify - a covariance that never conditioned, so the fit cannot say what it determined, and a declined tilt pinned at a non-zero value from the file. A declined tilt over a file stating no tilt still qualifies, because reporting no tilt is then exactly what was measured; so does --no-refine-tilt, because a hold that was asked for is a stated choice and not a silent substitution. No single number separates the four. rms is 2.465 px against 1.44-1.64; the significance of all four lies between 2.93 and 4.47, so the gate is nearly a coin flip at these distances and moving it would only recalibrate on one population; and the failed fit has the TIGHTEST parameter sigmas of the set, because pinning the tilt removes the tilt/centre correlation that inflates a good fit's. The spot cross-check reads 13.5 px against 0.98-2.66, but 10.4 px of that is the pinned tilt moving the PONI - the same defect one step downstream, not independent evidence. On a failure rugnux says so, writes no .poni - a PONI file states where the detector is and has no field in which to say it does not know - writes the JSON with converged false and the reason beside it, and exits non-zero. The re-binning pass now prefers a converged refit over a non-converged one whatever its residual, so a tilt an earlier pass measured is not what a later one gets pinned at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 09:37:46 +02:00
leonarski_fandClaude Opus 5	6081b6bc43	docs: credit the L test, FFT indexing, TORO, Niggli, peakfinder8 and SparseCCL ... Six methods the pages name or describe carried no citation: Padilla & Yeates (the L test), Steller, Bolotovsky & Rossmann (the projection/FFT autoindexing MOSFLM implements), TORO (what ffbidx implements), Krivy & Gruber and the ITA lattice-character table (the reduction and Bravais assignment), Cheetah's peakfinder8 (the per-ring background statistics of the adaptive finder) and Hennequin et al.'s SparseCCL (already credited to traccc, now also to its authors). Each gets its ACKNOWLEDGEMENT.md paragraph, a References entry in CPU_DATA_ANALYSIS.md, and a one-line credit at the algorithm. The Sheriff & Hendrickson / Popov & Bourenkov entry is re-scoped so each claim sits on the paper that supports it - P&B 2003 is titled, and credited for the sigma-aware anisotropy estimation its statistic modelling contains, not for the tensor and its constraints. All DOIs verified against the publishers; the SparseCCL DOI resolves to IEEE document 9049184 (IEEE blocks content scraping, so verified by the resolved document id plus two independent sources). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:36 +02:00
leonarski_fandClaude Opus 5	47ec939c39	merged MTZ: FreeR_flag in the convention its own column name carries ... The column was written with 1 for the test set and 0 for the working set - the phenix/CNS numbering - under FreeR_flag, which is CCP4's column name. CCP4's own freerflag writes 0..19 with 0 as the test bin, and REFMAC5's default FREE 0 reads it that way, so REFMAC5 stopped on every file we wrote: "more than half of reflections are in free R set", then "Cannot switch free R flag", exit 1. phenix auto-detects either numbering with equal confidence (measured on both, score 3 each), and rugnux's own reference-MTZ reader already takes flag 0 as the test set, so 0 = test is the numbering that works everywhere and the one the rest of the code assumes. This changes every merged .mtz we write: a script that reads FreeR_flag == 1 as the test set has to be inverted. The mmCIF's _refln.status_free is a separate item with its own convention and is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	762c093b10	unmerged MTZ: export the events the merge would keep, not every event ... The export declared its rows FULL - LDTYPE=2 in the batch header, M=0 in M/ISYM - while filtering them on --min-partiality alone (0.02), where the 3D combine also applies --min-captured-fraction (0.7 on rotation) to the same summed event. AIMLESS reads the FULL declaration, reports "all runs have only fulls" and ignores FRACTIONCALC, so an event that caught a twentieth of its rocking curve entered the merge whole, with a small sigma, and was weighted heavily. There was no cut for the reader to make: the column that would let it make one is the one the reading program has been told to ignore. Dropping those events moves AIMLESS's Rmerge at 1.8 A from 1.353 to 0.694 and CC(1/2) from 0.985 to 0.993. FRACTIONCALC itself is unchanged, values above 1 included. Each part's partiality is the erf pair the predictor computed on that frame, from that frame's own refined lattice and mosaicity, so the parts of one event do not tile the rocking curve exactly - the offset steps by the wedge to within 12% of it, and the sums land in a peak at 1.000 whose 95th percentile is 1.09. It is an honest estimate of a captured fraction, and it is not the number rugnux scales on, so clamping it would only hide the spread. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	b33cae9068	unmerged MTZ: a batch header for every image the sweep spans ... The header set was built from the batches that produced an observation, so an image that indexed nothing left a hole in the phi series. AIMLESS starts a new run at such a discontinuity: measured on a 360 degree sweep with 67 unindexed images it made 23 runs of one sweep, its scale model diverged, Rmeas overflowed to -1266 and the result no longer correlated with rugnux's own merge (Pearson 0.0018). Renumbering the batches contiguously does not help - the split is on phi, not on numbering - so the fix is a header per image over the span the observations cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:22 +02:00
leonarski_fandClaude Opus 5	fb07263025	predict: integrate every centring node, so a fixed space group costs no data ... With -S the prediction rejected the fixed group's centring absences, so those reflections were never integrated. Two things followed, and only the second was known. The P1 cross-check was withheld on such a run, because a P1 merge missing whole centring classes is misleading rather than merely small - 50% of the nodes on an I lattice, 75% on F, 67% on R. That was the documented reason and it was right. The unknown one is that it cost intensity accuracy. Every predicted reflection marks its signal region so a neighbour's background ring can exclude it (BraggIntegrationEngineCPU, the reflection mask); an unpredicted node is an unclaimed patch of detector, and the neighbouring reflections sweep those pixels into their background and over-subtract - worst at high angle, where the background dominates. On a fixed F-centred group that is three quarters of the nodes: measured against the de-novo run of the same data, <I/sigma> 16.07 against 17.21, CC1/2 0.9862 against 0.9895, ISa 10.93 against 11.92. Predicting them costs nothing downstream, because both merges already decide absence against the group they are merging in: the run's own merge drops them again, and the P1 cross-check keeps them because P1 has none. One integration, two correct merges. The -S output becomes byte-identical to the de-novo run on the five crystals measured, which is the point - pinning a group should not change the answer - and such a run can never be slower than de novo, since it predicts the same reflections and additionally skips the space-group search. The de-novo path is untouched by construction: it already predicted in P. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-02 09:18:13 +02:00
leonarski_fandClaude Opus 5	ecb0571d8d	integration: the observed centroid is the signal's, not the disk's ... The centroid was a first moment of the RAW counts over the signal disk, so it weighted signal plus background. The background is flat over a disk centred on the PREDICTION, which makes its own centroid the prediction exactly: it adds nothing to the displacement and everything to the denominator, and the measured offset comes out shrunk by I/(I + n*bkg). That factor is worst where the background dominates the signal, which is at high resolution - so the one consumer of this quantity, the geometry post-refinement, fits the beam centre and the detector distance on displacements that are systematically too small, by a factor that varies with resolution. An estimator whose bias depends on the very coordinate it is correcting. Subtracting a flat pedestal from a first moment is exact, and the background is not known until the ring has been read, so the positions of the pixels behind the intensity sum are accumulated alongside it and the correction is applied afterwards: sum(x*(px-bkg)) = sum(x*px) - bkg*sum(x). Both engines, identically. Where nothing rises above background there is no signal centroid to compute and the raw one is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 19:55:47 +02:00
leonarski_fandClaude Opus 5	9c19646f6a	refine: fit the direction of the goniometer axis, not its length ... The residual applies angle_rad * |rot_vec| and rot_vec is a free three-vector, so the first pass has been fitting a goniometer rotation SCALE nobody asked for. GoniometerAxis::Axis() then normalises it away on write-back, and RotationIndexer scores the candidate with the normalised axis - so the cell that won the fit is judged under a rotation model the fit did not use. Measured over 43 rotation datasets: the length reaches 1.2%, and the fit-vs-score disagreement a median 0.124 deg and up to 6.19 deg of goniometer angle, against rocking widths of 0.05-0.36 deg. That score picks the lattice class, which nothing later revisits. The fitted length is not a usable measurement of anything either: on synthetic data it recovers 54% of a known scale error, repeated first passes on one dataset disagree with each other in sign, 26 of 43 datasets disagree with themselves, and on the one dataset with a proven 1.3% stage fault it comes out negative. It is absorbing other systematics. The rotation scale is measured properly, once, with cross-validation and gates, in PostRefine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-09-01 18:36:32 +02:00
leonarski_f	d32c8d526b	geometry: make which tilt component is restrained two numbers rather than a shape ... Both directions of the tilt now go through the same restraint, with a budget each in shared direct-beam pixels; zero means free. Swapping which component is held, or holding both, is a change to two constants. The perpendicular budget is zero today, which is the arrangement the measurements support: it is the component whose conditioning tracks the 2theta the fit reaches, while the parallel one's does not move with 2theta at all - a conditioning number that ignores the data is the prior talking. This is a clarity change, not a precision one, and the record should not read otherwise: the standing 'freeing the tilt inflates both beam components' cost is the perpendicular component's doing, not the parallel one's, so restraining the parallel one recovers essentially none of it. What it buys is that a beam-centre error is no longer reported as an angle.	2026-09-01 15:34:06 +02:00
leonarski_f	657b6c5d1a	geometry: say that the gauge argument is local to the fit that co-refines the orientation ... The restraint is correct here because the crystal orientation is free alongside the tilt and absorbs the spindle-parallel difference. A stage that freezes the orientation has no such compensator, so whether that component is measurable THERE is a separate question; the comment no longer reads as a claim about the tilt everywhere.	2026-09-01 15:33:51 +02:00
leonarski_f	c0bb159c98	geometry: take the gauge direction in the frame each parameter lives in, and tighten the tilt's budget ... Two corrections to the restraint added in the previous commit, both measured. The beam prior compared the goniometer vector's LABORATORY components against an index into the PIXEL-frame beam centre. det_matrix is PoniRotMatrix * DetectorOrientation::Matrix(), so on a quarter turn of 1 or 3 the pixel X axis is the laboratory Y axis and the comparison picks the wrong component - pinning the determined one and freeing the gauge one, which is worse than having no prior. Two datasets in the corpus are in that state. The direction now comes from projecting the spindle onto the pixel axes' own laboratory images, which is exact for any orientation, any tilt and a spindle at any angle, and equal to the old comparison when the orientation is the identity. The tilt keeps the spindle's laboratory components, because rot1 and rot2 are laboratory rotations applied outside that orientation matrix - same physical direction, each in the frame its own parameters live in. And the tilt's restraint is three times tighter than the beam's, in shared direct-beam pixels, because the data determine the SUM of the two: at equal budgets the shift splits evenly and the reported tilt still followed the starting beam centre at 41% of one-for-one. At a one-pixel budget it follows at 8%, from 79% before this work, while the free component moves by 0.0035 deg over the same eight-pixel swing and the indexing rate does not change. The comment says plainly what the change does not do: it does not make the tilt accurate.	2026-09-01 15:33:51 +02:00
leonarski_f	e441434644	geometry: restrain the spindle-parallel tilt, the same gauge the beam prior already restrains ... The beam prior at XtalOptimizer already treats the beam component parallel to the spindle as the gauge direction of a single-axis rotation experiment and restrains it toward the value it was handed. The detector tilt is that same gauge described a second time, and it was left free with a flat +-3 deg box. The correspondence is not an observation about one beamline, it follows from the convention: PoniRotMatrix builds the detector matrix as R(-rot3,z)R(-rot2,x)R(rot1,y), which puts the direct beam at (beam_x - rot1*D/pixel, beam_y - rot2*D/pixel). rot1 IS beam_x written as an angle and rot2 IS beam_y, component for component. So on a horizontal spindle the gauge tilt is rot1 and rot2 is refined; on a vertical spindle it is the other way round. With only one end of the alias restrained, a beam-centre error the prior refuses to absorb reappears as an angle: measured elsewhere in this campaign, refined rot1 tracks the starting beam centre at 0.072 deg per pixel against a geometric 0.080, while the refined beam never leaves its anchor by more than 0.24 px. Restraining both ends, in the same direction and to the same three-pixel budget, leaves the determined component - the direct beam, and the tilt perpendicular to the spindle - alone. Both restraints now take their direction from one projection of the spindle onto the detector plane rather than from two independent snaps to whichever of X/Y dominates, so they cannot disagree, and a spindle at any angle is handled. On a spindle along a detector axis the projection is the snap.	2026-09-01 15:33:51 +02:00
leonarski_fandClaude Opus 5	f98442077e	post-refine: bound the beam move on the measurement, not on the header ... PostRefine step B refuses a fitted beam centre more than 15 px from the geometry the run started from. Two things were wrong with that and they compounded. The 15 px was BOTH the Ceres search box and the acceptance threshold, so a fit that wanted to move further was pinned at the box face, landed at exactly hypot >= 15.0, and was then refused for being there. The gate never saw the fit it was judging. Measured on one crystal at three detector distances, the fit came back at 15.128, 15.133 and 15.130 px - the box corner, three times. And it was measured from the header, which is the value most worth correcting exactly when it is most wrong. The run already knows better: the default beam-centre check fits the centre from the isotropy of the scattered background on every rotation run, reported it, and then discarded it. The bound is now measured from whichever of the nominal centre and that measurement is nearer - accept a move that is small relative to something independent, rather than small relative to the header alone - and the search box is widened to match, so the gate decides rather than the box. The measurement is used only where it was precise enough to be adopted as a centre in its own right; a run without one is bounded exactly as before. Step B also now says WHICH of its four tests refused a fit, since all four leave the same geometry behind. Measured over 113 rotation datasets against the same corpus, one binary per arm, with a control arm that reproduced the stored battery on 112 of 112 reports: the bound fired and refused a real correction 1 6ukf the bound fired and caught a runaway 0 the bound never fired 112 On 6ukf the header centre is 22.1 px from what the background fit measures. The old fit, pinned at the box, improved the held-out positional residual 9-fold and was refused; the new fit lands 0.46 px from the measurement and improves it 418-fold. ISa 4.42 -> 6.71, R_meas 18.5 -> 12.8 %, CC1/2 0.987 -> 0.995, d_min 1.119 -> 0.973 A; at matched resolution, 1.68 A goes I/sigma 8.9 -> 15.1 and R_meas 28.8 -> 13.4 %. The refined cell moves from 0.4 % off the deposited one to under 0.1 %. It survives perturbation, and inverts it: over +-0.5 mm of detector distance the corrected run is flat (ISa 6.62 / 6.71 / 6.94) while the uncorrected one is bistable (4.02 / 4.42 / 6.90). The change removes an instability rather than exploiting one. No space group anywhere in the corpus moves, and no other dataset's report changes at all - 112 of 113 are identical. The distance half of the gate is untouched: it is separately argued, and it is what legitimately refuses the one dataset whose distance wants to move 1.05 %. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 20:19:36 +02:00
leonarski_fandClaude Opus 5	efca844bce	calibration: --no-refine-tilt holds the header tilt, it does not zero it ... GuessInitialGeometry resets rot1/rot2/rot3 to zero before seeding, and the branch that puts the header's tilt back was guarded by `refine_tilt && !tilt_refined`. With --no-refine-tilt the tilt is never free, so tilt_refined is false, so the guard is false, so the restore never ran - and the fit reported a zero tilt while the CLI printed "held at the header value". The guard only needs !tilt_refined: a tilt that was declined and a tilt nobody asked to refine both want the header put back and the beam centre and distance refitted around it. That is what the branch already does. Measured on a LaB6 sweep with a tilt imposed on the command line and asked to be held, --calibration spots: before Rot1= +0.0000 deg (+0.001400 rad from the header) after Rot1= -0.0802 deg (+0.000000 rad from the header) The rings path reaches this only when the spot-derived start wins, which is why it does not show on a file whose profile start is chosen; the spots path always did. A calibration whose header tilt is already zero is unaffected - checked, byte-equal distance either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 19:47:02 +02:00
leonarski_fandClaude Opus 5	8691bf4876	calibration: write the result as JSON, not only as a PONI file ... A calibration run produced one file, and it was a pyFAI PONI - which pyFAI and its neighbours read and nothing in this system does. Carrying the answer back into the instrument meant a person reading numbers off a printed report and retyping them into a dataset_settings body, and the report is where the two points a "beam centre" can mean are easiest to confuse. So write <prefix>.json beside it. Its "dataset_settings" member holds the geometry under the property names broker/jfjoch_api.yaml gives them and holds nothing else, so it is a valid dataset_settings body as it stands: curl -X POST -H 'Content-Type: application/json' \ -d "$(jq -c .dataset_settings det.json)" http://broker:5232/start beam_x_pxl is the PONI, as everywhere here. The three poni_rot*_rad ride along whenever any is non-zero and are left out when all are zero: a body without them does not leave the tilt unstated, it states a FLAT detector, so they travel together or not at all - the same rule the report's JFJOCH_DATASET_SETTINGS block already follows. The "calibration" member holds what the run knows about that geometry: the residual, the fit's own sigmas and the correlation between the tilt and the beam centre, whether the tilt cleared its significance test or was declined and pinned, where the direct beam lands, and where the spots independently put the beam. A calibration that has gone wrong looks exactly like one that has not until those are read, and a machine-readable file that carried only the geometry would be the easiest possible way to feed a bad one into an instrument. Tested against the model generated from the spec rather than against a list of field names written out by hand: the file's dataset_settings member is parsed into org::openapitools::server::model::Dataset_settings and validated, so a field renamed or newly required in jfjoch_api.yaml fails here rather than at someone's POST. The tilted and untilted branches are both covered, and the PONI/direct-beam distinction is asserted rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:47:51 +02:00
leonarski_fandClaude Opus 5	16b368739e	calibration: re-integrate the images about the geometry that was fitted ... Everything the ring fit reads was binned with the geometry the run started from, and a calibration is run precisely because that geometry is in doubt. Binning is not something a later fit can undo: which pixel landed in which (q, phi) bin was decided when the images were read. Get the distance wrong and the radial sampling is the wrong scale - a run recovering 110 mm from a 250 mm header ended at rms 1.07 px where the same data from a right header gives 0.42. Get the centre wrong and every ring is smeared across its own sectors, and past about a hundred pixels the fit leaves rms 4.8 px even when it is started from the exact answer, because there is nothing left in the profile to fit. So integrate the images a second time, binned about what was fitted, and fit that. A calibration run is a handful of images and the answer is worth far more than the extra read. Re-binning about the FIRST fit is not enough on its own. Where the beam centre was badly out that fit is itself wrong, and binning about it digs deeper - rms 3.75 -> 5.66 on an exposure 200 px off. The spots' geometry is the one that does not degrade there, having never read the header, so both are tried where they differ and whichever comes back better is kept. On that exposure the first candidate gives 288 points at rms 5.66 and the second 369 at 0.81. The second pass is taken only if it is actually better, by the same rule that ranks everything else here - at least half as many ring points and a smaller residual - and it is skipped altogether when the fitted geometry moves a ring by less than the radial width of one profile bin, since re-binning would then put every pixel back where it already is. On the 110 mm exposure with its own header that is 1.77 px against a 1.84 px bin, so the run is untouched and its answer bit-identical. Measured on the 110 mm LaB6 exposure, true PONI x 765.90 at 110.03 mm. The calibration is now independent of the header it was given: beam-x out by +20 +40 +100 +200 +400 px -> 765.9-766.6, 110.02-110.05 mm 250 mm header, beam-x 780 -> 765.998 at 110.037, rms 0.394 250 mm header, beam-x 867 -> 765.723 at 110.038, rms 0.413 900 mm header, beam-x 1167 -> 766.496 at 110.038, rms 0.804 30 mm header, beam-x 967 -> 766.486 at 110.039, rms 0.807 All of those failed before this, most of them catastrophically. The residual inflation is gone with them: the 250 mm case now leaves 0.394 px, better than the same data from its own correct header. The four other datasets are unchanged where the re-bin fires, and slightly better where it does: rms 0.433 -> 0.423 at 150 mm and 0.555 -> 0.550 at 200 mm, with the geometry moving under a tenth of a pixel. A run that needs the second pass costs about twice one that does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 18:17:17 +02:00
leonarski_fandClaude Opus 5	49dd01331b	calibration: let the spots vote on the geometry in ring mode too ... The rings and the spots fail in different regimes, which is the whole reason to carry both. The header is right on a well-configured instrument and is the thing a calibration is run to check. The profile's own estimates are exact while the error stays small and stop meaning anything beyond that - past about half a ring spacing each ring reaches the azimuthally averaged profile as two horns rather than one peak, and the distance search reads a list of horns as a list of rings. The circle through the spots reads nothing from the header at all: measured on a 110 mm LaB6 exposure, --calibration spots returns the same geometry from a header 400 px out in the centre AND eight times out in distance. So ring mode now finds spots as well - half a second - and offers what they make of the geometry as one more starting hypothesis, fitted like the others with the residual left to choose. It is added whole rather than as a centre alone: GuessGeometry votes for the circle centre, clusters the radii into rings and takes the distance from the innermost, and those two belong together. Taking only its centre would not have helped, because the distance candidates come from a profile averaged about the header's centre, and a ring smeared over hundreds of pixels cannot be un-smeared by reading its bins differently. Measured on the 110 mm exposure, whose true PONI is 765.90 at 110.03 mm. A header 40 px out in the centre now lands within 0.6 px (it landed 41 px away before). The cases with BOTH wrong, which failed before this and equally before the beam centre work, now come out: 250 mm / 780 px gives 110.041 mm and 766.10 px against 42.3 mm and 782.7; 250 mm / 867 px gives 110.065 and 765.48. All five datasets are unchanged from their correct headers and the distance still recovers from any header between 25 and 1200 mm. Past about a hundred pixels nothing rescues ring mode, and the reason is the profile rather than the seeding: binned about a centre that far out it shows each ring smeared across its own sectors, so even started from the exact answer the fit leaves rms 4.8 px and drifts. Re-binning would fix it and would need the images read a second time; --calibration spots, which never touches the profile, already covers it. That regime is now visible rather than silent. The spots' beam centre is printed beside the fitted one as a cross-check - two methods sharing no assumption, so a reader can see at a glance whether they agree. It costs nothing, the spots having been found already, and it separates cleanly: 0.3-0.4 px on the good runs against 201.4 px on the exposure whose profile could not be fitted at all. Reported as a fact and not gated on, since at long distance both methods weaken together and the honest thing is to show the number (5.2 px at 300 mm, 12.6 at 500). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:58:26 +02:00
leonarski_fandClaude Opus 5	c84b91be8a	calibration: take the beam centre from the rings too ... The header's beam centre was the last input the ring fit had to be roughly right about. Each ring is looked for in a window a few pixels of radius wide, and a centre wrong by (dx, dy) puts a ring at a different q in every sector, so past about ten pixels the ring leaves that window over much of the turn - and the fit then reads its cos(phi) signal off whichever sectors are left, which are the ones where the signal is weakest. A 20 px error ended 31 px wrong. The rings answer this without a calibrant and without a distance. A powder ring is a conic centred on the beam, so a wrong centre makes EVERY ring's radius oscillate once per turn by the same amount: r(phi) = R + dx cos(phi) + dy sin(phi), solved directly and pooled over every ring the profile shows, with each ring searched about its own measured radius rather than about where a standard says it should be. Using it needs the extraction to follow the rings sector by sector, which is what ProfileRingTrack now does - exactly, and in all five parameters at once, by walking the ring in the geometry believed true and asking the binned geometry what q and azimuth it would have given each point. That replaces the flat-detector distance correction it grew out of. Following the rings is not free, and the reason is worth stating: a window that moves with phi makes every systematic of the peak finder - where the background line is taken, how the centroid sits in the window - vary with phi as well, and phi is exactly the axis the beam centre is read off. Measured, it costs rms 0.415 -> 0.525 px on a good 110 mm fit, and 0.831 when the window follows the fitted tilt too. So a second measurement is taken with a window that is the same in every sector - the binned geometry with only its DISTANCE replaced, which is phi-independent by construction - and both are offered to the same rule that ranks everything else here. Acquire by following, measure by holding still. The seeded centre is likewise a hypothesis and not a belief. It reads a once-per-turn wobble, and a tilt puts a term of that shape there too - one that grows as the radius squared, where a centre error does not - so pooling the rings absorbs part of the tilt into the centre. Believed outright it made a good 110 mm fit worse; offered as an alternative start it costs one more fit and needs no rule about when it applies. It is skipped entirely below a pixel, where it is not a different hypothesis at all, which keeps a well-headed run at 0.71 s. Measured on the 110 mm LaB6 exposure, whose true PONI is 765.90: a header centre 20 px out now lands within 0.5 px, where before it landed 31 px away. All five datasets are unchanged from their correct headers, and the distance still recovers from any header between 25 and 1200 mm. The limit is now understood rather than merely reached. Past a few pixels the azimuthally averaged profile stops showing rings: a ring tracing r(phi) piles up density where that turns round, so it averages into the two HORNS of the sinusoid, at R-|d| and R+|d|. The radius finder reports two rings where there is one, and the gap between them is 2|d| - the search window shrinks to exactly the offset it was meant to span. That caps recovery at roughly half the ring spacing, about 20 px here and failing by 40. Beyond it nothing is left in an azimuthally binned profile, and --calibration spots, which works from the spot positions themselves, is the method that still can. One pre-existing limit measured and NOT introduced here: a wrong distance together with a centre more than about 5 px out fails, because the centre error splits the radius list the distance search reads. The committed code before this change fails identically on those cases. Also fixed: fit_from now takes a whole geometry rather than a distance, and the declined-tilt refit was inheriting rot1/rot2 from it - pinning the tilt at exactly the unvalidated value the gate had just rejected. Same fault the gate exists to catch, one level up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:38:36 +02:00
leonarski_fandClaude Opus 5	01ad1e1743	calibration: only report a tilt the fit actually measured ... The detector tilt is refined by default, and on a pattern that cannot separate it from the beam centre the fit returns one anyway - there was nothing to stop it. Both displace a ring's radius as cos(phi), and only how that amplitude grows with the ring's radius tells them apart, which takes two well-sampled rings. At 500 mm on the LaB6 series only two rings reach the detector and the outer one is barely there: the tilt came out at the opposite sign to every shorter distance, dragged the PONI 28 px, and bought a residual of 0.960 px against 0.962 pinned. The covariance says so plainly - 0.1 sigma, and a beam centre quoted to +-180 px. So ask it. A tilt is kept only where the fit had it free AND it stands at least three times its own uncertainty; otherwise rot1/rot2 go back to the header's values and the beam centre and distance are refitted around them. Over the series the tilt stands at 50, 33, 15 and 8 sigma at 110 to 300 mm and 0.1 at 500 mm, so any threshold between 2 and 5 gives the same verdict on all five - this says which regime a fit is in, not where a line was drawn. The declined 500 mm fit lands on a direct beam of 773.53 px, against 773.56 for the pinned fit measured independently. It is a rejection criterion and nothing more. Clearing it does not certify a tilt: that estimator is limited by systematics rather than by this sigma, and a coherent half-pixel error in the ring positions fakes a tilt of the usual size while leaving sigma small. The report says "refined", never "verified". Writing the gate turned up a related fault in the pass loop. RingOptimizer pins the tilt by itself when every point it is given lies on one ring, and on a barely-sampled pattern a later pass lands in exactly that state - which froze the tilt at whatever the FIRST pass had produced and returned it with sigma zero, an unmeasured tilt wearing the appearance of a fixed one. The gate reads that as "not measured" and refits pinned, which is why it is stated over the geometry that gets reported rather than over what the last fit happened to do. Both paths are covered, profile and spots; the spots path was reporting a refined tilt as declined for the same reason. Two things measured and NOT taken: A robust loss. A Cauchy loss scaled to the previous pass's median residual changed nothing on the series - rms 0.415 to 0.421 at 110 mm, no case improved, every direct beam within 0.06 px. Ring points are per-sector peaks that already had to stand 3 sigma clear of their own background, so there are no gross outliers left to reject. Recorded at the call site rather than left as an unused option. A quality gate that refuses a bad calibration. Three candidate signals, all measured against naming the wrong standard on LaB6 data: sigma(PONI) does not see it at all (0.52-0.65 px, indistinguishable from healthy); the residual only half sees it (3.2-3.6 px wrong against 0.4-1.0 right, but a correct run from a wrong header sits at 1.0-2.4 and would be caught too); and the seed's match score is dominated by how many rings the calibrant lists, scoring 0.29 for a perfect LaB6 fit against 0.21 for a wrongly named silicon. None of the three separates, so no gate is shipped. What the run does say is the recovered distance against the header, and a wrong standard moves that to 446 mm on a 110 mm exposure - unmissable, and the operator's call rather than a threshold's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 17:07:37 +02:00
leonarski_fandClaude Opus 5	a5f416fcdc	calibration: take the detector distance from the rings, not from the header ... A powder calibration is run because nobody is sure the header is right, and the header's distance was the one number the fit could not survive being wrong about. The ring search is local - each ring is looked for inside a window a few pixels of radius wide - so a distance more than a percent or two out puts every ring outside its own window, and the fit then converges on whatever background fluctuation each window contains. It does not fail: a 110 mm exposure told the detector was at 150 mm reported 149.8 mm, with 146 ring points and exit 0. Only its residual said anything, 5.3 px against 0.4 px, and nothing read it. Measure the distance from the rings instead. The peaks of the azimuthally averaged profile give ring RADII, and a radius does not depend on the assumed distance at all - bin i holds the pixels at one particular radius whatever q that radius was called - so the radii are a property of the image. Against the calibrant's d-spacings, r = D tan(2 asin(lambda/2d)) then has one unknown. It is scanned rather than solved because the pairing of observed rings to d-spacings is unknown too, and the winning basin is solved in closed form. Nothing here reads the header distance except to bin the profile; it needs only the wavelength, the pixel size and the detector's extent. A powder pattern has genuine distance aliases, so one answer is not enough. A cubic primitive standard puts its rings at radii proportional to sqrt(N), and scaling the distance by sqrt(2) maps ring N onto ring 2N - most of the comb still lands on peaks. Measured: the 110 mm exposure with a 115 mm header scored its best at 156.5 mm, which is 110*sqrt(2). No adjustment of the score removes an alias the lattice really has, so the scan hands back the few best distances and each is fitted, the header among them as one hypothesis of several. The residual then separates them - 0.4 px against 5.2 px on that case - subject to an attempt explaining a comparable share of the pattern first, because a start so wrong that one ring point survives leaves a residual of exactly zero. Each attempt re-extracts at the geometry it converged to and fits again. The seed is measured from blended peaks and is good to about a per cent, close enough to converge from but far enough to sit every search window a few pixels off its ring, and an off-centre window takes its background off the ring's own flank. Nothing is re-read from disk, so the loop is free. Measured on the LaB6 distance series. A 110 mm dataset now recovers 110.03-110.17 mm from any header between 25 and 1200 mm, against +-2 mm before. All five datasets recover their own distance from a fixed wrong 250 mm header. With correct headers, four of the five are bit-identical to before and the 500 mm one moves by a single ring point - the two-ring fit whose tilt is 0.1 sigma anyway. Run time is unchanged at 0.62 s. The residual is larger on a run whose header was wrong (1.1 px against 0.4 px on the 110 mm case), because the profile was still binned at the wrong distance and its radial sampling is correspondingly coarse. The geometry is right; only the scatter about it is inflated. Re-running with the recovered distance recovers the residual too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:48:32 +02:00
leonarski_fandClaude Opus 5	d275bbcae7	calibration: -C overrides the calibrant, with absences from -S ... The five named standards are a convenience, not the limit of what a powder calibration can be run against. A unit cell given with -C now IS the standard in --mode calibration, and its rings are enumerated from that cell. A cell alone does not give a ring list, though: the centring and any glide decide which hkl the lattice actually diffracts into, and the fit pairs the innermost OBSERVED ring with the innermost LISTED one - so a list opening with a reflection that is not there scales the whole calibration by the ratio between them. Where -S is given, the absences come from the space group itself via gemmi, which covers centring, glides and screws in one mechanism rather than the three hand-written conditions the built-in table uses. The two agree exactly on LaB6 and CeO2, which is the cross-check that says the gemmi route is safe to hand a user's cell. They do NOT agree on silicon, and the test now pins that: Fd-3m's symmetry absences are only the F centring, while silicon's 222 and its relatives are extinguished by its two-atom basis - a structure-factor absence, not a symmetry one, so no symmetry handler can know it. gemmi offers 24 rings where the diamond condition gives 18. The extra rings do not move the first one, so the distance is not scaled, but they are rings with no intensity offered to the matcher - which is why --calibrant si stays, and why the usage text says -S supplies symmetry absences only. Without -S the cell is taken as primitive and the log says "assumed primitive". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:23:37 +02:00
leonarski_fandClaude Opus 5	5a80d2df53	calibration: fix four ways the powder fit quietly loses its input ... None of these changes the answer on a well-separated cubic standard - the LaB6 distance series is bit-identical by both methods - but each one is a case where input is dropped or mis-assigned without saying so. The circumcentre vote grid was a fixed 4000x4000 box, and the caller never passed anything else. That allocated 128 MB whatever the detector, and on a detector larger than 4000 px in either direction it put the beam centre outside the grid, so every vote was discarded and the guess failed with "Beam center not found". Span the spots' own bounding box instead: a powder ring encloses its centre, so that is where the answer has to be. uint32 votes while there - the most any bin can take is C(500,3). Spots were assigned to the FIRST calibrant ring within a fixed 0.1 1/A, not the nearest. Silver behenate's orders sit 0.108 1/A apart and hexagonal ice has three rings inside 0.06, so for those two standards the window reaches the neighbour and every point lands on the lower-q ring of the pair, biasing the distance. Take the nearest ring, and clamp the window to half the gap to the neighbour - which is what the profile path already did inline, now shared as RingMatchWindow and covered by a test that checks it actually narrows on the crowded standards and not on LaB6. A profile bin no pixel fell in is NaN. SectorPeakQ dropped such a sector by accident, through NaN comparisons falling false; check the four background bins and return explicitly. Ice-ring handling is switched off in calibration mode. Flagged spots are sorted last by the spot budget and so discarded first, which for --calibrant ice throws away exactly what is being calibrated on. The two per-ring std::cout lines in GuessGeometry are gone: a library has no business writing to a terminal, and constructing a Logger to keep them would emit a version banner from inside a fit. What matched belongs in the result struct, which the quality gating still to come needs anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:18:27 +02:00
leonarski_fandClaude Opus 5	fba5435c38	calibration: report what the ring fit knows about its own answer ... The ring fit reported the SCATTER of its measurements (rms, and the beam-centre standard error that follows from it) but nothing about how well the fit pinned each parameter. Those two part company exactly where a calibration is worth doubting: as the rings run out, the tilt and the beam centre stop being separable - both displace a ring's radius as cos(phi) and only the way that amplitude scales with radius tells them apart - so the fit can sit tightly on the few points it has while being free to spend tens of pixels of beam centre on a tilt the data do not support. Take the covariance of the converged problem from Ceres and report it. Measured on a LaB6 distance series, the fitted tilt is 50 sigma at 110 mm and 0.1 sigma at 500 mm, where only two rings reach the detector; at 500 mm the fit quotes its own beam centre to +-180 px and its tilt to +-2.9 deg on a 0.35 deg value, and the correlation between them is 1.000. Nothing acts on this yet - it is printed so the next change can gate on it. Ceres returns the bare (J'J)^-1 of an unweighted problem, so it is scaled by chi2 per degree of freedom; that leaves the sigmas in pixels, mm and radians whatever unit the residual is stated in. Cost is one 5x5 SVD per run, below the noise of the surrounding I/O. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:11:18 +02:00
leonarski_fandClaude Opus 5	f34d9fe62b	calibration: move the powder fit into image_analysis/geom_refinement ... CalibrateFromProfile/CalibrateFromSpots/WritePoniFile sat in rugnux/, so the only way to reach them was to link the Rugnux library - which drags in JFJochWriter and gemmi. Nothing in them needs either: the includes are all common/ and image_analysis/geom_refinement/, next to the RingOptimizer and RingsFromProfile they call. Moving them to image_analysis/geom_refinement/PowderCalibration.{h,cpp} puts the powder fit beside the rest of the geometry refinement and makes it reachable from anything that already links JFJochImageAnalysis - the receiver and so the broker included, which is what an online geometry calibration would need. Pure move: the file contents differ from their previous form only in the include paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NfuDvf5ipV3Hi8TiCUKD27	2026-08-31 16:09:42 +02:00
leonarski_fandClaude Opus 5	ba24946c0f	twinning: do not report a twin the L-test rules out ... The two indicators can only move one way under twinning - <|L|> down from 0.500 toward 0.375, the second moment down from 2.0 toward 1.5 - but they were combined with an OR, so a narrow intensity distribution could report a twin on its own while the L-test said the opposite. That is not a marginal disagreement. It fires on small-cell rotation data, where a twin fraction of 0.50 was reported for a crystal whose <|L|> was 0.63: a value no twin can produce, and one that points at the opposite situation - a structure whose intensities behave centric. Suppress the flag when <|L|> sits at or above its untwinned value; the two indicators are otherwise combined exactly as before. Measured over 113 stored reports: two flip, both of them wrong today, and nothing else moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:34:16 +02:00
leonarski_fandClaude Opus 5	71cca91ec1	rugnux: rank screw candidates without the control count ... A screw's control class is the complement of its absent class on one axial row, so the two move together: a candidate that predicts more of the row absent leaves fewer reflections to be judged against. The Beta tail AbsenceEvidence computes grows with that control count, so the stricter candidate was charged for the very reflections it correctly called extinct, and a group whose predicted-absent class is a strict superset of another's - with the extra reflections equally dead - could score LOWER than the group that explains only part of the row. Two mechanisms, one dataset each. * The control-count minimum gated the row MEAN, which is the scale the zone's evidence is stated in, on the same count as the row MEDIAN, which is a violation threshold. A candidate could therefore forfeit a whole zone by being right: on a tetragonal 4_1/4_3 wedge, thirteen 00l reflections measured at 0.1% of the two l = 4n beside them scored ZERO because only two control reflections were left, while the nine of them a 4_2 also predicts absent scored 38.7. The mean now stands on two, the median still on three. * The Beta tail is replaced, for a SCREW zone only, by its b -> infinity limit - the same statistic with the control count dropped. A p-value computed against each candidate's own null is not one scale across candidates; what is left is the likelihood ratio of the absent class against Wilson at the row's own mean, which is a sum over reflections and therefore comparable. Asymptotically it is n_absent * (log(1/ubar) - 1), so an equally dead superset can no longer score lower. Measured on a tetragonal 4_1/4_3 crystal: 29 dead 00l against 8 control read 47.7 nats where a subset of 19 of them against 18 control read 55.5. The centring statistic is untouched: its control is the whole present population, not the complement of a claim on one row, so neither mechanism applies to it. Measured. Analytic superset monotonicity, on a 176368-point grid with no calibration and no corpus: 1.29% violations -> 0.00%, exactly monotone wherever both candidates keep a control class. Over 112 stored merges (open, in-house and private arms) exactly ONE decision changes and it is a gain, a 4_2 -> 4_1/4_3 swap on a deposited 4_3; the screw column over the deposited arm goes 33/41 -> 34/41 with zero screw over-calls before and after, and the four centring outcomes are bit-identical. An eight-wedge reproducer whose 00l row a strict candidate eats goes from two mis-calls to none, end to end. The refused side moves the safe way: 103 zones the old statistic scored negative go lower, none turns positive, and the minimum evidence of an adopted screw rises 20.4 -> 23.8 against an unchanged bound of 20. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 15:33:29 +02:00
leonarski_fandClaude Opus 5	6d37b31948	report: give the resolution the CC1/2 fit reached, not only the one it was cut at ... The automatic cutoff fits the CC1/2 fall-off, takes the crossing of the target (0.30 by default) and then deliberately keeps one more shell. Only the extended limit was reported, as INCLUDE_RESOLUTION_RANGE, and that is the number a reader takes for "the resolution of this dataset" - so the report quoted a limit that is generous on purpose as if it were the measurement. The generosity itself is right and stays: a shell that is included can still be downweighted or dropped by refinement, while one that was truncated cannot be put back. What was missing is the other number. ComputeCCHalfLogisticCutoff now also returns the crossing, ApplyResolutionCutoff passes it out, and the report prints it as FITTED_RESOLUTION beside the range the reflections were written to, with a sentence saying which is which. The log line names both as well. Nothing about the cut, the merge or the written reflections changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 11:02:37 +02:00
leonarski_fandClaude Opus 5	8d719c1383	symmetry: a zone whose absences were never measurable must not outrank a dead one ... sum_u is a sum of max(0, E^2) over a zone's predicted-absent reflections, in units of their control class's mean, so it is EXACTLY zero when every one of them merged non-positive. The Beta tail then diverges and the clamp that caught it made each such reflection worth about 690 nats. That was harmless while the value only had to clear a bound of 20. Since the absence evidence became the primary sort key, summed across zones with no minimum count per zone, it has ranked the candidates - and it ranked them backwards. A zone of two absences that were never measurable scores 1378 nats where a genuine zone of six absences at 1% of its own row scores 22, so the candidate claiming a screw on an UNMEASURED row beat the one whose rows are actually dead, by sixty times. The comment on the ranking claimed an extra condition "LOSES the zone's evidence when the row is not dead"; an unmeasurable row could not lose, it won outright. No measurement places a merged intensity at exactly zero, so sum_u is floored at a thousandth of the control mean per absent reflection. The constant is an order of magnitude below the precision any real merge reaches - a thousandth of the row mean needs I/sigma ~ 1000 against that row, where ISa tops out near 40 - so it can only remove the singularity, never suppress evidence a measurement could have produced. Measured across the realistic range it changes no genuine zone at all (22.0, 34.7 and 65.4 nats, unchanged to four figures) and takes the unmeasurable ones to 13 and 36, below and around the bound respectively. AbsenceEvidence moves to the header, where the comments already named it, so the ordering it has to satisfy can be tested directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:17:26 +02:00
leonarski_f	527a5187f4	lattice: restore the character that names a centred monoclinic mI reduced form ... ITA character 43 was absent from the table. It is the type-II reduced form of a centred monoclinic lattice with no length equality - mC, mI, mA and mF are one Bravais lattice in four settings, and this row names the one whose conventional cell comes out I-centred. With the row missing such a cell reaches character 44 and is reported as triclinic, losing its centring outright. It accounted for 30 of the 31 demotions left after the two fixes before this one, and for 4.1% of random centred-monoclinic lattices. Both of its conditions are equalities on scalar products - International Tables gives them as 2|D+E+F| = A+B and |2D+F| = B - so the three angle tests are vacuous for it and the second equality is what selects it. cond_2DF was declared in the character struct and never tested against anything, because until now no row used it. Audited over 14000 exact lattices, the row fires 35 times: 32 are centred monoclinic lattices it recovers correctly and 3 are triclinic cells it promotes, all three of which Le Page promotes at the same tolerance. No over-call is attributable to it. The row is taken from International Tables rather than derived - no single integer matrix of determinant 2 covers more than 68% of these cells - and the test checks the conventional cell it produces has two right angles and twice the primitive volume. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit 4c2baf0cc3b2177196a9448df06e24b5446de96a)	2026-08-31 07:17:00 +02:00
leonarski_f	931d7acc7c	lattice: judge a structurally-zero scalar product against the size of the cell ... The Niggli type of a reduced cell is the sign of its three scalar products, and gemmi's reduction was asked to decide those signs against an absolute tolerance of 1e-9 while the products themselves are 10^3 to 10^5 A^2 and the cell is held in float. A product that is structurally zero therefore arrives carrying about 1e-4 A^2 of rounding and is read as definitely signed. The reduction lands on the wrong side of the type-I/type-II boundary, the character written for the other side matches nothing, and the lattice comes back with less symmetry than it has. It is not a corner case. Take an exactly body-centred tetragonal lattice with c > a*sqrt(2) and merely ROTATE IT IN SPACE: 38 of 60 rotations lose the 4-fold, and it comes back C-centred. Every lattice this code classifies is a refined, rotated one; the two tetragonal-I cases already in the tests are axis-aligned, which is the one corner where the rounding vanishes. The tolerance is now scaled by the cell's own magnitude. Over 14845 exact lattices that puts it on a plateau three decades wide - demotions 461 -> 31 - with the over-call count unchanged at every point of it, and with the three fixes that follow the plateau is four decades wide and the demotions are zero. The constant is 100x the one Grosse-Kunstleve et al. give because theirs is calibrated for a double-precision cell and this one is float: measured, their value recovers 6% of these lattices and this one 93%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N (cherry picked from commit b82f217719450e1e46e94be5b5a43ed8ffbf2ca4)	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	493be2cf19	symmetry: enumerate the settings the cell can host, not only the reference ones ... The space-group search offered a candidate only if gemmi calls it the reference setting. A setting is a statement about direction, so that restricted the search to the axes the convention chose: a crystal whose 2-fold lies on c had no rung between P1 and 222 and fell to P1, and one whose screws lie on b and c was reported as the group with a single screw on c - the wrong group, not a lower one, because the candidate that predicts a subset of the real absences and nothing else wins on no evidence at all. Both stages now enumerate more, under refusals rather than thresholds. Stage B offers the non-reference settings of the chosen point group. Their rotation set is equal to the chosen one, not merely contained in it, so this cannot raise the symmetry; what it adds is a screw or a centring on the axis the data show it on. A candidate is offered only if the cell's own metric admits the rotations its setting names, and one predicting exactly the absences another candidate already predicts is dropped as the same hypothesis under a second name. A non-reference candidate whose centring class this merge does not contain is refused outright: the reference path may adopt an untested centring because the caller's centred-lattice re-test backs it, and a non-reference setting has no such backing. Stage A offers the rotation sets no reference setting carries - the a-unique and c-unique monoclinic 2-folds, and the two rhombohedral-axes trigonal groups - and only those, so every point group reachable before is still reached by the same group in the same setting. A rung reached only that way may be ADOPTED but does not judge anything else: it is skipped when the reference chi^2 is formed and when a higher promotion's parents are collected. Without that it made higher promotions strictly harder - a promotion answers to the most damning of its parents, and offering two more order-2 subgroups of 222 refused 222 and 432 on crystals that had them. The run report gains SPACE_GROUP_NAME beside SPACE_GROUP_NUMBER, since the number alone does not say which axes a group's symmetry lies on, and a group that is not a reference setting is printed as its extended Hermann-Mauguin name. The two-arm reconciliation compares point groups by that name rather than by number for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-31 07:17:00 +02:00
leonarski_fandClaude Opus 5	26fc4b02b3	symmetry: carry the space group as the group, not as its number ... The adopted space group travelled the pipeline as a bare int and was rebuilt downstream with find_spacegroup_by_number, which returns the reference setting. So every setting a number cannot name was destroyed one line after it was determined: P 1 1 2 came back as P 1 2 1, I 1 1 2 as C 1 2 1, R 3:R as R 3:H. DatasetSettings now holds the gemmi::SpaceGroup itself, DiffractionExperiment exposes it as GetGemmiSpaceGroup() / GetSpaceGroupOrP1(), and everything that used to take an int - HKLKeyGenerator (its int constructor is gone, so the compiler finds the callers), the merge, the R-free flags, French-Wilson, the reindexing ambiguity, the completeness enumeration, the MTZ and mmCIF exports, the model validation - takes the group. -S keeps the setting the symbol names rather than reducing it to a number. The end message carries both spellings and a reader prefers the name, since only the name keeps the setting while the number is what a reader written before the name understands. It carries them over CBOR too: the determined group was never serialised at all, so a group rugnux chose reached the master file only when the same process wrote it, and an online writer fell back to whatever the user had supplied at the start. Both keys are optional additions, so an older reader skips them and a newer one reads an older sender. On disk the master's /entry/sample/space_group carries the extended Hermann-Mauguin name and is what the reader takes the group from, so a setting survives a _process.h5 and the --mode scale that re-reads it; the number stays beside it and is the fallback for files written before. Every one of the 230 reference settings the old writer could produce reads back as itself, so older files are unaffected. Stage A and Stage B of the search still enumerate reference settings only, so this determines no group differently today - it is what the enumeration needs before it can be widened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N	2026-08-31 07:16:43 +02:00
leonarski_fandClaude Opus 5	03456dcddd	calibration: an option to hold the detector tilt fixed ... --mode calibration fits five parameters - beam centre, distance and the two PONI tilts - and a program that cannot express a tilted detector has nowhere to put the last two. Dropping them after the fact is worse than never fitting them: the centre and the distance of a tilted fit have already absorbed the tilt, so the flattened geometry is right nowhere. rugnux --no-refine-tilt, the "Refine detector tilt" tick box on the viewer's Calib page and RingOptimizer's refine_tilt argument hold rot1/rot2 at the value the geometry came in with and fit the remaining three. That is the best flat-detector answer, and the one such a program would refine to itself. Measured on a five-distance calibrant series. At short distance the tilt is real and reproducible - three independent fits agreeing to 0.01 deg, radial rms 1.4 -> 0.4 px - and its direct beam agrees with the background beam-centre estimator to 0.05 px, so the tilted model is the physically right one. The pinned fit then displaces the centre 2.6 px to absorb the tilt and lands within 0.03 px of the same place at every distance. Past ~300 mm, where only two rings reach the detector, the tilt is instead under-determined: it comes out with the opposite sign to every short-distance fit and drags the PONI 28 px while the rms does not move (0.960 against 0.962). The existing degeneracy guard only fires on a strictly single ring, so it does not catch that; declining a tilt that does not pay for itself in rms is left for a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxrrPcxodNiXzhNiECCVp5	2026-08-30 20:44:12 +02:00
leonarski_fandClaude Opus 5	97f57a430d	beam centre: run the four consistency restarts at once ... FindBeamCenterFromSpotSymmetry calls Estimate() five times - once for the answer and once from each of four starts 25 px away - and three quarters of each of those is an 861-point brute-force grid over the spindle. So making the uncertainty gate live was paid for by multiplying the estimator by five, which is the whole of the pre-scan's cost. The four restarts share nothing: each takes its own copy of the geometry and only reads the spots. What is wanted from them is a max, which is order-independent, so running them concurrently gives the same number. Measured on three rotation crystals, the committed centre, the reported sigma and the fitted spindle angles are unchanged to every printed digit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T3yNBXk4wKdMZy1ak2NY7f	2026-08-30 08:37:55 +02:00
leonarski_fandClaude Opus 5	fed077e683	geometry: hold the detector plane as axis vectors, and give the discrete part its own home ... The detector plane was three PONI angles and nothing else, so the two things it cannot express - an image mirrored in Y, and one mounted at a multiple of 90 degrees - had no home at all. They are now the DetectorOrientation carried by the detector setup, composed with the PONI rotation into one orthogonal matrix whose columns ARE the fast axis, the slow axis and the sample->PONI normal: lab = R(rot1, rot2, rot3) * Delta * ( (x-bx)*p , (y-by)*p , distance ) GetFastAxis/GetSlowAxis/GetNormalAxis read those columns and DetectorAxes() sets the plane from them, decomposing back to the angles; PoniRotMatrix and PoniAnglesFromMatrix are the conversion in both directions, exact on the canonical branch (rot2 in [-pi/2, pi/2]) and with a stated convention at gimbal lock. The angles stay stored rather than re-derived, so a geometry given as angles is written back as the same angles, to the bit. Delta is never inferred. In particular an arbitrary rot3 is NOT decomposed into a quarter turn plus a residual: rot3 is a fitted quantity, and a least-squares step must not be able to turn the stored image. It is set only where something states it - the detector setup, --detector-mirror-y / --detector-quarter-turns, or the value a file this system wrote records - and defaults to the identity, which makes the whole change a no-op for every existing detector and every existing file. It is a different setting from DetectorSetup::mirror_y, which flips the MODULE LAYOUT while an image is assembled and so decides what the stored pixels are. Merging the two would apply the mirror twice for every modular detector, or change the pixel content of every file written; both are ruled out. The new one earns its keep exactly where the old one is a no-op: a detector whose image arrives already assembled has no layout to flip. Both generators are signed permutations of the in-plane offset, so they preserve the distance from the PONI. That is why almost nothing downstream changes: everything needing an azimuth already goes through LabCoord, and everything that does not needs only a radius. The two hand-written copies of the rotation - XtalResidual and RingOptimizer - take the discrete part as four constants next to cos_rot3/sin_rot3, since it acts in the detector frame where rot3 acts in the laboratory and cannot be folded into it. RingOptimizer needs it despite being a radial fit: it fits the tilt, and the discrete part changes which way the tilt tips a ring. Carried as two optional CBOR keys and two detectorSpecific datasets, both back-compatible; the NXmx module axis vectors and the translation direction stop being hardcoded and are computed from it, reproducing today's values exactly at the identity. GetPoniRotMatrix is renamed GetDetectorMatrix, because it is no longer only the PONI rotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 23:00:35 +02:00
leonarski_fandClaude Opus 5	172a845cbb	rotation indexing: refine twelve candidate lattices instead of four ... The first pass refined only min(candidates, 4). The comment at the selection already conceded that the pre-refinement indexed fraction is an unreliable discriminator, and on one dataset degenerate cells occupied three of those four slots - so the right cell, ranked fifth, was never refined. The preceding commit's volume guard frees those slots but does not widen them. This is a one-for-one trade, measured by rebuilding with the old value to confirm the attribution. It gains one dataset: R_meas 27.2% to 23.8%, with mean I over sigma better in EVERY shell at identical shell edges, so it is not a resolution ramp. It costs another: a clean abort becomes a wrong four-fold supercell in P1 at 36% completeness. That is the same shape of trade already accepted for the plane-normal cap search in the preceding commit - an honest failure becoming a wrong answer - on a different dataset. It is committed separately so it can be reverted on its own if the maintainer weighs this instance differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:35 +02:00
leonarski_fandClaude Opus 5	4e8db41785	indexing: refuse a coplanar candidate, and search the plane normal when the shortlist is flat ... Three related changes to the FFT candidate path, batteried together because they touch the same function. A COPLANAR CANDIDATE REACHED REFINEMENT. ReduceResults filtered triples on lengths and angles only - the 30-150 degree bound admits any flat combination - and there was no volume test. On one dataset 41 of 5535 candidates had |V|/abc below 0.05, with a clean decade gap to the next, and three of them reached the optimizer. UnitCell is float, and for a cell that flat the metric determinant is around 1.5e-7, so float32 gets its sign wrong 19% of the time where float64 never does. The guard against a negative argument to sqrt then CREATES the singularity it was meant to prevent: it puts c in the a-b plane, the reciprocal volume is 1/0, and the residual is 0 times infinity. Ceres reported a not-a-number Jacobian and wrote several hundred lines of solver output per failed solve. VolumeFraction() is |V|/(|a||b||c|), rejected below 0.02 - about 1.1 degrees off flat, ten times below the flattest real candidate observed and a thousand times above where float loses the sign. It is enforced at the producer and at the two optimizer entry points. Note the existing sanity checks use ABSOLUTE volume, which a 320 cubic-angstrom flat cell passes. The same reciprocal-volume division is now guarded at the two remaining sites that share the pattern. A SHORTLIST CONFINED TO ONE PLANE cannot close a cell, and the row it is missing is the plane normal. That is detected from the scatter-matrix eigenvalue ratio - measured, degenerate clouds score 2e-5 to 3.3e-4 against 0.026 or more for every non-degenerate one, a factor of eighty - and one further transform is spent with the same direction count inside a three-degree cap about the normal, so the plan and buffers are untouched. More directions cannot substitute: at the exact true direction the long axis ranks 1422 of 16384 by prominence while the shortlist cut is four times higher. Ranking, not sampling, is the obstacle. A four-fold denser grid was measured and rejected - it reaches the same answer to three decimal places and takes a run from 2.5 to 8 GB of device memory. fft_min_unit_cell_A is reachable as --fft-min-unit-cell and is lowered automatically by -C, mirroring how the maximum is already raised. The default of 10 is unchanged: a lower floor admits spurious sub-cells on protein data, and over 73 protein runs the floor was never lowered while the sibling maximum did fire twice, so the path is live and correctly inert. Corpus of 93 datasets, both arms, one build: 72 bit-identical on report content and p.hkl checksum, 13 failing identically, and the count of working datasets rises by one. The volume guard fires on 58 of 93 and 47 of those stay bit-identical - it fires constantly and almost never changes an answer, which is what it should do. Solver chatter falls from 919 lines across three datasets to none. The cap fires on 4 of 93, none of them in the in-house or private arms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 20:31:20 +02:00
leonarski_fandClaude Opus 5	8634e9a72b	symmetry: score screw absences per zone, and choose centring by likelihood ... Two places where the absence tests threw away information they already had. The screw evidence was POOLED across axial rows, so one unmeasured row vetoed a confirmed one. A crystal reading h: +18/+24 and k: -6/+8 answered a silent P222 rather than "2_1 along a confirmed, b undetermined". Scoring each zone separately, and letting a zone abstain, is what POINTLESS has always done; the conditions are independent, so their log-likelihoods add. Centring was decided on a COUNT of net absences while screws used a Beta-tail likelihood. On one crystal that made F222 (1662 absent, 356 violations, absent class at 34% of present) beat C222_1 (1098 absent, 15 violations, 0.4%) purely because 1306 > 1083. Giving centring the same likelihood separates them by 1717 nats - F222 -357.9, C222_1 +1359.3, I222 -787.6 - and needs no bound: min_absent_observed and max_absent_present_ratio are untouched, and nothing was tuned to this case. The deposited structure for that crystal is C222_1. Corpus of 94 datasets, both arms at -N 6: 78 comparable, 73 bit-identical, point group correct on 68 against 66, space-group number on 54 against 51, nothing regressed. The 73 bit-identical results are also the determinism control - two binaries cannot agree byte-for-byte that often by chance. An earlier ranking key summed each group's single gating number and demoted a P2_12_12_1 to P2_12_12, because pooling three genuine screws reads weaker than two when the third row is shallower. Summing the per-zone log-likelihoods is what fixes that, and the whole corpus was re-run on the corrected binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:57 +02:00
leonarski_fandClaude Opus 5	10873eee7a	lattice search: reject an impossible character, and supplement every angle ... Two defects in the Bravais character walk, both of which silently cost symmetry. An impossible character MATCHED. The walk computes acos(cond_F/sqrt(A*B)); when the character is geometrically impossible the argument leaves [-1,1], acos returns NaN, and the acceptance test fabs(NaN - actual) > tol is FALSE - so the character is taken. One corpus crystal matched a monoclinic-C character whose implied cos(gamma) is 1.086. The type-boundary retry covered beta only. All three angles carry the Niggli boundary, and negating two basis vectors supplements the OTHER two, so each angle needs its own flip. A crystal whose reduced gamma sits at 89.900 degrees needs the alpha/beta flip and never got it. Measured on 84369 exact lattices spanning all 14 Bravais classes: nothing is lost in any class, and orthorhombic-I recovery rises from 60.3% to 87.2%. Over 440000 random and perturbed cells the first change only ever demotes a monoclinic-C match to triclinic and the second only ever promotes out of triclinic - it is a strict superset of the beta-only retry. On the 94-dataset corpus, two crystals gain their correct point group and none regresses. Both regression tests fail without the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 19:57:41 +02:00
leonarski_fandClaude Opus 5	6319d5c600	indexing: tell an indexer that failed apart from one that found nothing ... IndexerThreadPool collapsed two different outcomes into the same empty reply. A worker that threw set result = nullptr, and so did a dispatch that never found a free worker; both then became a default-constructed IndexerResult, indistinguishable downstream from an indexer that ran and found no lattice. The first says nothing about the frame at all - the indexer never looked, and it will fail again on the next one - while the second is a real negative result about the crystal. The visible cost was the advice a failed run gave. With the card full, rugnux printed "Indexer thread 0 failed: CUDA (GPU) error" and then ended with "Two-pass rotation indexing found no lattice. Check the beam centre (--beam-x / --beam-y), raise --max-spots ..." - sending the operator to look at geometry that was never wrong, for a machine that was simply out of memory. None of those remedies can help when the frames were not examined. IndexerResult gains an optional error, set by the worker and by the pool's own catch; the remaining nullptr path keeps its meaning of "not attempted" and deliberately carries no error. RotationIndexer records it and exposes GetIndexerError(), and rugnux's first pass branches on it at the throw site, naming the resource failure instead. The error travels as DATA through image_analysis/ rather than as an exception, because RotationIndexer::RunIndexing() is on the online path - IndexAndRefine calls it on a schedule from the broker and the receiver, where dropping a frame is the right failure and killing a live acquisition is not. Only rugnux, which owns the "this run is over" decision, turns it into one. The failure result is byte-identical to the default-constructed one it replaces, so any_executed and the online frame-drop behaviour are unchanged. The new IndexerError category exists because the category is only the display prefix on what() - Category() is read nowhere - and the old line read "Processing failed: Input parameter invalid" for a GPU fault, which is the same defect one layer up. SpotFinderError is the precedent. msg.indexing_result is deliberately left alone: of its three states, "not attempted" is the honest one for a frame the indexer never examined, and asserting false would be the same collapse again. Verified on a squeezed card (246 MB free, -N 4): exit 1, zero dropped frames, the new message, and no mention of --beam-x. On a free card, against the previous binary at -N 6 on the same input, the logs differ only in paths and timing - same space group, cell and merge statistics. Targeted Catch2 cases pass, including three that drive the pool through the online path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5JG6kJqZoCWaoZ43JGTW	2026-08-29 15:30:28 +02:00
leonarski_fandClaude Opus 5	dce8d877d0	Scaling: make Run() idempotent, and re-calibrate the promotion it was hiding ... RotationScaleMerge::Run() resumed its alternating-least-squares loop from wherever the previous call left off instead of restarting from the ingested data. A de-novo run calls it four times, so pass N was scaled at 3N iterations, not 3. Two identical calls on the same object produced merges correlating at 0.847 with ZERO bit-identical intensities; they are now 68010 of 68010 identical. Where that mattered is narrower than it sounds, and worth stating exactly. In a determined point group the fit is stationary from iteration 2 - over the battery the collapsed-scale guard drops 377 frames at three iterations and 397 at twelve, and 30 of 39 crystals drop nothing at any count - so production merges were barely touched, 38 of 39 rows character-identical. But the SEARCH merges in P1, where a reflection has two or three observations, and there the fit never reaches a fixed point at all: the same guard drops 417 frames at one iteration and 1101 at three. A factor 1.05 against a factor 2.6, same code. So the defect lived in the one arm whose output the user cannot recover from downstream - the arm that picks the space group. Fixing it moves the statistics the promotion gates are calibrated on. The population as a whole does not shift (median change 0.9-2.7% over 280 matched candidates) but the tails do, p90 by 8-14%, and that is exactly where a promotion is decided. On the two crystals that decide, the H statistic's two populations SWAP ORDER: a genuine tetragonal 4->422 goes 1.685 -> 1.819 while a trigonal 3->32 twin law goes 1.906 -> 1.665. No H bound exists any more that keeps the genuine case and refuses the twin - the noise the loop was adding had been acting as a brake. Of the four statistics only b and R_meas still order the two correctly. Set max_systematic_b_ratio 1.90 -> 1.78, by a rule written down before the population statistic existed: take the statistic with the largest spurious_min/genuine_max on the population its gate actually sees, put the bound at the geometric midpoint. That is sqrt(1.711 * 1.853) = 1.78. Battery 36/39 space groups matching XDS, the same as before the scaling fix and one better than the scaling fix alone; the three mismatches are the known references where XDS is wrong. The refusal now reaches the user by name. The bound's margin is 4% either side - SMALLER than the p90 shift this very commit produced in that statistic. It separates the two crystals that exist and is not robust to another input shift, which is written into the header rather than left to be discovered. Two hazards go with it: re-measuring it must hold the scaling correction surfaces fixed, since changing the iteration count re-rolls their cross-validated on/off decision on 7 of 39 crystals for ~0.3 in ISa, all-or-nothing and non-monotone; and --scaling-iterations stays at 3 on both arms, confirmed rather than assumed (production 2->3 moves <I/sigma> +1.38% and 3->6 only +0.13%; the search gives the identical space group on all 39 crystals at 2 and at 3, and loses one at 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 22:44:21 +02:00
leonarski_fandClaude Opus 5	e78fbd55b7	Indexing: let the FFT search reach past 500 A, and stop clipping the peak background ... fft_max_unit_cell_A was both the default and an enforced check_max, so 500 A was the longest basis vector the FFT could ever return: FFTIndexer sizes its projected histogram from that value and the transform's last usable bin IS that length. Of the PDB's 206950 X-ray entries, 1091 (0.53%) have an axis longer than that and were unindexable by construction. The accepted range now goes to 1200 A, which leaves 8. The DEFAULT is unchanged at 500 - the histogram is sized from the value in use, so nothing pays for the wider range unless a caller asks for it. The peak picker's running-mean background was truncated at the ends of the spectrum rather than slid inward, so a peak within bg_half (~15 A) of either end - which is exactly where the longest cells sit - was judged on a one-sided background, biasing its prominence by however much the spectrum sloped there. Keep the window a constant width and slide it. Both bounds stay monotonically non-decreasing in j, so the GPU kernel's running sum is still valid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-28 20:12:36 +02:00
leonarski_fandClaude Opus 5	4951538368	rugnux: measure the spot budget instead of taking it as given ... --max-spots was a fixed 1000, and on a rotation sweep it sets the DENOMINATOR of the per-frame acceptance gate, which admits a frame when at least a fifth of its spots index. Detections are not all reflections: on a strongly diffracting crystal with heavy solvent background there are 2372 a frame, 1416 of them on ice bands, and only 178 index - so a budget that takes essentially the whole list puts the entire frame population on the gate (median indexed fraction 0.271, tenth percentile 0.218) and the run integrates 83% of its images. The same run with a smaller budget gains 211 frames and loses none, and the frames it gains are the ones with the MOST detections. That is how a larger budget integrates fewer images. So measure it: over the first pass's validation frames, with the sweep's lattice known, tally each spot by rank as +1 if it lies on the lattice and take the budget at argmax over N of n_indexed(N) - 0.20 * n_counted(N) which rises exactly while spots at that depth index better than the gate's own floor and falls after. The 0.20 is that floor, not a new constant - it is lifted out of the function-local it already lived in. It means: as deep into the intensity-ordered list as the image is still showing reflections of THIS crystal. The argmax alone would not do. Under the null that spots index at the same rate at every depth the tally is a driftless random walk, whose maximum is positive whatever the data, so a bare argmax shortens every dataset. The budget therefore has to clear the walk's own noise: the quantity it acts on is the fall from the peak to the end of the list, which is that walk read backwards, and the reflection principle gives its null law in closed form - P(fall > z*sqrt(g(1-g)T)) = 2(1-Phi(z)). That already pays for the search over ranks, so nothing further is owed to multiple comparisons. One false cut in a thousand measurements - a twelfth of one over a 39-crystal two-pass corpus - fixes z at 3.29. The level was chosen before the rule was written and was not revisited afterwards. Battery, same build, one changed default: 36/39 space groups in both arms, none lost, and 36 of the 39 crystals BIT-IDENTICAL. Two move materially - the strong crystal by +17.4% observations and +35.9 points of CC1/2 at 1.58 A, its indexing rate 83.4 -> 95.1%, and another by +93.5% observations with completeness 76.6 -> 98.4%. The significance requirement is what makes that list clean: it removed the one crystal the unguarded rule regressed, and with it two of the five gains, whose peaks do not clear 3.29 sigma on 60 frames. The lever for those is the sample and not the threshold - power grows as the root of the frame count while the bar stays where it is. Rotation only, and the first-pass lattice search always sees the full list: a fixed --max-spots 250 would have been much cheaper to write and fails a battery crystal outright, starving the de-novo FFT into a wrong cell that indexes 7 of 60 frames. Stills never reach the code path - verified bit-identical merged reflections on a serial set - and --max-spots N still pins it, as does the library default the broker and the FPGA use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 19:02:02 +02:00
leonarski_fandClaude Opus 5	df98bb8789	Space-group search: ignore the frames the crystal barely diffracted on ... A crystal that repeatedly leaves the beam over a full turn was read as P2(1) where it is P4(3)2(1)2, on data that merge at CC1/2 96-98% once that symmetry is assumed. The cell was right, the indexing was right - 83% of images - and every 422 operator correlation still came out between 0.08 and 0.38 against a gate of 0.30. The per-frame scale enters the intensities as 1/G. On this crystal the fitted scale spans 356x between its 5th and 95th percentiles and 15% of frames sit below a tenth of the run median, so those frames arrive amplified ten- to a hundredfold - and their sigmas are amplified by exactly the same factor, so nothing weighted by sigma can see it. What saves the PRODUCTION merge is multiplicity: in the determined symmetry a reflection is measured ten or twenty times and --reject-outliers throws the amplified observation out. The search merges in P1, where a reflection has two or three observations and no majority exists to call any of them an outlier, so the bad value lands in the merged intensity at full weight and the operator correlations pay for it. So drop those frames from the search merge only, beside the |zeta| filter and through the same snapshot that undoes it before the production merge, which keeps every frame. A tenth of the median because that is where the two populations sit: over the 39-crystal battery 33 crystals have NOT ONE frame below it, so the filter is inert on them by construction; of the six that do, five spend 0.1-3.5% of their frames there against this crystal's 15%. A twentieth leaves it under the gate; a fifth starts costing frames the battery says are real, the smallest legitimate min(G)/median(G) measured being 0.070. Battery 35/39 -> 36/39 space groups matching XDS, with 38 of the 39 rows character-identical: the only crystal that moves is this one. On it the operators go to 0.37-0.62, and every one improves in every resolution shell, most at low resolution - the signature of a scale error removed rather than noise removed. The run then merges to 1.41 A instead of 1.67, at CC1/2 97.7% and completeness 99.7%. Four other explanations were measured and refuted first: the search resolution cut (the correlations are flat in resolution, and a battery crystal with a finer detector and no cut at all gets 422 without trouble), the space group being inherited from the pass-1 geometry (handing the refined geometry in from the start changes nothing), the ordering of the corr snapshot around the collapsed-scale guard (0.02 in correlation, nothing in space group), and the ice-ring flagging (removing it entirely leaves the answer at P2(1)). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 17:20:36 +02:00
leonarski_fandClaude Opus 5	9f2696c1ac	Adaptive spot finding: the ring threshold replaces the floor, not the local test ... The self-calibrating finder was meant to replace the classic finder's FIXED PHOTON FLOOR with a per-resolution-ring threshold read off the image's own noise. As written it replaced the local-box SNR test as well, and that is the defect: a whole-ring threshold is an ABSOLUTE contour with no feedback from a pixel's own surroundings, so the area a spot puts above it grows as sigma^2*ln(peak/threshold) and never saturates. Measured on a strongly diffracting rotation set, the detected footprint grows by +8.05 pixels per e-fold of peak, so the brightest reflections came out as 100-500 pixel blobs and were then discarded for exceeding the size bound - every one of the ten strongest on an image. Intersecting with the local box gives -0.24 pixels per e-fold, the classic finder's own number to two decimals. WHY the local box is the right partner, rather than merely the incumbent: it is a prominence rule whose reference level is a 961-pixel mean. A spot inflates the box's own variance and the peak divides out of the acceptance test, so it cuts at a fixed FRACTION of the spot's own height. Referring that level to fewer pixels makes it inherit their shot noise - at FIXED footprint, estimating the level from 961 pixels, from 25, and from the single maximum gives centroid residuals of 0.524, 0.539 and 0.656 - so flat growth and a stable centroid turn out to be two ends of one dial. A contour on the bare maximum has the flattest growth of anything tried (+0.1) and merges worst. The two arms bind in different regimes, which is why intersecting beats choosing: on serial stills the ring threshold is 0.6x the classic floor, on this rotation sweep 2.3-6.0x. Stills are a strict no-op - 175 components against 175, identical per frame - so the +40% in stills indexing that the adaptive threshold was introduced for is untouched. What it buys, stated as one fact rather than two. Across five geometry pins spanning 1.1 mm it indexes the most frames of any arm tried, 0.831 against 0.803, and integrates 3.04 to 5.76% more observations - but those are the SAME number: regressing observation count on indexing rate over four arms leaves residuals of +/-0.7 percentage points against swings of -7 to +4.5%, so the extra observations ARE the extra indexed frames, not better data per frame. CC1/2, the only statistic here carrying per-observation quality, is +0.66 at one pin and -0.06 at the other: not harmed, not improved. <I/sigma>, ISa and R_meas cannot arbitrate on this data - across those pins each crosses zero as a monotone function of the pin. WHY an absolute contour indexes fewer frames, when its spot list is equal or better on every axis measured - recall, top-1000 recall, centroid, ice fraction, component count - is the interesting part, and it is not a detection effect at all: ITS OWN SIZE BOUND DELETES THE BRIGHTEST REFLECTIONS ON THE FRAME. A component is discarded because it grew past 200 px, and it grew past 200 px because it was bright, so the deletions are drawn from the head of the indexing budget rather than uniformly from it: they are 11x enriched in the top 250 of the thousand spots handed to the indexer, and the bound's own real deletions sit at MEDIAN RANK 12. Turning the bound off recovers 66% and 50% of the deficit at the two pins, against a bar registered at 33% before the run. Three of us dismissed this for most of a day on the grounds that the gates delete only ~4% of what is detected. That arithmetic was right and the denominator was wrong - a rate is not an impact when the thing being lost is selected for the property that makes it matter. Reworking the bound instead was measured and rejected: it recovers half the deficit, and it cannot be done without re-admitting what the bound is for - 68 components past 200 px, of which 8 are real and 60 are junk, where the intersect gets the 8 without the 60. The residual once the bound is off, +1.08%/+1.70%, is the contour itself. Component merging is ruled out separately: geometrically impossible here, 33.9 px minimum reflection separation against components spanning 10 px. So is a ranking effect - the intersect's lead runs +0.06% at --max-spots 250, +3.46% at 1000 and +14.26% at 2000, which is backwards for a selection artefact. Costs 0.48 ms per image in the finder, and 0.044 px of bright-spot centroid precision - measured convention-free, by fitting a line to a reflection's own centroid across five frames, after an XDS-referenced figure proved to be four fifths aperture convention. It also makes the compactness gate above it safe. On the absolute contour that gate is net damage, deleting 37 genuine reflections per ten frames; once the footprint stops growing nothing reaches its threshold at all. Also fixes a real but unexercised defect in PoissonThreshold, where the exact tail handed over to a normal approximation with a step. It changes nothing here: the clipped ring sigma is over-dispersed 1.2-4.9x against sqrt(mu) because it still contains diffraction, so the Gaussian arm wins every ring above mu=50 and none of the 522 thresholds move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FBumeJVx4oeXxiBRpkrE5H	2026-08-28 15:30:39 +02:00