58 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	6f7b136ec2	Bragg integration: a shared signal pixel belongs to the nearer reflection ... Nothing kept a neighbour's flux out of a reflection's own signal disk. The union mask keeps neighbour cores out of the BACKGROUND ring, but the r1 disk was read whole, so on a dense pattern a crowded reflection measures part of its neighbour as its own. Ownership is decided once per image into a per-pixel (quantised distance, reflection) key written with an atomic minimum, so the nearest predicted centre wins whatever order the writes arrive in and the lowest index breaks a tie. `--overlap exclude`, now the default, drops the pixels a nearer neighbour owns from the profile fit. A profile fit is the amplitude of a normalised profile, so leaving pixels out renormalises the estimator by construction and the reflection stays unbiased rather than being discarded; the summation-fallback guard is scaled back to the disk the box-sum seed actually read, so it still compares like with like. `--overlap reject` is the XDS MINPK alternative - drop the reflection when less than `--overlap-minpk` of its expected profile is cleanly its own. A box sum has no profile to renormalise with, so `exclude` is a no-op there and only `reject` acts on it. Widening the split - keeping a pixel only where no other centre is within its distance PLUS a margin - was built and measured, and it is worse monotonically: the residual bias of the pixels that were kept grows from +0.072 to +0.209 in ln intensity at 0 to 3 px of margin. What the margin removes is the reflection's own profile, not the neighbour's tail, so the plain nearest-centre split is the rule. Measured on the full 38-crystal rotation battery against the same binary with the treatment off: ISa better 15 / worse 8, summed shortfall against XDS 39.7 -> 28.1. Three of the losses are the two-pass loop taking its other branch - their median mosaicity moves between the two known attractors - rather than the change under test; excluding those it is better 15 / worse 5 and the shortfall goes 31.3 -> 14.4. The two crowded crystals gain 38% and 52% of their ISa, one of them passing XDS. High-shell CC1/2 over the 35 crystals that neither flipped branch nor carry a collapsed error model is better 7 / worse 7. Space groups unchanged at 35/38. The owner map is built only when a treatment is asked for and costs 1.1% of the battery's wall clock - 23% on a genuinely crowded crystal, nothing where no two predictions touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 05:02:31 +02:00
leonarski_fandClaude Opus 5	adf87e8675	Merging: export the XDS-comparable ISa under jfjoch_diffrn_ISa ... The mmCIF's _reflns.jfjoch_diffrn_ISa carried the strong-reflection asymptote, a tier XDS has no equivalent of, while the name invites comparison with XDS's ISa - which is the whole-range 1/sqrt(a*b). rugnux_vs_xds.py reads that item for the battery's ISa column, so the comparison that column exists to make was between two different quantities, flattering rugnux by the difference between the tiers. Write the whole-range value there, move the asymptote to _reflns.jfjoch_diffrn_ISa_asymptotic, and add _reflns.jfjoch_error_model_a and _b in XDS's convention so the number can be re-derived from the file rather than taken on trust. On a broadband rotation dataset the battery column now reads 13.25 against XDS's 21.18 where it read 15.6 before, and the two error models can be compared term by term for the first time: a 1.538 vs 1.249 and b 3.71e-03 vs 1.78e-03, so the gap is in BOTH the counting and the systematic term (1.23x and 2.08x, and sqrt(1.23*2.08) = 1.60 = 21.18/13.25). This is a deliberate redefinition of an exported item, not an addition: a file written by an earlier version carries the asymptote under the old name and there is no version marker to tell them apart. Noted in the changelog and in docs/CPU_DATA_ANALYSIS.md. Nothing reads the item back into the pipeline - it is written and never parsed by rugnux itself - so no stored file is reinterpreted in a way that changes a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:55:46 +02:00
leonarski_fandClaude Opus 5	ac06b5c64f	Merging: report the error model in XDS's convention ... rugnux fits sigma^2 = a*sigma0^2 + (b*<I>)^2, so its `b` is a fraction of the intensity. XDS fits sigma^2 = a*(sigma0^2 + b*I^2) and prints ISa = 1/sqrt(a*b). The two `a` are the same number, but the two `b` are not - b_xds = b^2/a - so the pair rugnux printed could not be read against a CORRECT.LP, which is the only reason anyone looks at it. Convert at the report. The fit, the merge weights and both engines' variance expressions are untouched, so this is a re-expression and not a change: on a rotation dataset the merged intensities move strictly less between before and after than they do between two runs of the SAME binary (99.9% identical, max |dI/I| 9.1e-4 against the run-to-run control's 7.5e-3), with the same reflection set. The rotation path also printed the wrong ISa for the comparison it invites. What it calls ISa is the strong-reflection asymptote, a tier XDS has no equivalent of and which can only ever be the more optimistic of the two; XDS's ISa is the whole-range 1/sqrt(a*b), which in rugnux units is exactly 1/b. Print both, labelled. On a broadband rotation dataset that is 13.2 (whole range) and 15.6 (asymptote) against XDS's 21.18 - so the number previously compared was flattering rugnux by 2.4. A third, unrelated `b` lives in the space-group search: fitted with the sigma^2 coefficient held at 1, with gate constants calibrated in that convention, and a ratio bound does not survive the mapping (1.90 would have to become 3.61) while the absolute floor has no correct value at all, there being no `a`. It is now commented as such, since making the three consistent is the obvious wrong move. Also corrects three comments and two doc passages that still described a merged-sigma systematic floor deleted in 72efb75a8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:50:22 +02:00
leonarski_fandClaude Opus 5	61d24db59f	Bragg integration: elongate the background ring per reflection ... The signal disk and the r2..r3 background ring were fixed pixel circles, identical for every reflection at every resolution. A reflection is not round: a finite bandwidth streaks it radially by bw_sigma*Rpx, so at high resolution the ring sits within 1.3-2.2 sigma of the reflection's own profile and measures its tails as background. --integration-stencil <k> makes the RING an ellipse, elongated along the beam->reflection direction by k times that streak, capped at 2*r3. The tangential half-widths stay r2 and r3, and the r1 signal disk stays a circle: r1 drives the all-or-nothing n_inner_valid == n_inner gate, so growing it rejects any reflection carrying one bad pixel along a long streak, and the flux a circular r1 loses is a function of resolution alone, which the per-shell scale absorbs. The geometry lives in one shared header compiled by both the host compiler and nvcc, so the seven pixel-classification sites - the CPU mask/main/clip loops and the GPU mark_mask/main/trim/clip kernels - cannot drift apart. Rather than evaluate an ellipse, each pixel's squared distance has its radial part scaled down, d2 - q*rad^2 against r2^2/r3^2 with q = 1 - (r/(r+grow))^2, so grow = 0 gives q = 0 and both tests collapse onto d2 exactly in floating point. The width is the bandwidth streak alone, not the profile's full radial variance, which also carries the sensor parallax and weak-spot capture terms. Deriving the growth from those was implemented first and measured on the rotation battery: at k=1 it took Thau_9's high-shell CC1/2 from 75.8 to 27.9 and Benas_3's from 14.1 to 6.0, against cytC_10 +1.2 and lyso_ref flat. On a monochromatic beam they are the only terms there are, and C_CAPTURE is 64% of them. Keeping only the streak also makes the option exactly inert without a bandwidth, rather than merely small. Default 0. Measured on broadband rotation data with the bandwidth set to its spectroscopic value, matched resolution limits: high-shell CC1/2 30.6 -> 46.4 at k=4, and better in EVERY shell in both CC1/2 and R_meas (top shell R_meas 194.7% -> 138.7%), with completeness, multiplicity and space group unchanged and 28 of 98833 unique reflections lost. Anomalous peak height over 18 sites +0.107 +- 0.039 sigma (p = 0.013). The full 38-crystal rotation battery is unchanged to every reported digit, base against k=3. Two consequences of an elongated ring are handled rather than inherited. The neighbour exclusion marks the inner ELLIPSE in each neighbour's own frame, or an elongated neighbour leaks its tails into this reflection's ring. And the radial-background curvature kernel becomes a small table indexed by the growth, because its azimuthal average makes one kernel serve every reflection only while their stencils are identical; the GPU's radial window, previously a fixed 32 bins, is now sized on the host from the widest ring on the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 15:19:28 +02:00
leonarski_f	0b5fb4fb92	Merge branch 'fix56-work' into integration-variance-fixes	2026-08-09 21:08:39 +02:00
leonarski_fandClaude Opus 5	1239c49731	Bragg integration: separate the three things a bandwidth used to switch ... Setting a bandwidth flipped three unrelated switches at once: it changed the profile's radial capture term, it moved the width measurement from the signal disk to the whole fit grid, and it silently overrode the background clip and trim, so --background-clip under --bandwidth was ignored - the two runs were bit-identical. The width measurement was the damaging one. The fit grid is an azimuthally averaged stack, so its second moment is sigma_r^2 + sigma_t^2 and the radial smear of a bandwidth leaked into the tangential model - a tangential width of 3.04 px against a 1.06 px truth, inflating the effective background pixel count where the weak signal is. The result was a step rather than a slope: on genuinely monochromatic data, declaring a 0.2% bandwidth cost ISa 28.4 -> 22.2. Measure the two widths separately, accumulated in each spot's own radial/tangential frame over the signal disk, from the signed profile cells - away from the peak a learned cell is background noise centred on zero, so the signed sum is unbiased, while clamping it at zero turns that noise into a pedestal the r^2 weight reads as width. The radial term is then the measured excess or the analytic floor, whichever is larger. With the two widths separated there is nothing left for the broadband switch to select, so it is gone - which is the proof the three were independent. The background clip and trim now come from the settings in every case; the tuned 3-sigma broadband default moves to the rugnux front end, which is the only place that knows whether the user gave a value. Monochromatic data: declaring a 0.2% bandwidth now costs ISa 28.4 -> 27.9 rather than 22.2, and forcing the old 3-sigma clip in the new build reproduces the good result, so none of the step came from the clip. On large-bandwidth data CC1/2 improves in 8 of 10 shells. Across 12 monochromatic crystals the space groups are unchanged and CC1/2 moves by at most 0.2 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 21:08:29 +02:00
leonarski_fandClaude Opus 5	72efb75a8c	Merging: do not floor the merged sigma at the systematic term ... The merged sigma was floored at b*|I|, so I/sigma could never exceed the reported ISa. On one dataset every merged reflection came out at I/sigma <= 12.96 with a 99th percentile of 12.77 in every resolution shell alike, while the scatter of the observations implied about 44 and XDS reported 58. The floor is wrong in principle. `b` is fitted from the scatter BETWEEN a reflection's symmetry equivalents, i.e. from the part that is not common to them, so it averages down with multiplicity exactly like the counting term. 1/sqrt(sum_w) with the b-inflated per-observation sigma already gives b*I/sqrt(n); flooring at b*|I| puts the sqrt(n) back. That is the whole effect: 12.96 * sqrt(21.6) = 60, against XDS's 58. It was introduced on a comparison of our MERGED I/sigma against XDS's UNMERGED I/sigma. XDS's own merged low-resolution I/sigma exceeds its reported ISa on 30 of the 39 reference datasets here, median ratio 1.78 and up to 4.23. Merged low-shell I/sigma now lands where XDS's does: 22.4 -> 46.2 against 46.2 on one crystal, 26.7 -> 115.7 against 96.6 on another, 12.5 -> 45.0 against 58.0 on a third. Over the 38-crystal battery the space groups, the merged reflection sets, R_meas and CC1/2 are all unchanged - every one of them is sigma-independent, which is what makes them the right control - and <I/sigma> rises on 35 crystals with none worse. The asymptotic estimator that fed the floor stays, for the reported ISa only, and is repaired in the process: it subtracts a*sigma^2 rather than the raw sigma^2 (at a < 1 the difference is the same size as the b^2 being measured, which is what made it flip between 10.9 and 62.7 on consecutive passes of the same data), it rescales each group's variance median-unbiased before subtracting an unbiased counting term, its I/sigma gate uses the same convention, and it is bounded by the whole-range b - an asymptote exists to refine 1/b upward, not to report 0.3 because "strong" was selected on a sigma scale the fit itself rejects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 20:58:35 +02:00
leonarski_fandClaude Opus 5	a29c36600f	Beam-stop shadow detection, and a low-resolution limit for scaling ... rugnux finds the beam stop and its holder in a projection of 60 images and marks them in the pixel mask as bit 9 (--detect-beam-stop[=N|off], on by default). Reflections behind the stop are attenuated but not flagged, so they integrate low with a plausible sigma and nothing downstream catches them: the signal-box gate requires 100% valid pixels and shadow pixels are valid, the background clip is high-side only, and the |zeta| cut applies only to the space-group search merge. The detection compares each pixel's background against the typical background at the same radius on two channels. An azimuthal one (the ring median) finds the holder arm, which is a minority of its ring; a radial one (the background just outside) finds the disk, which the ring median cannot see because inside a fully blocked ring the median is the shadow itself. Pixels are pooled over a 5x5 box and tested only where the background has actually been counted, so low-background data no longer masks the whole detector. Recorded reflections are carved back out - a beam stop cannot block a reflection that was measured. Bit 9 belongs to the run that found it, not to the dataset: it is cleared when a run starts, so a mask read back from a file that carries one starts clear. The user mask (bit 8) is left alone. Scaling and merging gain a low-resolution limit, default 50 A (--scaling-low-resolution <num>, 0 removes it), applied per observation before scaling so it also protects the per-frame scale fit and the space-group search. 50 A is the value XDS configurations use; rugnux_vs_xds.py now matches both of XDS's resolution limits instead of only the high one, so the lowest shell is the same shell in the two programs. The viewer draws the detected shadow in coral with a "Show beam stop" switch in the side panel, exposes the low-resolution limit in the settings dock, and offers detection in its processing jobs. Adding an image marker meant giving the reader a MIN_REAL_PXL_VALUE, because several places classify a pixel by range rather than by equality and would otherwise read the new marker as a very negative intensity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 01:05:31 +02:00
leonarski_fandClaude Opus 5	df9a9c2a2c	Fix the defects found reviewing the branch before merge ... Image buffer: the per-image CBOR metadata headroom had been re-derived from the online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured worst case - reflections plus the capped spot list plus the three azimuthal arrays - is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and give it a name that both the code and its guard test read: written down twice, the two had drifted and the test kept passing against the value the code had left. Spot finding: an unset low_resolution_limit means no limit at that end, as an unset high_resolution_limit already did. An optional rather than a zero sentinel, because zero is not a natural "no limit" here - every pixel lies above it, so the plain comparison masked the whole image instead of none of it, and nothing validated the zero. The API field is no longer required; a zero is folded into the unset case at the boundary, where older clients still send it, so one spelling reaches the analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9> wraps above 512 A and would have masked everything. image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU file with none, and the path fed by bytes we did not produce. An unchecked synchronise returned the host-written sentinel as if it were a measurement, so the decode looked successful and the fallback to the host decoder never fired. rugnux: --stride no longer writes one past the end of the per-image arrays, whose count floored where the worker loop ceils, and the written process file links the images actually processed rather than the first N - each frame's picture now sits next to its own analysis. Powder calibration: the face-centred calibrants no longer list their systematically absent rings, so the distance fit starts from a reflection that exists rather than an extinct one; the triclinic calibrant covers both signs of h and k instead of a single octant, which is only valid for a diagonal metric. The test asserted the old behaviour - one ring formula for every cubic standard - and is rewritten. CBOR: skip an unknown tagged value in the end block, as the other four blocks already do. One advance lands on the tagged item rather than past it, so an older reader fed a newer end message threw and never finalized its file. Viewer: a settings value the setter rejects no longer escapes as an uncaught throw from a worker slot, and the field offers only what the setter accepts. Space-group search: judge stage B on the same "present" cut stage A already computes. Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged and the centering rescue switched itself off on exactly the weak data it exists for. Where the fixed cut is the smaller of the two they are equal and this is inert: over the 37-crystal rotation battery every crystal reports the identical space group and identical merge statistics, so it is a no-op there and the low-ISa case it targets remains unmeasured. rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped it; that mode also applies the same polarization default as every other mode. Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both spot extractors take their algorithm from, with its citation and its license. The rc.161 change list is brought back to one line per entry, and the user-visible changes that were missing from it added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 18:18:46 +02:00
leonarski_fandClaude Opus 5	5830f78d57	Revert the azimuthal-integration sigma clip ... Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping machinery in AzIntEngine. This is a partial revert of a6be35ccd - the ice-ring-mask removal that commit also carried stays. Sigma clipping remains where it started and where it is needed: inside the adaptive spot finder, at a fixed 3 sigma on raw counts, feeding the detection threshold and the ice score. The option made the workflow harder to reason about than the quantity was worth. It gave azimuthal integration two meanings behind one setting - the bin mean and the background under the peaks - which the azimuthal-integration workflows do not need. It also did not compose with the fused GPU engine, which supplies the profile from its PLAIN pass: on the default rugnux, viewer and receiver path the setting was silently doing nothing (measured, the profile came out identical to the unclipped run to 1e-6 with identical per-bin pixel counts). Making it correct is not a matter of gating that one shortcut - it means separating the workflows (azimuthal integration, MX rotation, MX stills, geometry calibration) and deciding per workflow what the profile is for, which is a larger change than the option earns. The default path is unaffected: over 20 images of a rotation dataset the radial profile, the per-bin pixel counts and the spot counts are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 15:31:19 +02:00
leonarski_fandClaude Opus 5	6194fe6fbf	viewer: calibrate the whole dataset from "Analyze dataset" ... The powder panel could only calibrate the image on screen. Calibration is now a third page beside MX and AzInt, so the dataset button runs it over every image the same way it runs the other two - which is the point, since a powder ring is measured far better by summing a run than by one frame. The page carries the calibrant and the method (rings or spots); the interactive Guess/Refine buttons stay where they were and now share the one calibrant selection, so there is no second combo to drift. analyzeDataset() carries the ProcessMode rather than a bool: a third state was coming, and two bools would have had one combination that cannot be valid. The calibrant list gains ICE, which it could not offer before: the widget worked in unit cells, and hexagonal ice has none that generates its rings correctly (P6_3/mmc would include systematically absent ones). FindCenter now takes the ring list its first line used to derive, so the interactive path gets ice as well. The result window leads with the residual rms rather than the fitted sigma. The sigma is a formal scatter estimate and understates a bad fit badly - measured on ice, 0.215 px reported against a 1.70 px residual - while the rms separates a usable fit from one that has locked onto the wrong thing. A rings run needs the profile binned in azimuth; below four sectors it returns nothing at all. The viewer raises the count to 32 exactly as the CLI does, and says so in the panel and in the job dialog rather than doing it silently. Also fixes a CLI inconsistency this comparison exposed: rugnux's calibration branch never applied the standard offline analysis defaults, so it measured the rings in a profile built with the file's polarization factor while every other mode - and the viewer - uses 0.99. Found because the two disagreed by 0.005 px in PONI x, and confirmed by reproducing the viewer exactly with --polarization 0.99. With it applied the CLI and the viewer write byte-identical .poni files on LaB6 by rings, LaB6 by spots, and an iced dataset over 1800 images. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:45:39 +02:00
leonarski_fandClaude Opus 5	6468dd13be	rugnux: --mode, and detector calibration from powder rings ... --azint-only and --scale are replaced by --mode mx|azint|scale|calibration, with mx the default. The old flags are removed rather than aliased. Calibration mode fits the detector geometry - PONI x/y, the two tilts and the distance - to a calibrant's powder rings and writes a pyFAI .poni alongside a report of how far each parameter moved from the header. Bragg data constrain the beam centre worst, because it is gauge-coupled to the crystal orientation; a powder ring has no orientation to couple to. --calibrant takes lab6, agbh, ceo2, si or ice. A calibrant is a list of ring positions rather than a unit cell, because hexagonal ice is P6_3/mmc: rings enumerated from its cell would include systematically absent ones. So the crystalline standards generate their rings from a cell and ice carries the measured list, and RingsFromAzimuthalProfile, GuessGeometry and OptimizeGeometry all take ring q. The calibrant table is shared with the viewer's powder panel, which previously carried its own copy. --calibration picks how the rings are measured: rings (default) sums the (q x azimuth) profile over every processed image and fits the arcs in it; spots pools the found spots and fits those. Both use the whole run, with -s/-e/-t selecting images. rings defaults --azim-phi-bins to 32, since a profile with one azimuthal bin has averaged the ring over every direction and cannot locate it. Two fixes this exposed: The extraction window is capped at half the gap to the neighbouring ring. The background under a peak is taken from the ends of its window, so a window wider than half that gap measures the next ring's flank as this ring's background - and hexagonal ice has three rings within 0.06 1/A. Ice calibration was 3.5 px out before this and 0.29 px after; LaB6 is unaffected. RingOptimizer holds rot1/rot2 fixed when only one ring is present. A tilt and a centre offset both move a ring as cos(phi) and are separated only by the tilt's amplitude growing as the ring radius squared, so on a single ring they are exactly degenerate. Measured. LaB6 at five distances: the fitted direct beam is within 0.36 px of an independent implementation out to 300 mm, and D = -0.046 + 1.000788 dtz with an rms of 0.011 mm. At 500 mm one ring is fully on the detector and a second only clips the corners, which is not enough to constrain a tilt - restricting the q range to the resolved ring recovers 0.06 px. Ice: 5.53 -> 0.29 px on one crystal and 4.71 -> 0.80 px on another, against XDS's refined direct beam. On an ice-free crystal the fit is worse than the header, which is the correct outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:00:03 +02:00
leonarski_fandClaude Opus 5	1df9556ec1	Bragg integration: default the radial background correction off again ... Auto rode in with the ice work rather than on its own evidence, and measured over the 37-crystal rotation battery it does not carry itself yet. It TARGETS correctly - it fires on ten crystals and every one is ice-positive, no failures, no space-group changes - but it costs 1.35x the wall clock (median +3 s per crystal, worst +29 s) and on the merge statistics it is the familiar sign-mixed trade: high-shell CC1/2 worse on three of the four crystals that move materially, mean -0.76. The case for it is real but rests on agreement with a fixed external model - 43 % of the ice bands' excess amplitude removed on smooth ice, the effect 7x stronger inside the bands than outside - which is the better arbiter and also the narrower one. That deserves settling on its own, not riding along with a set of ice defaults. `--background-radial=auto` keeps it a flag away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:29:42 +02:00
leonarski_fandClaude Opus 5	e9e3dac1b8	rugnux: say what the radial background correction decided, even in auto ... The "radial curvature correction on/off" line was printed only when --background-radial was given explicitly. In auto - the default - it said nothing, so a run's log carried no record of whether the correction had been applied. That is not cosmetic. Auto decides per image from that image's ice score, so two runs of the same data with different flags can differ substantially with nothing in either log to explain it: a crystal whose high-shell CC1/2 read 8.1 % with the correction pinned off and 4.5 % under the default looked like a regression for some time before the flag turned out to be the whole difference. Log the effective mode unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:22:22 +02:00
leonarski_fandClaude Opus 5	a6be35ccdb	Azimuthal integration: optional sigma clipping of the reported profile ... The profile is the MEAN of each bin, so a few strong reflections landing in a bin lift it exactly as a smooth powder ring does. That is the wrong quantity whenever the profile is wanted as a background rather than as a measurement of what is in the bin - the ice score being the case in point, where reading a plain profile INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals had no ice at all. The adaptive spot finder already computes the right thing, a sigma-clipped per-resolution-ring background, as a byproduct of its own threshold. Where it runs, the ice score uses that. Where it does not - --no-adaptive-spots, --azint-only, and anything reading the profile the broker wrote - there was no way to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip), 0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian bin and biases the estimate low rather than removing outliers. Two clip passes follow the plain one, matching the finder's recipe - the first pass's standard deviation is itself inflated by the peaks being removed, so one pass leaves a threshold that is still too generous. A bin with fewer than eight pixels is left alone: at the detector edge and behind the beam stop there is no spread to clip on. Both engines do it. On the GPU the accept range is computed by a small kernel and stays resident, so a clip pass is one more read of the same pixels and no round trip; the two accumulation kernels take the range as a pointer that is null on the plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict, against a 0.34 ms per-image total. Note what the result IS: the smooth background under the peaks, not the bin mean. It should not be switched on where a ring's integrated intensity is wanted - the powder-ring geometry fit reads ring peaks, and those are what a clip is designed to remove. Off by default, so nothing changes unless it is asked for. Not exposed over the REST API - that needs the generated model regenerated, which is a separate step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:18:52 +02:00
leonarski_fandClaude Opus 5	b5f5879a1d	rugnux: measure the ice in the first pass, and always find its own spots ... Ice handling was gated on a measurement the run only made AFTER the images had been processed, so the per-image pass could not use it. The flagging therefore ran unconditionally: ice-band spots were ordered last in the --max-spots budget and held out of the indexer seed and the geometry refinement on every crystal, iced or not. The eleven bands are fixed geometry holding 16-26 % of the unique reflections whether or not there is ice, so on a clean crystal that discards a fifth of the spots - the strongest first - for nothing. Measured on a crystal whose gate never fires, that moved the merged data by a mean of 0.85 sigma against a run-to-run floor of 9.3e-5. Measure it in the first pass instead. That pass already looks at ~100 images spread over the sweep, and it already stops at the spot finder, so it sees the azimuthal profile for the smooth channel and the unfiltered connected components for the spot channel. Both counts SpotAnalyze takes are pre-filter, so pooling them there is the run's own verdict, reached before anything has been discarded and in time for the pass that acts on it. Where the sample sees no ice, the run indexes on the ice-band spots too. It has to be the whole sample: the spot channel is a ratio pooled over images, because one frame carries a handful of control spots. A per-image gate is not an alternative - two of the crystals whose indexing this rescues fire on that channel alone, at profile scores of 1.12 and 1.22, so gating per image on the profile score would drop exactly the cases that matter. This also removes the first-pass spot reuse, and with it --redo-rotation-spots and the reuse path. Finding the ~100 first-pass spots costs little, and reusing was actively wrong here: the stored spots were found online at the acquisition's threshold and have already had their ice-band entries ordered last and dropped by its spot budget, so counting ice from them under-reads it by construction, and the lattice search never saw the spot-finding settings at all. It also removes the need for the machinery that re-found spots whenever a spot-finding option was named, which made those options impossible to A/B. IndexAndRefine cached index_ice_rings at construction, which happens before the first pass; it holds a reference to the experiment, so it now reads the setting where it uses it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 20:48:22 +02:00
leonarski_fandClaude Opus 5	1bc2ca9125	rugnux: do not re-find the rotation spots just because --detect-ice-rings was named ... First-pass rotation indexing reuses the spots the acquisition wrote, and does NOT re-mark them - their ice flags come from the file. So --detect-ice-rings can only invalidate those spots when it asks for something the file did not do. It was flagged as a spot-finding option, which forced reuse_rotation_spots off whenever it appeared, so naming it swapped the acquisition's spots for this program's own and moved the first-pass lattice by itself. Measured over the 37-crystal rotation battery, passing the SEMANTICALLY NULL --detect-ice-rings=on to files that already carry detect_ice_rings=1 changed the merged data on every crystal, sent one crystal's ISa from 1.66 to 0.38, and lost MyoB_13 to indexing failure outright. Both arms of any A/B on the flag therefore moved for a reason that had nothing to do with ice, which made the flag impossible to test. Re-find only when the requested value differs from the file's, and say so when it happens. A file with no key at all counts as "did not mark", which is what its stored spots show - such a dataset carries no per-spot ice flags to reuse. Verified over the battery, with the merge mask and the radial background pinned off so this is the only variable. --detect-ice-rings=on on the 36 keyed crystals: all 36 log "using the spots stored in the file", the re-finding line appears nowhere in the arm, unique counts are identical on every crystal, the largest mean |dI|/sigma is 3.4e-5 against a repeat-run floor of 9.3e-5, and not one of R_meas, CC1/2, CC1/2_hi, ISa, completeness, SigAno, d_min or space group differs anywhere. MyoB_13 indexes again. The one file carrying no key reuses under =off and re-finds under =on, as it should. --detect-ice-rings=off still re-finds, since it does differ from those files, and two crystals still fail to index there. That is not this change: a control that re-finds with ice marking ON indexes both. With the marking off, ice spots are no longer ordered last, so they consume the --max-spots budget and the first pass collapses. With the confound removed the flag can finally be measured, and on a comparison whose spot source is identical on both arms it is clearly worth having - though the win is at INDEXING rather than at the scale fit. Three crystals are saved outright (one would otherwise collapse to P1 at CC1/2 19%, one loses half its completeness and its screw axis, one loses its F-centred cubic lattice), two more only index with it on, and the remaining eight gate-fired crystals differ by well under 1% in R_meas and CC1/2 in both directions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 20:18:17 +02:00
leonarski_fandClaude Opus 5	f0cdb027e1	Ice: default the merge mask off, gate the radial background on smooth ice, and pick detection by geometry ... Three defaults, each settled by measurement rather than by argument. The arbiter throughout is structure-referenced - anomalous peak height where a crystal can carry it, and otherwise the agreement of the ice bands with a fixed external model against resolution-matched DECOY bands carrying no ice. The band-versus-decoy contrast is used because R-free here tracks completeness, and every one of these switches moves completeness. The damage is real and it localizes: over the rotation battery the ice bands' excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst, while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the ten largest excess peaks in a q scan land on hexagonal ring positions. Turning ice handling off leaves the contrast unchanged and forcing it on a clean crystal does not create one, so it is the ice and not the machinery. MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band reflections in the merge and exclude them only from the model fit; autoPROC is the sole exception. On the one battery crystal where the mask fires and an anomalous arbiter can score it, dropping the band moved the mean peak height at the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing 1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own 3.05 - better than average data - and costing 17 completeness points in that shell. It fires on 5 of 37 crystals, changes no space group, and those 5 disagree in sign: it clearly helps the two most heavily iced, is a wash on two and costs a third. So it stays as a switch, worth setting by hand on a badly iced crystal where it shows in the high shell, but it is not a default. RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background as a function of radius alone, and that is exactly when it works. On a crystal with pure smooth powder ice it removes 43% of the bands' excess amplitude, with the improvement 7x larger inside the bands than outside; on a crystal whose ice is discrete crystallite spots - no smooth ring to model - the excess amplitude GREW by half; on clean data it is inert to four decimals. The two ice channels already separate those morphologies, so --background-radial takes on|off|auto and auto applies it to an image when that image's peak-excluded score reaches --ice-min-score. Auto never engages without such a score, because the plain profile carries the Bragg peaks and cannot support an absolute threshold. Per image rather than per run, and that was tested rather than assumed: the gate fires on 100% and 94% of frames on the two crystals that want it, and on 1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice crystal. A seam statistic against off + f*(on - off) is null on both mixed runs, every merge statistic is bracketed by the pure arms, and the textured crystal's auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need the score before the pass that integrates, i.e. rotation-only plumbing, and buys nothing measurable. The kernel was already built unconditionally, so flipping the flag per image is free - except on the GPU, where the launches were gated on a construction-time n_rad. That is why the buffers are now allocated whenever the correction could run, and Run() decides per image. DETECTION -> the geometry's default when the file is silent: on for rotation, off for stills, with the command line and then the file taking precedence. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent systematic and the presence gate keeps it inert on a clean crystal; a serial stills run has too few spots per image to spend any on flagging. The master file's key is kept as written rather than collapsed to a bool, so "the file said nothing" is distinguishable from "the file said no" - it used to fall silently to off, taking the exclusion from the scale fit with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 19:06:30 +02:00
leonarski_fandClaude Opus 5	61a7c91b90	Ice: detect it on two channels, and only handle it when it is there ... The per-image ice score was read off the PLAIN azimuthal profile. That profile is a per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it exactly as ice would. Measured over 37 rotation crystals, that did not merely add noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all (4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its 99th percentile and 2.70 at its maximum, so that metric cannot support any absolute threshold whatsoever. The adaptive spot finder already computes the right input for its own threshold: a sigma-clipped per-resolution-ring background, in the same bins. A powder ring is azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at 2.08-2.37, against a decoy null that never exceeds 1.29. That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE spots and leaves the radial profile flat. So a second channel counts found spots on the rings against the same q width of ice-free flanks beside them. The two barely overlap - the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6, while a clean crystal reads 1.04 on both. Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now all skipped when neither channel sees any. The gate is applied in the full pipeline and in --scale, which reads the stored per-image values back out of the _process.h5. Also fixes the merge-time mask's control: the shoulder now excludes reflections that are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on the neighbours and the test compared ice against ice. Measured, that is the only thing this changes: it removes firings on those three rings and leaves every other firing's CC pair identical to three decimals. And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the same data got a narrower band online than the measured ~0.06 ring FWHM justifies. Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses. Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3 and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa 3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The one crystal that loses reflections improves on both R_meas and CC1/2. Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5 write/read and the receiver plots are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 16:17:23 +02:00
leonarski_fandClaude Opus 5	0e23fd3ab9	Bragg integration: propagate the background-estimate uncertainty, add an opt-in radial background correction ... Two independent pieces in the same code path. The background-estimate variance was never propagated. A reflection's background comes from a finite ring of n_b pixels, so subtracting it adds var(B)/n_b per signal pixel - sqrt(1 + n_d/n_b) = 1.109 with the shipped stencil. Both engines omitted it, which is exactly the 1.11-1.19 gap measured between the off-ring scatter and the reported sigma. Three lines each; it affects every dataset, not only iced ones. The radial correction is new and OFF by default (--background-radial). The signal disk and the background ring are concentric, so for any background LINEAR in position <B>_ann == <B>_disk identically and a plane fit buys nothing; the leading error is the CURVATURE of the radial background, which on a sharp ice ring reaches +26 counts on a single reflection. Since every reflection uses the same stencil, that error is a fixed kernel over radial offset - one short dot product per reflection and no extra pixel reads. Validated on empty apertures before any C++: mean |bias| over 9 bands / 3 crystals 4.33 -> 0.79 counts with the scatter unchanged. Three things it cost a battery each to learn, all now in the code: - the radial curve must be accumulated from CLIPPED annulus pixels, inside the clip pass, or it carries neighbour tails and zingers (so it is inert under --integrator boxsum, which has no clip pass); - the GPU version was a 1.8x slowdown from atomicAdd contention on a small radial array - staged in shared memory per block it now costs nothing measurable; - it is battery-NEUTRAL as a default, because the reflections whose bias it fixes are the ones the ice handling already excludes. Hence off by default. CPU/GPU parity extended with two radial sections: 9002 assertions. Also fixes a latent French-Wilson quadrature collapse: j_max = I + 8 sigma on a fixed 400-point grid degenerates to a single cell once sigma >> 50 <I>, giving F = 0.1 sqrt(sigma) with sigmaF -> 0. Harmless today, but any sigma-inflation scheme detonates it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 15:44:13 +02:00
leonarski_fandClaude Opus 5	24ac1a90fc	rugnux: name every space group the data cannot separate in the summary ... The search already knows when several groups share an absence pattern - it scores them identically, marks them all in the candidate table and prints "Best space group: I23 or I213 (indistinguishable from these data)". The one-line summary then dropped that and reported only the representative, so the run's headline answer claimed a decision the diffraction had not made. Carry the alternatives through to the summary. It already has them: ProcessResult holds the whole SearchSpaceGroupResult. Space group: I23 (No. 197) or I213 (No. 199) - indistinguishable from these data Some of these pairs are enantiomorphs (P4_1 vs P4_3), where the choice needs phasing or anomalous signal. Others are not, and are worth naming because they surprise: I23 vs I2_13 and I222 vs I2_12_12_1 differ only by a screw whose condition h00: h=2n is ALREADY implied by the I-centering condition h+k+l=2n, so the screw has no observable signature at all. Checked over 35936 reflections with gemmi, the two absence patterns are identical - not nearly, but exactly. Of the 65 chiral space groups, 13 classes are indistinguishable this way, the largest being the four-way P3_112 / P3_121 / P3_212 / P3_221. The representative stays the lowest space-group number, which is why a cubic insulin comes out I23 where the deposited convention is I2_13. That choice is a convention and the summary now says so instead of implying it was measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 22:20:43 +02:00
leonarski_fandClaude Opus 5	bb7df09086	rugnux: separate the merge-time ice-ring mask from ice detection ... --detect-ice-rings did two unrelated jobs at once: flagging ice spots so indexing de-prioritises them and keeping ice reflections out of the scale fit, AND gating the merge-time mask that drops a decorrelated ice ring and re-merges. Turning it off to de-confound a merge-stage experiment therefore also changed how the data were indexed - measured, that breaks indexing outright on two of the 37 rotation battery crystals - while leaving it on lets the mask land differently between two arms of an experiment and contaminate the comparison (measured on up to 19 of 37 crystals in response to a small intensity change). Add --ice-ring-mask[=on|off], default on, gating only the merge-time mask. Verified with =off: ice-spot flagging and the scaling exclusion still log and still apply, no mask line, no second merge, and the first error model is bit-identical to the =on arm. The full pipeline and the offline --scale path reach the same verdict on the same data, as they must. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 19:00:10 +02:00
leonarski_fandClaude Opus 5	09fb8e0306	Bragg integration: clip the background ring high side instead of trimming it ... The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A symmetric trim is not a consistent estimator of the mean of a right-skewed (Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true mean at every level, and with ~50 signal pixels in the r1 disk that under-subtraction adds ~5 counts to every partial on every frame. Measured two independent ways on four rotation datasets - stored background_mean against a plain ring mean over the same pixels on reflection-free frames, and directly on apertures that provably hold no reflection. Empty-aperture pedestal, counts: plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for monochromatic data. It rejects the same one-sided contamination the trim was there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds ~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The trim stays reachable with --background-trim for back compatibility; setting either estimator clears the other, so they can never stack. --integrator boxsum does not take the clip (matching what the shipped clip already did), so it now uses the plain ring mean unless --background-trim is given. The intensities get measurably more accurate: per-shell agreement with an independent processing of the same images improves on 14 of 16 crystals (weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR - absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and CC1/2 in the outer shell improves on 14/7. EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by information-free counts, so both get worse when the bias is removed; neither is evidence against this change. That fingerprint is exactly how the trimmed mean was accepted in the first place. Known cost: over the 37-crystal rotation battery the de-novo space-group count goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal whose merge fails the absolute-sanity gate under either background (R_meas 63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the very operator being scored, so its operator CC is diluted by construction. No other crystal changes space group, and twin protection is not weakened - the H-ratio veto that refuses genuinely twinned crystals gets MORE decisive (1.63 -> 1.84, 2.83 -> 3.99). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 18:03:11 +02:00
leonarski_fandClaude Opus 5	f25fea7024	rugnux: keep 1000 spots per image instead of 250 ... Offline reprocessing is not bound by the online spot budget, and the cap is applied at the end of SpotAnalyze, so it is exactly the spot list the indexer and the per-image refinement see. jfjoch_viewer already sends 1000, so the two front ends now agree on the same file. Measured as a paired A/B over the 37-crystal rotation battery, de novo, with the resolution and Friedel setting matched to the XDS reference, both arms from the same binary bar this constant: R_meas low shell 16 better 0 worse 19 unchanged R_meas 14 better 4 worse 17 unchanged ISa 14 better 6 worse 15 unchanged CC1/2 6 better 3 worse 26 unchanged Low-resolution R_meas is a clean sweep. Around half the battery is bit-identical: those frames never reach 250 spots, so the cap never bound. Wall clock is unchanged (10m00s vs 10m44s, uncontrolled for page cache). Known cost, and the reason this is its own commit: one crystal in the battery reproducibly loses symmetry, tetragonal 422 -> orthorhombic 222, doubling its asymmetric unit. Its R_meas and ISa "improve" there, but that is what merging in too low a symmetry always does, and the lower symmetry then admits a merohedral indexing ambiguity. An intermediate cap of 500 demotes it too, so it buys none of the safety. This is the known point-group-decision-moves-with-data-amount fragility of the space-group search rather than an argument for starving the indexer of spots - the search is the thing to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 22:08:21 +02:00
leonarski_fandClaude Opus 5	2473e03cf7	rugnux: default --spot-sigma to 4.0, the value the viewer already uses ... The two front ends disagreed on the fixed-threshold spot finder: rugnux started from 3.0, jfjoch_viewer from the SpotFindingSettings default of 4.0, so the same file processed either way could give different spots. Inert on the default path - the adaptive finder derives its threshold from each image's own per-resolution-ring noise and never reads signal_to_noise_threshold (only ImageSpotFinderCPU/GPU and DetModuleSpotFinder do). It changes behaviour only under --no-adaptive-spots, and there it now matches the viewer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 22:07:58 +02:00
leonarski_fandClaude Opus 5	0dd837e8e4	rugnux: name the third correction surface in the usage message ... --no-scaling-corrections said it disabled the decay and absorption surfaces. It disables the modulation surface too - GetCorrectionSurfaces() gates all three - and the flag has worked that way since the detector-plane modulation was added; only its description did not follow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:52:41 +02:00
leonarski_fandClaude Opus 5	ea667cb306	rugnux: handle ice rings in --scale as the full pipeline does ... --scale did none of the ice handling the run that wrote the _process.h5 had done, so re-scaling a stored dataset silently produced a different - and flatteringly more complete - answer than the pipeline it was meant to reproduce. Three separate gaps: * --detect-ice-rings was accepted and ignored. The --scale block returns before the line that applies it. * Reflections were never flagged as sitting on an ice ring, so the per-image scale fit included them. The flag is not stored per reflection, so it has to be recomputed from the resolution. * RotationScaleMerge was constructed with the ice half-width hardcoded to zero. That is what turns a resolution into a ring index, so every ice test inside the merge was a no-op whatever was passed to it. The CC1/2 ring test that decides which rings to drop moves into FindDecorrelatedIceRings, shared with the full pipeline so both reach the same verdict on the same data, and --scale now re-merges with the mask the way the pipeline does. The stills branch re-runs only the merge: the scaling has already been applied to the reflections and repeating it would compound it. Measured on a rotation dataset with three decorrelated rings, --scale went from 8765 unique / 36.3% completeness / R-meas 18.5% / <I/sig> 1.1 to 7638 / 31.6% / 18.0% / 1.3, against the full pipeline's 7692 / 31.8% / 17.9% / 1.3 - the reported completeness had been inflated by reflections the pipeline drops. The full pipeline is bit-identical across the refactor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 19:49:55 +02:00
leonarski_fandClaude Opus 5	bfb8cb813c	rugnux: report per-image cost honestly instead of per-worker blocked time ... Each stage timer measures wall time inside one worker, so it counts whatever that worker spent blocked on a contended resource - above all the single GPU - as well as its own work. Those waits overlap across workers, so the mean was printed as if it were the per-image cost when it is roughly the per-image cost times the worker count. At the default thread count on a large detector the reported total came out more than twenty times the truth, and single stages were printed as several times the entire per-image budget of the run. That is the one output anyone tuning performance reads, and it sent this investigation at the wrong stage for a while. Divide by the worker count. It is a lower bound - a worker idle rather than blocked is not counted - so rather than hide the remainder, report the image loop's own wall time next to it, and with it the time spent OUTSIDE the loop. Nothing measured the latter before, yet on a rotation run the first-pass indexing and the scaling and merging can be more of the run than the per-image work is: on a large-detector run here it is 5.1 s against 3.0 s. Both figures are for the last pass, and a two-pass rotation run does all of it twice. Also stop printing nan. The per-image indexing and scaling timers are never fed on the two-pass rotation path, because the lattice is forced rather than searched per image and the merge happens outside the loop, so every default rotation run reported "indexing nan scaling nan". A stage that did not run is now simply absent. Measured against the loop's own wall clock on a 18 Mpx dataset: 5% at one worker, 11% at eight, 29% at thirty-two, versus 23x too high before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-01 14:49:45 +02:00
leonarski_fandClaude Opus 5	2238290d6d	rugnux: make spot settings reach rotation indexing, and stop over-claiming ... Three small honesty and cost fixes on the two-pass rotation path. Spot-finding settings did not reach the step that determines the unit cell. The two-pass first pass reuses the spots stored in the file whenever it has them, and reuse is the default, so both sampling schemes and the validation loop ran on acquisition-time spots while only the per-image pass saw the command line. Every --spot-* option was therefore a no-op for the lattice search on any file written by this software, silently, and the lattice was cross-validated against one spot set and applied to another. Giving spot settings now implies re-finding them for the first pass as well, and plain reuse says so in the log. The summary printed a space group and unit cell even when nothing indexed. With a zero indexing rate the cell is whatever the lattice search happened to return, no reflection was ever measured on it, and no output file is written - so stating it as the run's answer claims a result the data do not support. Say that no lattice was determined instead. The second pass re-indexes de novo so the cell comes out self-consistent with the post-refined geometry, and its result was already checked against the first pass - once by the supercell test and once by the centring test - but only after every image had been integrated with it, so a disagreement cost a whole extra pass on a dataset that ended up on the first pass's lattice regardless. Compare them at the point the lattice is adopted instead, using the same two tests and the same fallback. A triclinic de-novo cell is left alone, being the demotion the merge reindexes. Measured over the 37-crystal regression set: merge statistics are unchanged on every crystal (the two that move are the known rotation-indexing non-determinism - one observation in 2.9 million, and a zero-score lattice landing on no partials instead of a few). Crystals whose data were already cached in the reference run are unchanged in wall time. The one dataset that was burning a discarded pass went from three passes to two, 509 s to 198 s, against 0.81x for the same-detector dataset that was already running two passes - so about 214 s of the saving is the removed pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 23:13:11 +02:00
leonarski_fandClaude Opus 5	797a28a572	rugnux: correct the stale adaptive-spots default comment ... The declaration still said "stills on, rotation off". Adaptive detection has since been turned on for both workflows - adaptive_spots.value_or(true) - which the comment at the assignment already explains. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 15:03:33 +02:00
leonarski_fandClaude Opus 5	0ca159449f	Bragg integration: integrate as far as the detector reaches, not to a fixed 1.0 A ... BraggIntegrationSettings::DMinLimit_A had a setter that nothing anywhere called, so it was always its 1.0 A default - in rugnux, the viewer and the broker alike, with no option or API field to change it. It feeds the predictor as high_res_A, which discards any reflection with |q| > 1/d_min, so integration simply stopped at 1.0 A however far the detector reached. Five of the 33 rotation test datasets have detectors reaching past it, down to 0.981 A. On one of them, run with no resolution limit, the shell table ended dead at 1.00 A with that shell still at CC1/2 55.6% and <I/sig> 3.4 - cut mid-shell rather than fading out. This branch had already made the sibling limits detector-driven (spot finding, scaling), so the pipeline was finding spots the detector could see and then refusing to integrate them. Make it a std::optional: unset means as far as the detector reaches, a value limits. The limit is only a bound on how far the lattice walk goes, never a second opinion on what is measurable - both predictors independently drop reflections that miss the detector (BraggPrediction.cpp, BraggPredictionRot.cpp) - which is what makes the detector's own reach the right default. rugnux gains --integration-high-resolution (0 = no limit, as for --spot-high-resolution); the derived per-axis prediction range resolves against the same number, so the two cannot drift. Full battery: 30/33 space groups, unchanged from before, 0 failures and the same three known mismatches; 22 of 32 crystals bit-identical and nothing worse than 5 observations in ~500k. The datasets that gain do so because their detector reached past 1.0 A - the effect is understated here because the harness caps each merge at the XDS resolution anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 14:03:34 +02:00
leonarski_fandClaude Opus 5	b0e315e73c	Bragg prediction: derive the lattice walk from the cell, and expose it in the API ... Follow-up to making max_hkl a setting: it is now an optional, and unset means "take it from this crystal". The predictor keeps only |q| <= 1/d_min and h = a.q for the real-space axis a, so |h| <= a/d_min exactly - and likewise |k| <= b/d_min and |l| <= c/d_min. max(a,b,c)/d_min therefore bounds all three at once: nothing that could be predicted lies outside it, and nothing inside it is reached by a shorter axis. It applies to rotation and stills alike, both going through the one place the prediction settings are built. Offline (rugnux, viewer) the default is unset, so every crystal gets its own range; --max-hkl overrides it. Online the broker holds a concrete number, because the cost is the cube of it per image and a live acquisition should not have its frame rate decided by whichever sample is mounted: max_hkl joins bragg_integration_settings in the OpenAPI with a default of 100, so an omitted field arrives as that default (the generated model carries it) rather than as "derive it", and the frontend exposes it next to the integration model. Measured against a fixed 100 on six rotation crystals: three are bit-identical, two were being truncated and recover 419k and 5.8k observations with the high-shell CC1/2 going 15.1 -> 25.8% and 52.1 -> 55.3%, and the space group is unchanged 6/6. It reproduces a fixed 200 exactly, which is the bound being tight rather than merely safe. The sixth is worth recording: a 149/83/226 A cell derives 227, and because a single scalar has to cover the longest axis the cube is ~16x what a per-axis box would be - 22% wall clock, for a net 22 observations out of 364k (the per-frame 65536-reflection cap re-selects at the margin when more candidates are offered) and identical CC1/2, ISa and space group. Per-axis limits would remove that; the predictors already map a thread index to h, k and l separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 13:10:58 +02:00
leonarski_fandClaude Opus 5	a06c06931f	Bragg prediction: how far to walk the lattice is a setting, not a literal ... max_hkl was hardcoded to 100 at the one place production builds the prediction settings, so the only way to change it was to edit and rebuild - and it is not a constant of the method, it is a property of the cell. An axis is truncated once a/d_min exceeds it: 100 covers a 150 A axis at 1.5 A, but the same axis at 1.0 A, or a 250 A axis anywhere, loses its outermost reflections with nothing said. Move it into BraggIntegrationSettings next to the other prediction/integration parameters and add rugnux --max-hkl (1..511, default 100 - no behaviour change). Like the integration radii and the background trim it stays out of the OpenAPI, so the broker keeps the default it has today and live analysis cannot be handed a range that would not finish; the offline front end, which knows its cell, can ask for more. RugnuxCommandLine emits it when it is not the default. Measured on five rotation crystals at --max-hkl 200: two are bit-identical at no cost, and three were being truncated - one gains 419k observations (+17%) and takes its high-shell CC1/2 from 15.1% to 25.8% for +14% wall clock, the other two gain 12k and 5.8k observations with CC1/2 76.6->82.4% and 52.1->55.3% for +9% and +1%. ISa is unchanged throughout, and no frame overflowed the prediction buffer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 12:00:45 +02:00
leonarski_fandClaude Opus 5	dd30061005	rugnux: two settings the CLI collected and then discarded ... --polarization was applied with the other geometry overrides, but configure_offline_output runs afterwards and calls ApplyRugnuxExperimentDefaults, which sets the polarization factor unconditionally. Every full-analysis run used 0.99 whatever was asked for, so the Lp correction was wrong at a beamline with different polarization. Apply it after the defaults instead, and stop claiming in RugnuxDefaults.h that nothing here is user-selectable. --scale built a bare ScalingSettings and re-derived the rotation/stills split by hand rather than calling RugnuxDefaultScalingSettings, which is what the split was factored out for. It got scale-fulls, smooth-G, min-captured-fraction and outlier rejection right and dropped CaptureUncertaintyCoeff on the floor: 1.0 in the pipeline, 0.0 here. So re-scaling a rotation _process.h5 gave different sigmas and ISa than the run that wrote it - the exact failure the block's own comment says it exists to prevent. Start from the shared defaults and apply the overrides on top, which also picks up --mosaicity and --search-min-zeta, and let -C bind here too. REJECT_OUTLIERS_DEFAULT_NSIGMA had no reader left afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:50:41 +02:00
leonarski_fandClaude Opus 5	e8a7696ffe	Viewer: process a dataset with the same defaults the CLI uses ... "Analyze dataset" and `rugnux` with no options are two front ends onto the same library and are meant to agree, but they decided their defaults separately and the two lists had drifted. The viewer was missing: - the de-novo starting point. The CLI discards the cell and space group stored in the input file before it does anything; the viewer left them on the experiment. Rugnux only searches for a space group when none is set, so the search was skipped entirely and the stored group was reported straight back. That is self-reinforcing: a finished job's own _process.h5 becomes the active snapshot, so a run that ended in P1 pinned every later run to P1 - which is what "lysozyme keeps coming out P1 in the viewer" was. - the polarization factor, so the Lp correction was omitted altogether. The missing factor is azimuthal and intensity-proportional, and symmetry mates sit at the same 2-theta but different azimuth - the exact "unequal intensities forced together" signature the space-group search vetoes as pseudo-symmetry, which can land a genuinely de-novo run in P1 on its own. - five rotation scaling defaults: the smooth-G range, the minimum captured fraction, the capture-aware sigma, outlier rejection and --search-min-zeta. The CLI's own comments tie the captured-fraction default to a crystal recovering its true space group instead of P1. Put the policy in one place (RugnuxDefaults) and have both front ends start from it. The CLI now takes its defaults from there and applies user options on top; its output is unchanged, verified bit-for-bit on four battery crystals. The stored cell/group is still available: the job dialog offers "Use the stored unit cell / space group", off by default, shown only when the file has one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 18:07:14 +02:00
leonarski_fandClaude Opus 5	504393d10e	rugnux: report bad input instead of aborting on it ... main had no enclosing try/catch, but plenty of ordinary input reaches a setter that throws: --polarization 2, --detector-distance 0, -q 0, --azim-max-q 20, --scale combined with a reference MTZ, and every failure inside the pipeline itself. All of them ended as "terminate called after throwing an instance of 'JFJochException'" and exit 134, with the message nowhere to be seen. Move the body into RunRugnux and let main report what was thrown, exit 1. Two options also still bypassed the numeric parser that exists to prevent this: --scaling-high-resolution used atof, which turns a typo into 0 and then throws from the setter, and --integration-radius used std::stof, which throws on non-numeric input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:16:19 +02:00
leonarski_fandClaude Opus 5	0f1851cf82	rugnux: drop the 1.5 A spot-finding limit on rotation data ... Rotation kept a 1.5 A high-resolution limit for spot finding on the strength of one indexing-rate measurement (100.0 -> 96.8% on a strong set). Measured properly, over the whole 33-crystal rotation battery, it does not earn its place: * no space-group decision changes - the same 30/33 agree with XDS, and the three that differ are the known pre-existing cases, unchanged; * 29 of 33 crystals are identical to the digit - same indexing rate, R_meas, CC1/2, ISa. The limit was doing nothing on the large majority; * where it does bite, the limit is the worse setting. The one crystal that loses appreciable indexing rate without it (99.50 -> 94.22%) comes back with lower R_meas (29.4 -> 28.1), higher high-resolution CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses 0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames indexed, better data from them; * runtime is unchanged (16m46s vs 17m32s over the battery). So the indexing-rate cost is real but does not carry through to the merged data, which is what the limit was protecting. Unset now means "as far as the detector reaches" for rotation as well as stills; --spot-high-resolution still sets a limit for weak, high-background data where the extra high-resolution spots are genuinely noise. This also removes the flag that distinguished "the user asked for no limit" from "the user said nothing" - with no rotation default left, both mean the same thing. While rewriting the comment block, corrects its neighbouring claim that rotation keeps the fixed-threshold finder; adaptive detection has been the default for both workflows since 6f4917dce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 10:39:23 +02:00
leonarski_fandClaude Opus 5	59977ae910	rugnux: --spot-high-resolution 0 really means no limit on rotation data ... Passing 0 reset the limit to "unset" and logged "No high resolution limit for spot finding: as far as the detector reaches" - and then, 700 lines later, the rotation default put 1.5 A back, because unset carried two different requests: the user said nothing, or the user asked for none. The log said one thing and detection did another, and there was no way to lift the limit on rotation data at all. Remember whether the option was given, and apply the rotation default only when it was not. Documented in the usage message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:35:36 +02:00
leonarski_fandClaude Opus 5	40acf1ffc4	rugnux: search the space group twice by default on rotation data ... --search-min-zeta now defaults to 0.85 for rotation, so the de-novo search runs on a merge of all the observations AND on a merge of only the well-measured ones, and keeps whichever found more symmetry. Previously it shipped off and the second opinion had to be asked for. Rotation battery, 33 crystals, NO flags beyond the resolution limit: fixed-threshold finder 30/33 - zero crystals differ from the single search adaptive finder 30/33 - the same three mismatches Both arms now agree crystal for crystal, which they have not done before. The last disagreement was a thaumatin set whose 4-fold sits 88.9 deg from the spindle: at defaults it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken) where it read P222. The classic arm is a strict no-op - zero differences against both the explicitly-flagged run and the run predating the dual search - so the default costs nothing where the geometry is not the problem, and 47.2 s against 47.8 s on the same crystal back to back. The default is safe to set because the two searches can only disagree by a LOST operator: discarding observations starves an operator correlation, it cannot invent one. That also makes the 0.85 itself uncritical - too aggressive a cut only means the second opinion contributes nothing and the full merge wins. --search-min-zeta 0 restores the single search. Docs: CHANGELOG gains a 1.0.0-rc.161 section covering the branch, and CPU_DATA_ANALYSIS records the four analysis changes of this work - the confidence-weighted per-image refinement, the collapsed per-frame scale guard, the opt-in per-image rejection, and the operator-disagreement criterion with the two-search rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:56:36 +02:00
leonarski_fandClaude Opus 5	f2b92e3f4d	rugnux: --search-min-zeta drops badly-measured observations from the symmetry search ... zeta is the sine of the angle between a reflection's rocking path and the spindle. Near 0 the reflection crosses the Ewald sphere almost tangentially, spends many frames in diffracting position and is measured worst. The de-novo space-group search asks how EQUAL an operator's paired intensities are, so its answer is dominated by whichever reflections are measured worst - and when the spindle lies in a lattice plane, an operator that permutes the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. That is not a fair comparison, and it can make a real symmetry operator look like a twin law. Measured on a thaumatin set mounted that way (its 4-fold is 88.9 deg from the spindle), the added operators' disagreement is 1.74x the parent's over pairs where both reflections have zeta < 0.85 and 1.003x - i.e. the symmetry is exact - over pairs where both are above it. The search consequently refuses the 422 promotion and merges the crystal in P222, while the same data forced to the right group give CC1/2 99.2% at multiplicity 10.7, matching XDS. With the option the de-novo pass ignores those observations (the final merge keeps everything - there completeness is the point): zeta cut observations ignored H ratio adopted 0 (off) - 1.47 P222 0.5 1620648 1.44 P222 0.7 3006013 1.34 P21212 0.85 4536724 promoted P4212 (correct point group) OFF BY DEFAULT, and it must stay off, because the same cut costs four other crystals their space group (P41212 -> P212121, I23 -> P2, I23 -> I222 twice): at 0.85 it discards 40-80% of all observations, which on a crystal whose geometry is not the problem simply starves the search. Two independent implementations - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade exactly the same crystals, so this is a property of the cut and not of where it is applied. Verified bit-identical to the previous binary when off. The companion diagnostic is already there: the run now reports how close a symmetry axis lies to the spindle, which is the geometry that makes this option worth reaching for. Implementation note for anyone tempted by the cheaper route: excluding these observations from the ASU grouping alone does NOT work. The 3D combine selects partials on corr, not on their group, so their intensity still reaches the fulls and the merged intensities are unchanged - measured, the statistic did not move by 0.03 while 67% of observations were nominally excluded. Zeroing corr is what removes an observation from the combine, the merge and the error model alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 20:03:43 +02:00
leonarski_fandClaude Opus 5	6f4917dcee	rugnux: adaptive spot detection is the default for rotation data too ... It was held back because a 33-crystal rotation battery showed it breaking three crystals deterministically - a lost space group, a halved indexing rate and a collapsed merge. None of those causes turned out to be in detection. The extra spots adaptive finds are real. Measured per spot against a finder-neutral local background: 64% recur at the same position on the adjacent frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a conventional local SNR >= 4 test, median local SNR 34. What they include is genuine peaks belonging to no lattice the indexer found, and the damage they did scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed finder), not with their quality - which is why nothing aimed at judging individual spots ever worked. The three failures fell to fixes elsewhere: merge collapsed - a per-frame scale free to collapse toward zero amplified two junk frames by 546x (704098712, ec7a82613). Not a detection problem at all: the fixed-threshold finder trips the same bug on a different frame range. space group lost - the per-image geometry refinement was dragged 2.3-2.8 deg off by the weak-spot tail in an unweighted fit; weighting each spot by how strong it is FOR ITS RESOLUTION fixed it (ae126c3d5), and gained a point group for the fixed finder too. indexing halved - gone with the same two; that crystal is now better under adaptive (CC1/2 92.6 -> 95.9, high-shell 44.9 -> 56.4). Battery, 33 crystals, adaptive vs the fixed finder: exact space group matching XDS 26/33 vs 25/33 point group matching XDS 29/33 vs 30/33 ISa better on 6 crystals recovers a screw axis the other misses (P321 -> P3121) The one point group it loses is a tetragonal crystal where adaptive collects 2.4x the observations at better R_meas (21.4% vs 28.3%) and better ISa (4.58 vs 2.91), and forced to the right group gives CC1/2 99.2% at multiplicity 10.7 - matching XDS. Only the automatic symmetry call fails there, and six candidate causes have been measured and refuted (mixed indexing hands, off-crystal frames, a badly integrated minority, radiation damage, pseudo-tetragonality, uncorrected anisotropy). It is left as the subgroup, which is the recoverable direction: -S gives XDS-quality data from the same run, whereas the failures this unblocks were not recoverable. --no-adaptive-spots reverts to the fixed-threshold finder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:59:37 +02:00
leonarski_fandClaude Opus 5	4895dc1018	Rotation: let --min-image-cc drop frames that disagree with the merged reference ... The flag was accepted on rotation data and did nothing - it is read only by the stills merge (Merge.cpp), and the CLI warned about that rather than fixing it. Meanwhile RotationScaleMerge already COMPUTES a per-frame correlation against the merged reference and writes it to the per-image table; nothing acted on it. Wire the two together. A rejected frame has its partials' corr set to 0, which is how a frame already leaves the pipeline - every consumer requires corr > 0, so the combine, the merge and the error model all drop it together. The GPU path reuses the SmoothCorr kernel with a ratio of 0, so one implementation covers both. Off by default (0), and verified bit-identical to the previous binary when off. What it catches, on the two rotation datasets that have a population to catch: a two-lattice crystal - two lattices in two physical AREAS of the sample, so the sweep passes from one to the other and whole blocks of frames measure a different crystal from the one being merged (frames 500-700 index perfectly well at a per-frame CC of 0.22 against 0.47-0.56 either side, in 11 contiguous runs). R_meas 28.6 -> 24.6%, CC1/2 93.6 -> 95.1, high-shell CC 23.4 -> 38.3. a second dataset with 9.5% of frames below CC 0.30: R_meas 24.3 -> 23.4%, CC1/2 92.6 -> 93.4. The criterion is "this frame disagrees with the merged reference", NOT "this frame is off-crystal". It happens to catch both, because a frame that measures nothing and a frame that measures a DIFFERENT crystal fail the same test, and it does not need to know which. For the two-area case that is a workaround, not a treatment: it recovers one crystal by discarding the other, where processing the two as separate sweeps would keep both. The frame-block structure is clean enough that such a split could be detected automatically. WHY THERE IS NO DEFAULT. The per-frame CC is not comparable between datasets - it is as much a measure of data quality as of frame validity. Measured medians across the battery run from 0.30 to 0.81, so one absolute bound removes 13 frames from one dataset and 584 of 1800 from another: battery at --min-image-cc 30, 33 crystals: no point group changed (30/33), four crystals clearly better (one +5.4 CC1/2 points, the two-lattice case above, and ISa gains of 1.3-4.6 on three others) - and one healthy crystal lost a third of its frames and with them its high-resolution shell (CC1/2_hi 26.2 -> 2.0). This is the same trap as an absolute bound on any per-operator or per-frame agreement statistic, and the same one the per-frame scale guard avoids by measuring against the run's own median. A principled version would cut on the SHAPE of the per-frame CC distribution - a dataset with a bad subpopulation is bimodal, a uniformly weak one is not - rather than on an absolute value. Until that exists this stays opt-in, and the per-image CC it keys on is already in the _image.dat table for anyone choosing a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:45:49 +02:00
leonarski_fandClaude Opus 5	c9b52857e0	rugnux: warn when --min-image-cc is ignored; drop a dead robust estimator ... --min-image-cc is consumed only by the stills merge (MergeOnTheFly); RotationScaleMerge never reads it. On rotation data it was accepted and then silently did nothing, so a run that looked filtered was not. It now says so. FitProfileRadius_MAD had zero callers - a robust twin sitting uncalled next to the non-robust estimator that is actually used is a trap, so it goes. Neither changes any result: verified on a rotation dataset (indexing rate, cell, space group and merge statistics identical, warning emitted). Context for anyone tempted to wire that estimator in: I tested exactly that today and it is NOT justified. The population it would clip is truncated by construction - a spot is only marked `indexed` when its fractional-Miller norm is inside the indexing tolerance - and is measurably shorter-tailed than Gaussian (kurtosis 2.85). Across four serial-stills datasets a MAD-clipped variant only narrowed the prediction window (-17% integrated reflections everywhere), which was neutral on strong data and destroyed real signal on weak data (one set lost completeness 96.0 -> 93.9%), with R-free 0.3753 -> 0.3767. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 10:46:52 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8	a8c1006c49	Choose min-pix-per-spot adaptively per image for serial-stills indexing ... For stills indexing the minimum-pixels-per-spot filter is now chosen per image instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total) is kept, then integrated once at that min-pix. The fraction factor keeps a smaller min-pix's extra spots only when the lattice actually explains them, so strong frames retain their real weak spots (extending resolution) while noise-flooded frames stay strict. The mode is selected by the presence of --min-pix-per-spot, now optional (SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the adaptive per-image path, give a value to force a fixed min-pix. It applies only to the stills indexing path -- rotation indexing builds one global lattice and keeps a fixed min-pix, and the online receiver and the FPGA host path always carry a concrete value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame indexed, to drive the per-image selection. Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and adaptive-min-pix checkboxes, each greying out the control it overrides); the broker uses neither. Validated on the full rotation regression battery (no regression) and the whole serial-stills target battery at full image count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 18:31:35 +02:00
leonarski_fandClaude Opus 4.8	014e43a4c9	Remove non-helping stills merge/scaling knobs ... Trims three opt-in stills parameters that did not improve data quality on the external-reference (PDB R-free) battery and only added code: - --partiality-uncertainty: the (1-p)/p merge-sigma term was null on all four serial-stills datasets of the battery vs their reference structures (and neutral-to-harmful at higher coefficients); removed the flag, setting and CorrectedSigma term. - --stills-modulation: the detector-plane flat-field surface was net-negative on flooded data; removed the flag, setting and MergeOnTheFly::RefineModulation (the rotation modulation in RotationScaleMerge is unaffected). - --min-indexed-fraction: every value other than the 0.20 default collapsed CC1/2; removed the override flag/setter, keeping the fixed 0.20 acceptance floor. Default behaviour is unchanged (all three were off / at their default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:40:48 +02:00
leonarski_fandClaude Opus 4.8	20bbcb1cd3	Remove --soft-weight and --local-snr spot-finder options ... Both were opt-in adaptive-spot refinements that did not help. Soft per-spot weighting was index-rate neutral across the battery (re-ranking only bites when spots exceed the max-spot cap, which weak serial data does not reach). The local-SNR gate was neutral on index rate and degraded merged CC1/2 on flooded XFEL data. Drops the flags, ApplyWeights/FilterByLocalSNR, the per-spot weight field, and the by-weight FilterSpotsByCount branch (now strongest-first only). --adaptive-spots itself is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:31:35 +02:00
leonarski_fandClaude Opus 4.8	f72b4484e2	Remove dead per-crystal deltaCChalf rejection ... Drops the --reject-delta-cchalf flag and MergeOnTheFly::DeltaCChalfReject. The CLI value was parsed but never consumed (the method had no call site), so the flag was already a no-op. Wiring it up and testing against an external reference structure showed it is confirmation bias: on a spurious-crystal flood it raised internal CC1/2 while CCref (correlation to the true structure) fell, and it never improved R_meas. The merge weights are already correct; per-crystal merge-side rejection has no genuine lever here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 15:05:06 +02:00
leonarski_fandClaude Opus 4.8	ecf79af018	Remove threshold-free persistence spot-detection variant ... Drops --persistence-spots and AdaptiveSpotFinderCPU::RunPersistence (the 0-D topological-persistence detector added in 5a33b0743). It was a research variant that never beat the hard-threshold adaptive detector on a CC1/2 basis and is a GPU dead-end (global candidate sort + union-find), so it is not a production path. The hard-threshold --adaptive-spots detector is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 12:41:44 +02:00
leonarski_fandClaude Opus 4.8	ca7cbe206a	Add opt-in local-SNR spot gate and acceptance-fraction knob (serial stills) ... Two opt-in tools for weak serial-stills tuning; both default-off, so the default pipeline is bit-identical (verified: a serial-stills reference run reproduces HEAD's 7.85% indexing rate exactly). --local-snr <sigma> (AdaptiveSpotFinderCPU::FilterByLocalSNR): after the loose per-ring adaptive threshold builds connected-component spots, drop any spot that does not stand this many sigmas above its OWN LOCAL background (robust median/MAD of a square annulus), not just the azimuthal ring mean. On structured-background (XFEL) frames the ring mean underestimates the local diffuse level in some sectors, so the ring threshold floods; a real Bragg peak still stands many local sigmas proud. Validated on XFEL stills to separate real peaks from flood at the pixel level (real median local-SNR ~70 vs flood ~2.6; SNR>=5 keeps ~99.8% of real peaks, ~14% of flood). GPU-portable (a per-spot local reduction). NOTE: on the current serial-stills battery it is index-rate/CC1/2 neutral -- the flood that survives as CC clusters overlaps weak-real spots, and only lattice-fit separates those -- but it is the correct tool for genuinely floody data (ice/jet/loosened detector) and the right substrate for the online FPGA path. --min-indexed-fraction <f>: exposes the previously hardcoded 0.20 minimum indexed-spot fraction (AnalyzeIndexing) as a per-run setting. Lowering it admits weaker/sparser crystals; on flooded XFEL data the extra lattices are spurious (pair with --min-image-cc to gate them), on clean synchrotron data there are no marginal frames so it is a no-op -- useful as a gating-experiment primitive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-24 11:12:22 +02:00