213 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	7bbf072ad2	Scaling: do not report an ISa that was never measured ... The error model's systematic term b is identified only by the spread of I^2/sigma^2 across the intensity bins the fit uses, and those bins hold equal COUNTS. So when fewer reflections are strong than one bin holds - a sixteenth of the pool - the top bin's median sits at an intensity where b cannot be measured at all, and the fit hands it the bins' own noise-selection slope instead: sorting noise by its group mean squared makes dev2 rise with I2 even when the true b is zero, and with no strong bin to out-vote it that slope becomes b. The result is not a small error. On the battery's weakest crystal, 2.2% of whose fulls reach I/sigma 2, the fit returns b = 5.6 - sigma -> 2*I at the strong end - and since corrected_sigma applies b at the GROUP MEAN, sigma^2 = a*sigma^2 + (b*mean)^2 is a per-group constant that caps merged |I/sigma| at sqrt(n)/b. The cap lands at 2.3, so 98.8% of merged reflections come out below 3 and the reported ISa is 0.50, on data whose CC1/2 is 99.3% at multiplicity 18.7. XDS fits 6.13 from the same images. Feeding XDS's own scaled observations through this estimator returns 0.84, so it is the estimator and not the data; synthetic data built with b = 0 and 1.8% strong reproduces a = 0.51 and ISa 0.50 to two digits, and recovers the truth as soon as the strong fraction passes one bin. So refuse to report what was not measured: when the strongest bin's own (I/sigma)^2 is below 4, fit a alone, hold b at zero and warn that ISa is unmeasured. The threshold is not delicate - the two crystals it fires on sit at 0.22 and 0.84 while the next crystal in the battery is at 31.7 and a healthy one at 342, so anything from 4 to 25 selects the same two. Full 38-crystal rotation battery: it fires on those two crystals and no others, and space groups are unchanged at 35/38. Dropping the spurious term also fixes the merge weights it had been distorting - on the worse of the two, R_meas 19.2 -> 13.6%, low-resolution R_meas 13.8 -> 6.9% against XDS's 14.1%, CC1/2 98.7 -> 100.0%, with chi2 1.11 on the one-parameter model. Two further crystals move slightly; the guard never fires on either, and they are marginal crystals of the kind whose two-pass branch any recompilation can shift. This reports the parameter as unmeasured rather than clamping it to something plausible, because the honest statement is that the data do not reach far enough for a systematic error to be seen - not that there is none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 06:14:46 +02:00
leonarski_fandClaude Opus 5	b40abe31cf	Scaling: one exact-Bragg angle per rocking event ... Every partial's delta_phi was solved from its OWN frame's lattice - by the predictor, and again by SmoothGeometry. Per-frame geometry is re-refined against that frame's spots alone, so what is left of its jitter entered each frame of an event independently and the frames of one rocking event stopped sitting exactly one oscillation apart on the curve. Their partialities then no longer tile it, and because a broad rocking curve spans more frames, the error grows as 1/zeta - which is how it has been showing up: a zeta-graded systematic that nothing in the integrator could reach. For the frames of one event the geometry is exact. Each frame has already turned one oscillation further, so delta_phi is linear in frame number with slope minus the increment; the sign is checked against the data rather than derived, the measured mean frame-to-frame slope being -0.19996 deg/frame at an increment of 0.20000. Fit the one free number, the offset, over the event and lay its partials back on that line. The rms departure removed is 0.25 deg - larger than the oscillation itself, because a small orientation wobble is amplified by 1/zeta. The raw-hkl runs the merge already builds give the grouping, so this costs one pass over the partials and no extra sort. Full 38-crystal rotation battery against the same binary without it, on unchanged data (observations +0.20%, unique reflections +0.03%, so none of this is selection): R_meas_lo better 22 / worse 5, summed -41.0 pp; excess against XDS -46.9 -> -87.9 R_meas better 22 / worse 5, summed -24.7 CC1/2 better 17 / worse 2, summed +36.8 ISa better 15 / worse 22, summed +6.87; shortfall against XDS 28.1 -> 21.2 space groups unchanged at 35/38 The low-resolution R_meas gains land on the crystals that have carried this gap: 19.3 -> 10.8, 20.7 -> 13.8 (now past XDS), 25.6 -> 19.7, 17.5 -> 12.4 per cent. Exactly one crystal shows any change in the two-pass branch fingerprint, so unlike most changes on this path the result is not confounded by that bistability. ISa falls on more crystals than it rises, and that is the estimator becoming honest rather than the data getting worse: every crystal whose ISa dropped materially was over-optimistic against its own R_meas_lo and moved toward consistency, and the median ratio of reported ISa to the value its own R_meas_lo implies goes 1.09 -> 1.01, against 1.19 for XDS. The one real loss is a crystal going 1.11 -> 0.95 on that ratio. High-shell CC1/2 is worse on 22 crystals, by about 1.2 points each. It is the one metric that dissents, and it is also the one that has failed as an arbiter repeatedly on this data, while overall CC1/2, R_meas, R_meas_lo and reflection count all improve on an unchanged number of observations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 05:25:47 +02:00
leonarski_fandClaude Opus 5	6f7b136ec2	Bragg integration: a shared signal pixel belongs to the nearer reflection ... Nothing kept a neighbour's flux out of a reflection's own signal disk. The union mask keeps neighbour cores out of the BACKGROUND ring, but the r1 disk was read whole, so on a dense pattern a crowded reflection measures part of its neighbour as its own. Ownership is decided once per image into a per-pixel (quantised distance, reflection) key written with an atomic minimum, so the nearest predicted centre wins whatever order the writes arrive in and the lowest index breaks a tie. `--overlap exclude`, now the default, drops the pixels a nearer neighbour owns from the profile fit. A profile fit is the amplitude of a normalised profile, so leaving pixels out renormalises the estimator by construction and the reflection stays unbiased rather than being discarded; the summation-fallback guard is scaled back to the disk the box-sum seed actually read, so it still compares like with like. `--overlap reject` is the XDS MINPK alternative - drop the reflection when less than `--overlap-minpk` of its expected profile is cleanly its own. A box sum has no profile to renormalise with, so `exclude` is a no-op there and only `reject` acts on it. Widening the split - keeping a pixel only where no other centre is within its distance PLUS a margin - was built and measured, and it is worse monotonically: the residual bias of the pixels that were kept grows from +0.072 to +0.209 in ln intensity at 0 to 3 px of margin. What the margin removes is the reflection's own profile, not the neighbour's tail, so the plain nearest-centre split is the rule. Measured on the full 38-crystal rotation battery against the same binary with the treatment off: ISa better 15 / worse 8, summed shortfall against XDS 39.7 -> 28.1. Three of the losses are the two-pass loop taking its other branch - their median mosaicity moves between the two known attractors - rather than the change under test; excluding those it is better 15 / worse 5 and the shortfall goes 31.3 -> 14.4. The two crowded crystals gain 38% and 52% of their ISa, one of them passing XDS. High-shell CC1/2 over the 35 crystals that neither flipped branch nor carry a collapsed error model is better 7 / worse 7. Space groups unchanged at 35/38. The owner map is built only when a treatment is asked for and costs 1.1% of the battery's wall clock - 23% on a genuinely crowded crystal, nothing where no two predictions touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 05:02:31 +02:00
leonarski_fandClaude Opus 5	06a5a118bf	Bragg integration: widen the background ring to r3 = 13 ... The background is estimated from the r2..r3 ring and then subtracted from every pixel of the r1 disk, so the ring mean's own error enters the intensity n_inner times over: var(I) carries n_inner^2 * bkg / n_B. That term is first-order in sigma, and it is set by how many pixels the ring holds - not by anything about the reflection. At r3 = 10 the ring holds about 200 px against the disk's 50. Widening it to 13 roughly doubles that. The signal disk is untouched, and the pixels gained lie further from the reflection rather than nearer, so nothing is traded for them. The effect is not subtle once looked for. Matched observation by observation on one crystal, halving the ring's pixel count leaves the intensity alone and inflates sigma by 4.7%, and the inflation rank-orders with the ring collapse across the battery. Over the whole rotation battery, against the same binary at r3 = 10: ISa better on 14 crystals and worse on 4, the summed shortfall against the reference 164.7 -> 155.9, the summed low-resolution R_meas excess 69.7 -> 59.1 percentage points, and one more crystal reaching the reference space group (33/37 -> 34/37, a trigonal case that was over-promoting). Largest gains where the ring was starved worst; the four losses are 0.25 to 2.16 in ISa and none of them changes a space group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 00:07:30 +02:00
leonarski_fandClaude Opus 5	27d0b1db74	docs: state the reflection-file conventions ... The mmCIF carries eleven items rugnux invents, and the rule they follow - a jfjoch_ prefix inside whichever standard category the quantity belongs to - was nowhere written down, so the only way to learn what was in a merged .cif was to read WriteReflections.cpp. Tabulate them, with the values a reader needs in order to interpret each one (the untwinned and perfect-twin values for the L test and the second moment, the sign convention for the radiation-damage B). Two of them need more than a name. The compatibility note records that jfjoch_diffrn_ISa changed meaning and that a file carries no marker saying which. And the HKLF-4 .hkl has two properties that are invisible in the file and change what a comparison means: Bijvoet mates are separate records, and the intensities carry a single global rescale so the largest fits F8.2 - harmless to SHELXC and ANODE, which use ratios, but not something to compare magnitudes across. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 19:08:58 +02:00
leonarski_fandClaude Opus 5	adf87e8675	Merging: export the XDS-comparable ISa under jfjoch_diffrn_ISa ... The mmCIF's _reflns.jfjoch_diffrn_ISa carried the strong-reflection asymptote, a tier XDS has no equivalent of, while the name invites comparison with XDS's ISa - which is the whole-range 1/sqrt(a*b). rugnux_vs_xds.py reads that item for the battery's ISa column, so the comparison that column exists to make was between two different quantities, flattering rugnux by the difference between the tiers. Write the whole-range value there, move the asymptote to _reflns.jfjoch_diffrn_ISa_asymptotic, and add _reflns.jfjoch_error_model_a and _b in XDS's convention so the number can be re-derived from the file rather than taken on trust. On a broadband rotation dataset the battery column now reads 13.25 against XDS's 21.18 where it read 15.6 before, and the two error models can be compared term by term for the first time: a 1.538 vs 1.249 and b 3.71e-03 vs 1.78e-03, so the gap is in BOTH the counting and the systematic term (1.23x and 2.08x, and sqrt(1.23*2.08) = 1.60 = 21.18/13.25). This is a deliberate redefinition of an exported item, not an addition: a file written by an earlier version carries the asymptote under the old name and there is no version marker to tell them apart. Noted in the changelog and in docs/CPU_DATA_ANALYSIS.md. Nothing reads the item back into the pipeline - it is written and never parsed by rugnux itself - so no stored file is reinterpreted in a way that changes a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:55:46 +02:00
leonarski_fandClaude Opus 5	ac06b5c64f	Merging: report the error model in XDS's convention ... rugnux fits sigma^2 = a*sigma0^2 + (b*<I>)^2, so its `b` is a fraction of the intensity. XDS fits sigma^2 = a*(sigma0^2 + b*I^2) and prints ISa = 1/sqrt(a*b). The two `a` are the same number, but the two `b` are not - b_xds = b^2/a - so the pair rugnux printed could not be read against a CORRECT.LP, which is the only reason anyone looks at it. Convert at the report. The fit, the merge weights and both engines' variance expressions are untouched, so this is a re-expression and not a change: on a rotation dataset the merged intensities move strictly less between before and after than they do between two runs of the SAME binary (99.9% identical, max |dI/I| 9.1e-4 against the run-to-run control's 7.5e-3), with the same reflection set. The rotation path also printed the wrong ISa for the comparison it invites. What it calls ISa is the strong-reflection asymptote, a tier XDS has no equivalent of and which can only ever be the more optimistic of the two; XDS's ISa is the whole-range 1/sqrt(a*b), which in rugnux units is exactly 1/b. Print both, labelled. On a broadband rotation dataset that is 13.2 (whole range) and 15.6 (asymptote) against XDS's 21.18 - so the number previously compared was flattering rugnux by 2.4. A third, unrelated `b` lives in the space-group search: fitted with the sigma^2 coefficient held at 1, with gate constants calibrated in that convention, and a ratio bound does not survive the mapping (1.90 would have to become 3.61) while the absolute floor has no correct value at all, there being no `a`. It is now commented as such, since making the three consistent is the obvious wrong move. Also corrects three comments and two doc passages that still described a merged-sigma systematic floor deleted in 72efb75a8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:50:22 +02:00
leonarski_fandClaude Opus 5	61d24db59f	Bragg integration: elongate the background ring per reflection ... The signal disk and the r2..r3 background ring were fixed pixel circles, identical for every reflection at every resolution. A reflection is not round: a finite bandwidth streaks it radially by bw_sigma*Rpx, so at high resolution the ring sits within 1.3-2.2 sigma of the reflection's own profile and measures its tails as background. --integration-stencil <k> makes the RING an ellipse, elongated along the beam->reflection direction by k times that streak, capped at 2*r3. The tangential half-widths stay r2 and r3, and the r1 signal disk stays a circle: r1 drives the all-or-nothing n_inner_valid == n_inner gate, so growing it rejects any reflection carrying one bad pixel along a long streak, and the flux a circular r1 loses is a function of resolution alone, which the per-shell scale absorbs. The geometry lives in one shared header compiled by both the host compiler and nvcc, so the seven pixel-classification sites - the CPU mask/main/clip loops and the GPU mark_mask/main/trim/clip kernels - cannot drift apart. Rather than evaluate an ellipse, each pixel's squared distance has its radial part scaled down, d2 - q*rad^2 against r2^2/r3^2 with q = 1 - (r/(r+grow))^2, so grow = 0 gives q = 0 and both tests collapse onto d2 exactly in floating point. The width is the bandwidth streak alone, not the profile's full radial variance, which also carries the sensor parallax and weak-spot capture terms. Deriving the growth from those was implemented first and measured on the rotation battery: at k=1 it took Thau_9's high-shell CC1/2 from 75.8 to 27.9 and Benas_3's from 14.1 to 6.0, against cytC_10 +1.2 and lyso_ref flat. On a monochromatic beam they are the only terms there are, and C_CAPTURE is 64% of them. Keeping only the streak also makes the option exactly inert without a bandwidth, rather than merely small. Default 0. Measured on broadband rotation data with the bandwidth set to its spectroscopic value, matched resolution limits: high-shell CC1/2 30.6 -> 46.4 at k=4, and better in EVERY shell in both CC1/2 and R_meas (top shell R_meas 194.7% -> 138.7%), with completeness, multiplicity and space group unchanged and 28 of 98833 unique reflections lost. Anomalous peak height over 18 sites +0.107 +- 0.039 sigma (p = 0.013). The full 38-crystal rotation battery is unchanged to every reported digit, base against k=3. Two consequences of an elongated ring are handled rather than inherited. The neighbour exclusion marks the inner ELLIPSE in each neighbour's own frame, or an elongated neighbour leaks its tails into this reflection's ring. And the radial-background curvature kernel becomes a small table indexed by the growth, because its azimuthal average makes one kernel serve every reflection only while their stencils are identical; the GPU's radial window, previously a fixed 32 bins, is now sized on the host from the widest ring on the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 15:19:28 +02:00
leonarski_fandClaude Opus 5	52ea727650	Reader: reconstruct the background variance a legacy file does not store ... A _process.h5 written before background_variance existed was read with var_bkg = 0, on the reasoning that zero leaves the combine with the signal term alone, "which is what it had before". It does not. Before, the combine back-derived the non-signal variance from sigma itself, and on a weak reflection that is essentially the whole of sigma^2; zero deletes the dominant term and weights the reflection by roughly 1/I instead of 1/sigma^2. Recover it from the integrator's own identity, sigma^2 = I + var_bkg, when the dataset is absent. Measured by re-scaling a stills _process.h5 with the dataset deleted, against the same file with it intact: the automatic resolution cutoff was reading 1.66 A where the intact file reads 1.81, with 14731 unique reflections against 11508 - i.e. the zeroed file looked good enough to merge 0.15 A past its own limit. Reconstructed, it reads 1.78 A and 11926, within 0.03 A of the intact file. The residue is the profile-fit path, where var_bkg is not exactly sigma^2 - I and only the box-sum identity is exact; that is recoverable to a closer approximation only by storing it, which is what files written from now on do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 05:52:56 +02:00
leonarski_fandClaude Opus 5	a7c5d7a89b	CBOR: carry the reflection's background variance ... var_bkg was added to Reflection and to the HDF5 writer but never to the CBOR reflection map, so it survived only where rugnux drives the writer in-process. Everything that reaches the writer over the wire - i.e. every acquisition the broker records - wrote /entry/reflections/*/background_variance as an array of zeros, presented as a measured quantity, and re-processing such a file fed the merge a non-signal variance of zero. Encode and decode it. The key is optional on both sides, so a stream from an older version still reads and one from this version still reads on an older client. The round-trip test only checked h, k, l, the predicted position and d - which is why a missing float was invisible. It now gives every field a distinct value and checks all of them, so the next field added to Reflection and forgotten here fails immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 05:52:44 +02:00
leonarski_fandClaude Opus 5	bc1c4c6800	Rotation: land the rest of the bandwidth term ... f4e281b2f described this change in full but committed only one of its six files. What went in was RotationScaleMerge.cpp - the merge widening the partiality it recomputes from the smoothed mosaicity. That is precisely the part which is unsafe on its own, by the original message's own argument: without the mosaicity fit subtracting the term before fitting, the bandwidth is counted twice, and without the predictor widening its acceptance window, the partiality the merge recomputes no longer matches the one integration measured. Add the five files that were left behind: the rotation predictor and its GPU twin widen the acceptance window and the partiality handed to integration, the settings struct carries the term, and CalcMosaicityXDS deconvolves it before fitting so what it returns is the intrinsic mosaicity rather than the mosaicity plus the beam. Monochromatic data is untouched by construction - every hunk is guarded on a non-zero bandwidth, which is read from incident_wavelength_spread or --bandwidth and is absent from every dataset in the rotation battery. Verified on the one dataset that has a bandwidth: at --bandwidth 0, the merge table is identical to the branch tip; with the bandwidth set, the fitted mosaicity drops 0.0718 -> 0.0694 deg as the deconvolution takes effect and CC1/2 in the outermost shell recovers 30.3 -> 31.4%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 22:26:26 +02:00
leonarski_fandClaude Opus 5	1239c49731	Bragg integration: separate the three things a bandwidth used to switch ... Setting a bandwidth flipped three unrelated switches at once: it changed the profile's radial capture term, it moved the width measurement from the signal disk to the whole fit grid, and it silently overrode the background clip and trim, so --background-clip under --bandwidth was ignored - the two runs were bit-identical. The width measurement was the damaging one. The fit grid is an azimuthally averaged stack, so its second moment is sigma_r^2 + sigma_t^2 and the radial smear of a bandwidth leaked into the tangential model - a tangential width of 3.04 px against a 1.06 px truth, inflating the effective background pixel count where the weak signal is. The result was a step rather than a slope: on genuinely monochromatic data, declaring a 0.2% bandwidth cost ISa 28.4 -> 22.2. Measure the two widths separately, accumulated in each spot's own radial/tangential frame over the signal disk, from the signed profile cells - away from the peak a learned cell is background noise centred on zero, so the signed sum is unbiased, while clamping it at zero turns that noise into a pedestal the r^2 weight reads as width. The radial term is then the measured excess or the analytic floor, whichever is larger. With the two widths separated there is nothing left for the broadband switch to select, so it is gone - which is the proof the three were independent. The background clip and trim now come from the settings in every case; the tuned 3-sigma broadband default moves to the rugnux front end, which is the only place that knows whether the user gave a value. Monochromatic data: declaring a 0.2% bandwidth now costs ISa 28.4 -> 27.9 rather than 22.2, and forcing the old 3-sigma clip in the new build reproduces the good result, so none of the step came from the clip. On large-bandwidth data CC1/2 improves in 8 of 10 shells. Across 12 monochromatic crystals the space groups are unchanged and CC1/2 moves by at most 0.2 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 21:08:29 +02:00
leonarski_fandClaude Opus 5	a29c36600f	Beam-stop shadow detection, and a low-resolution limit for scaling ... rugnux finds the beam stop and its holder in a projection of 60 images and marks them in the pixel mask as bit 9 (--detect-beam-stop[=N|off], on by default). Reflections behind the stop are attenuated but not flagged, so they integrate low with a plausible sigma and nothing downstream catches them: the signal-box gate requires 100% valid pixels and shadow pixels are valid, the background clip is high-side only, and the |zeta| cut applies only to the space-group search merge. The detection compares each pixel's background against the typical background at the same radius on two channels. An azimuthal one (the ring median) finds the holder arm, which is a minority of its ring; a radial one (the background just outside) finds the disk, which the ring median cannot see because inside a fully blocked ring the median is the shadow itself. Pixels are pooled over a 5x5 box and tested only where the background has actually been counted, so low-background data no longer masks the whole detector. Recorded reflections are carved back out - a beam stop cannot block a reflection that was measured. Bit 9 belongs to the run that found it, not to the dataset: it is cleared when a run starts, so a mask read back from a file that carries one starts clear. The user mask (bit 8) is left alone. Scaling and merging gain a low-resolution limit, default 50 A (--scaling-low-resolution <num>, 0 removes it), applied per observation before scaling so it also protects the per-frame scale fit and the space-group search. 50 A is the value XDS configurations use; rugnux_vs_xds.py now matches both of XDS's resolution limits instead of only the high one, so the lowest shell is the same shell in the two programs. The viewer draws the detected shadow in coral with a "Show beam stop" switch in the side panel, exposes the low-resolution limit in the settings dock, and offers detection in its processing jobs. Adding an image marker meant giving the reader a MIN_REAL_PXL_VALUE, because several places classify a pixel by range rather than by equality and would otherwise read the new marker as a very negative intensity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 01:05:31 +02:00
leonarski_fandClaude Opus 5	df9a9c2a2c	Fix the defects found reviewing the branch before merge ... Image buffer: the per-image CBOR metadata headroom had been re-derived from the online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured worst case - reflections plus the capped spot list plus the three azimuthal arrays - is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and give it a name that both the code and its guard test read: written down twice, the two had drifted and the test kept passing against the value the code had left. Spot finding: an unset low_resolution_limit means no limit at that end, as an unset high_resolution_limit already did. An optional rather than a zero sentinel, because zero is not a natural "no limit" here - every pixel lies above it, so the plain comparison masked the whole image instead of none of it, and nothing validated the zero. The API field is no longer required; a zero is folded into the unset case at the boundary, where older clients still send it, so one spelling reaches the analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9> wraps above 512 A and would have masked everything. image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU file with none, and the path fed by bytes we did not produce. An unchecked synchronise returned the host-written sentinel as if it were a measurement, so the decode looked successful and the fallback to the host decoder never fired. rugnux: --stride no longer writes one past the end of the per-image arrays, whose count floored where the worker loop ceils, and the written process file links the images actually processed rather than the first N - each frame's picture now sits next to its own analysis. Powder calibration: the face-centred calibrants no longer list their systematically absent rings, so the distance fit starts from a reflection that exists rather than an extinct one; the triclinic calibrant covers both signs of h and k instead of a single octant, which is only valid for a diagonal metric. The test asserted the old behaviour - one ring formula for every cubic standard - and is rewritten. CBOR: skip an unknown tagged value in the end block, as the other four blocks already do. One advance lands on the tagged item rather than past it, so an older reader fed a newer end message threw and never finalized its file. Viewer: a settings value the setter rejects no longer escapes as an uncaught throw from a worker slot, and the field offers only what the setter accepts. Space-group search: judge stage B on the same "present" cut stage A already computes. Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged and the centering rescue switched itself off on exactly the weak data it exists for. Where the fixed cut is the smaller of the two they are equal and this is inert: over the 37-crystal rotation battery every crystal reports the identical space group and identical merge statistics, so it is a no-op there and the low-ISa case it targets remains unmeasured. rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped it; that mode also applies the same polarization default as every other mode. Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both spot extractors take their algorithm from, with its citation and its license. The rc.161 change list is brought back to one line per entry, and the user-visible changes that were missing from it added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 18:18:46 +02:00
leonarski_fandClaude Opus 5	5830f78d57	Revert the azimuthal-integration sigma clip ... Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping machinery in AzIntEngine. This is a partial revert of a6be35ccd - the ice-ring-mask removal that commit also carried stays. Sigma clipping remains where it started and where it is needed: inside the adaptive spot finder, at a fixed 3 sigma on raw counts, feeding the detection threshold and the ice score. The option made the workflow harder to reason about than the quantity was worth. It gave azimuthal integration two meanings behind one setting - the bin mean and the background under the peaks - which the azimuthal-integration workflows do not need. It also did not compose with the fused GPU engine, which supplies the profile from its PLAIN pass: on the default rugnux, viewer and receiver path the setting was silently doing nothing (measured, the profile came out identical to the unclipped run to 1e-6 with identical per-bin pixel counts). Making it correct is not a matter of gating that one shortcut - it means separating the workflows (azimuthal integration, MX rotation, MX stills, geometry calibration) and deciding per workflow what the profile is for, which is a larger change than the option earns. The default path is unaffected: over 20 images of a rotation dataset the radial profile, the per-bin pixel counts and the spot counts are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 15:31:19 +02:00
leonarski_fandClaude Opus 5	a8d289e7cf	Powder calibration: write Poni1/Poni2 in pyFAI's frame, not ours ... The same frame mismatch as the rot2/rot3 fix, in the other two fields. Our pixel coordinates are pixel-centred - 948.0 is the CENTRE of pixel 948 - while pyFAI measures from the edge of the sensor and puts the centre of pixel i at (i + 0.5) * pixel size. Poni1/Poni2 went out as beam * pixel size, so anything reading the file placed the pattern half a pixel (37.5 um at 75 um pixels) off ours. The previous commit's "Poni1/Poni2 need no such change" was right about the axis directions and wrong about the origin. The proof was already in the tree. The pyFAI reference values in DiffractionGeometryTest were computed for a .poni with Poni2: 0.150 and a 75 um pixel, which the tests translate to beam_x = 2000 - but pyFAI's numbers are reproduced only at 1999.5. At 2000 every one of them is out by 2.6e-3 nm^-1, which the 1e-2 tolerance hid. The tests now use the beam centre those headers actually mean, and agree with pyFAI to 1e-6 - float precision - across untilted q, azimuth, rot1, rot1+rot2, rot3, rot1+rot2+rot3 and the solid-angle correction. Tolerances drop to 1e-4 (1e-5 for solid angle): ~100x the observed float noise, and 26x tighter than the half pixel they were blind to. The viewer's calibration window printed "PONI x = ... mm" from the un-offset value beside the path of the file it disagreed with; it now matches the file. Also moves the viewer's beam-centre cross half a pixel down and right, where the spot, prediction, top-pixel and saturation markers already are. Our coordinates are pixel-centred and the Qt scene's are pixel-cornered, so the map between them is +0.5, and DrawBeamCenter was the one overlay missing it. The convention itself is now written down in docs/DETECTOR_GEOMETRY.md, with the conversions to XDS ORGX/ORGY and to the edge-of-sensor programs, this being the second bug to come out of it. Only exported and displayed values change; the fitted geometry, spot positions and integration were always self-consistent. A .poni written by an earlier build is half a pixel off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 03:55:42 +02:00
leonarski_fandClaude Opus 5	6194fe6fbf	viewer: calibrate the whole dataset from "Analyze dataset" ... The powder panel could only calibrate the image on screen. Calibration is now a third page beside MX and AzInt, so the dataset button runs it over every image the same way it runs the other two - which is the point, since a powder ring is measured far better by summing a run than by one frame. The page carries the calibrant and the method (rings or spots); the interactive Guess/Refine buttons stay where they were and now share the one calibrant selection, so there is no second combo to drift. analyzeDataset() carries the ProcessMode rather than a bool: a third state was coming, and two bools would have had one combination that cannot be valid. The calibrant list gains ICE, which it could not offer before: the widget worked in unit cells, and hexagonal ice has none that generates its rings correctly (P6_3/mmc would include systematically absent ones). FindCenter now takes the ring list its first line used to derive, so the interactive path gets ice as well. The result window leads with the residual rms rather than the fitted sigma. The sigma is a formal scatter estimate and understates a bad fit badly - measured on ice, 0.215 px reported against a 1.70 px residual - while the rms separates a usable fit from one that has locked onto the wrong thing. A rings run needs the profile binned in azimuth; below four sectors it returns nothing at all. The viewer raises the count to 32 exactly as the CLI does, and says so in the panel and in the job dialog rather than doing it silently. Also fixes a CLI inconsistency this comparison exposed: rugnux's calibration branch never applied the standard offline analysis defaults, so it measured the rings in a profile built with the file's polarization factor while every other mode - and the viewer - uses 0.99. Found because the two disagreed by 0.005 px in PONI x, and confirmed by reproducing the viewer exactly with --polarization 0.99. With it applied the CLI and the viewer write byte-identical .poni files on LaB6 by rings, LaB6 by spots, and an iced dataset over 1800 images. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:45:39 +02:00
leonarski_fandClaude Opus 5	6468dd13be	rugnux: --mode, and detector calibration from powder rings ... --azint-only and --scale are replaced by --mode mx|azint|scale|calibration, with mx the default. The old flags are removed rather than aliased. Calibration mode fits the detector geometry - PONI x/y, the two tilts and the distance - to a calibrant's powder rings and writes a pyFAI .poni alongside a report of how far each parameter moved from the header. Bragg data constrain the beam centre worst, because it is gauge-coupled to the crystal orientation; a powder ring has no orientation to couple to. --calibrant takes lab6, agbh, ceo2, si or ice. A calibrant is a list of ring positions rather than a unit cell, because hexagonal ice is P6_3/mmc: rings enumerated from its cell would include systematically absent ones. So the crystalline standards generate their rings from a cell and ice carries the measured list, and RingsFromAzimuthalProfile, GuessGeometry and OptimizeGeometry all take ring q. The calibrant table is shared with the viewer's powder panel, which previously carried its own copy. --calibration picks how the rings are measured: rings (default) sums the (q x azimuth) profile over every processed image and fits the arcs in it; spots pools the found spots and fits those. Both use the whole run, with -s/-e/-t selecting images. rings defaults --azim-phi-bins to 32, since a profile with one azimuthal bin has averaged the ring over every direction and cannot locate it. Two fixes this exposed: The extraction window is capped at half the gap to the neighbouring ring. The background under a peak is taken from the ends of its window, so a window wider than half that gap measures the next ring's flank as this ring's background - and hexagonal ice has three rings within 0.06 1/A. Ice calibration was 3.5 px out before this and 0.29 px after; LaB6 is unaffected. RingOptimizer holds rot1/rot2 fixed when only one ring is present. A tilt and a centre offset both move a ring as cos(phi) and are separated only by the tilt's amplitude growing as the ring radius squared, so on a single ring they are exactly degenerate. Measured. LaB6 at five distances: the fitted direct beam is within 0.36 px of an independent implementation out to 300 mm, and D = -0.046 + 1.000788 dtz with an rms of 0.011 mm. At 500 mm one ring is fully on the detector and a second only clips the corners, which is not enough to constrain a tilt - restricting the q range to the resolved ring recovers 0.06 px. Ice: 5.53 -> 0.29 px on one crystal and 4.71 -> 0.80 px on another, against XDS's refined direct beam. On an ice-free crystal the fit is worse than the header, which is the correct outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:00:03 +02:00
leonarski_fandClaude Opus 5	d81d2e4696	docs: state the metric-symmetry rule rather than how it was arrived at ... CPU_DATA_ANALYSIS describes how the pipeline works; the account of which bar was tried first belongs in the commit that changed it. Same facts, no narrative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 09:30:10 +02:00
leonarski_fandClaude Opus 5	cfd3697ddb	docs: cover the azimuthal sigma clip, the indexing flag, and the metric-symmetry check ... Three things had reached the code without reaching the documentation. The azimuthal sigma clip had a RUGNUX.md row and a CPU_DATA_ANALYSIS section but no changelog entry - and the only "sigma clip" the changelog mentioned was the background ring's, which is a different thing at a different stage. --index-ice-rings was in the options table but nowhere in the changelog, so the entry describing the ice gate still implied that whether indexing uses the ice-band spots is decided per run, which it no longer is. CPU_DATA_ANALYSIS section 6 still described the Bravais class as simply "the highest-symmetry class that matches within tolerances", which is the behaviour that lost a crystal outright. It now records that the class is chosen from the UNREFINED candidate against a fixed angular tolerance, that a pseudo-symmetric lattice therefore gets promoted a class too far, and that the first pass settles it on validation-frame counts with a clear-majority bar - including why the bar is a majority rather than a margin, since that distinction is the whole reason the check is safe. It also records that the first pass finds its own spots rather than reading the acquisition's, which was not written down anywhere. Also a build note: M_PI is not standard C++ and MSVC does not define it, so the Bragg integrator's use of it broke the Windows viewer build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:56:08 +02:00
leonarski_fandClaude Opus 5	1b5e2e85fd	Regenerate the API documentation from the spec ... update_version.sh at 1.0.0-rc.161. The only substantive change is the one that had drifted: the spot-finding ice-ring half-width was still documented as 0.02 in the generated Python client and its docs while broker/jfjoch_api.yaml has said 0.03 since the band was widened to the measured ring FWHM. Anyone reading the client docs - or relying on the client's default when omitting the field - got a band two-thirds the width the pipeline actually uses. The TypeScript frontend client regenerates identically (the spec itself did not move), and python-client/ is not tracked here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:30:45 +02:00
leonarski_fandClaude Opus 5	1df9556ec1	Bragg integration: default the radial background correction off again ... Auto rode in with the ice work rather than on its own evidence, and measured over the 37-crystal rotation battery it does not carry itself yet. It TARGETS correctly - it fires on ten crystals and every one is ice-positive, no failures, no space-group changes - but it costs 1.35x the wall clock (median +3 s per crystal, worst +29 s) and on the merge statistics it is the familiar sign-mixed trade: high-shell CC1/2 worse on three of the four crystals that move materially, mean -0.76. The case for it is real but rests on agreement with a fixed external model - 43 % of the ice bands' excess amplitude removed on smooth ice, the effect 7x stronger inside the bands than outside - which is the better arbiter and also the narrower one. That deserves settling on its own, not riding along with a set of ice defaults. `--background-radial=auto` keeps it a flag away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:29:42 +02:00
leonarski_fandClaude Opus 5	a6be35ccdb	Azimuthal integration: optional sigma clipping of the reported profile ... The profile is the MEAN of each bin, so a few strong reflections landing in a bin lift it exactly as a smooth powder ring does. That is the wrong quantity whenever the profile is wanted as a background rather than as a measurement of what is in the bin - the ice score being the case in point, where reading a plain profile INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals had no ice at all. The adaptive spot finder already computes the right thing, a sigma-clipped per-resolution-ring background, as a byproduct of its own threshold. Where it runs, the ice score uses that. Where it does not - --no-adaptive-spots, --azint-only, and anything reading the profile the broker wrote - there was no way to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip), 0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian bin and biases the estimate low rather than removing outliers. Two clip passes follow the plain one, matching the finder's recipe - the first pass's standard deviation is itself inflated by the peaks being removed, so one pass leaves a threshold that is still too generous. A bin with fewer than eight pixels is left alone: at the detector edge and behind the beam stop there is no spread to clip on. Both engines do it. On the GPU the accept range is computed by a small kernel and stays resident, so a clip pass is one more read of the same pixels and no round trip; the two accumulation kernels take the range as a pointer that is null on the plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict, against a 0.34 ms per-image total. Note what the result IS: the smooth background under the peaks, not the bin mean. It should not be switched on where a ring's integrated intensity is wanted - the powder-ring geometry fit reads ring peaks, and those are what a clip is designed to remove. Off by default, so nothing changes unless it is asked for. Not exposed over the REST API - that needs the generated model regenerated, which is a separate step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:18:52 +02:00
leonarski_fandClaude Opus 5	3d4209d803	docs: the first-pass ice measurement, the promotion fix, and powder-ring geometry ... Changelog entries for the three changes above, and a new CPU_DATA_ANALYSIS section on determining detector geometry from powder rings: why a ring is an independent constraint on the beam centre (it has no crystal orientation to be gauge-coupled to, unlike everything else that fits geometry here), what a ring can and cannot determine, and how the ring points are obtained. The section states the harmonics correctly, which is worth writing down because the intuitive version is wrong: a detector tilt shows up as cos(phi), the same harmonic as a beam-centre error, and the two are separated by the amplitude growing as the ring radius SQUARED - so it takes at least two rings, and on one ring they are exactly degenerate. The genuine cos(2 phi) term is hundredths of a pixel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 22:12:21 +02:00
leonarski_fandClaude Opus 5	b5f5879a1d	rugnux: measure the ice in the first pass, and always find its own spots ... Ice handling was gated on a measurement the run only made AFTER the images had been processed, so the per-image pass could not use it. The flagging therefore ran unconditionally: ice-band spots were ordered last in the --max-spots budget and held out of the indexer seed and the geometry refinement on every crystal, iced or not. The eleven bands are fixed geometry holding 16-26 % of the unique reflections whether or not there is ice, so on a clean crystal that discards a fifth of the spots - the strongest first - for nothing. Measured on a crystal whose gate never fires, that moved the merged data by a mean of 0.85 sigma against a run-to-run floor of 9.3e-5. Measure it in the first pass instead. That pass already looks at ~100 images spread over the sweep, and it already stops at the spot finder, so it sees the azimuthal profile for the smooth channel and the unfiltered connected components for the spot channel. Both counts SpotAnalyze takes are pre-filter, so pooling them there is the run's own verdict, reached before anything has been discarded and in time for the pass that acts on it. Where the sample sees no ice, the run indexes on the ice-band spots too. It has to be the whole sample: the spot channel is a ratio pooled over images, because one frame carries a handful of control spots. A per-image gate is not an alternative - two of the crystals whose indexing this rescues fire on that channel alone, at profile scores of 1.12 and 1.22, so gating per image on the profile score would drop exactly the cases that matter. This also removes the first-pass spot reuse, and with it --redo-rotation-spots and the reuse path. Finding the ~100 first-pass spots costs little, and reusing was actively wrong here: the stored spots were found online at the acquisition's threshold and have already had their ice-band entries ordered last and dropped by its spot budget, so counting ice from them under-reads it by construction, and the lattice search never saw the spot-finding settings at all. It also removes the need for the machinery that re-found spots whenever a spot-finding option was named, which made those options impossible to A/B. IndexAndRefine cached index_ice_rings at construction, which happens before the first pass; it holds a reference to the experiment, so it now reads the setting where it uses it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 20:48:22 +02:00
leonarski_fandClaude Opus 5	f0cdb027e1	Ice: default the merge mask off, gate the radial background on smooth ice, and pick detection by geometry ... Three defaults, each settled by measurement rather than by argument. The arbiter throughout is structure-referenced - anomalous peak height where a crystal can carry it, and otherwise the agreement of the ice bands with a fixed external model against resolution-matched DECOY bands carrying no ice. The band-versus-decoy contrast is used because R-free here tracks completeness, and every one of these switches moves completeness. The damage is real and it localizes: over the rotation battery the ice bands' excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst, while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the ten largest excess peaks in a q scan land on hexagonal ring positions. Turning ice handling off leaves the contrast unchanged and forcing it on a clean crystal does not create one, so it is the ice and not the machinery. MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band reflections in the merge and exclude them only from the model fit; autoPROC is the sole exception. On the one battery crystal where the mask fires and an anomalous arbiter can score it, dropping the band moved the mean peak height at the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing 1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own 3.05 - better than average data - and costing 17 completeness points in that shell. It fires on 5 of 37 crystals, changes no space group, and those 5 disagree in sign: it clearly helps the two most heavily iced, is a wash on two and costs a third. So it stays as a switch, worth setting by hand on a badly iced crystal where it shows in the high shell, but it is not a default. RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background as a function of radius alone, and that is exactly when it works. On a crystal with pure smooth powder ice it removes 43% of the bands' excess amplitude, with the improvement 7x larger inside the bands than outside; on a crystal whose ice is discrete crystallite spots - no smooth ring to model - the excess amplitude GREW by half; on clean data it is inert to four decimals. The two ice channels already separate those morphologies, so --background-radial takes on|off|auto and auto applies it to an image when that image's peak-excluded score reaches --ice-min-score. Auto never engages without such a score, because the plain profile carries the Bragg peaks and cannot support an absolute threshold. Per image rather than per run, and that was tested rather than assumed: the gate fires on 100% and 94% of frames on the two crystals that want it, and on 1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice crystal. A seam statistic against off + f*(on - off) is null on both mixed runs, every merge statistic is bracketed by the pure arms, and the textured crystal's auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need the score before the pass that integrates, i.e. rotation-only plumbing, and buys nothing measurable. The kernel was already built unconditionally, so flipping the flag per image is free - except on the GPU, where the launches were gated on a construction-time n_rad. That is why the buffers are now allocated whenever the correction could run, and Run() decides per image. DETECTION -> the geometry's default when the file is silent: on for rotation, off for stills, with the command line and then the file taking precedence. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent systematic and the presence gate keeps it inert on a clean crystal; a serial stills run has too few spots per image to spend any on flagging. The master file's key is kept as written rather than collapsed to a bool, so "the file said nothing" is distinguishable from "the file said no" - it used to fall silently to off, taking the exclusion from the scale fit with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 19:06:30 +02:00
leonarski_fandClaude Opus 5	61a7c91b90	Ice: detect it on two channels, and only handle it when it is there ... The per-image ice score was read off the PLAIN azimuthal profile. That profile is a per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it exactly as ice would. Measured over 37 rotation crystals, that did not merely add noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all (4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its 99th percentile and 2.70 at its maximum, so that metric cannot support any absolute threshold whatsoever. The adaptive spot finder already computes the right input for its own threshold: a sigma-clipped per-resolution-ring background, in the same bins. A powder ring is azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at 2.08-2.37, against a decoy null that never exceeds 1.29. That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE spots and leaves the radial profile flat. So a second channel counts found spots on the rings against the same q width of ice-free flanks beside them. The two barely overlap - the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6, while a clean crystal reads 1.04 on both. Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now all skipped when neither channel sees any. The gate is applied in the full pipeline and in --scale, which reads the stored per-image values back out of the _process.h5. Also fixes the merge-time mask's control: the shoulder now excludes reflections that are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on the neighbours and the test compared ice against ice. Measured, that is the only thing this changes: it removes firings on those three rings and leaves every other firing's CC pair identical to three decimals. And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the same data got a narrower band online than the measured ~0.06 ring FWHM justifies. Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses. Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3 and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa 3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The one crystal that loses reflections improves on both R_meas and CC1/2. Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5 write/read and the receiver plots are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 16:17:23 +02:00
leonarski_fandClaude Opus 5	227f1bf1b4	Add rugnux_anomalous.py: judge partiality changes by anomalous peak height ... A change that touches partiality - a mosaicity estimator, a rocking-curve model, a background change - cannot be judged by the statistics we normally reach for, and this was learned the expensive way. ISa is anti-correlated with external accuracy and is the largest mover of any statistic; last-shell R_meas moves with its denominator, i.e. the wrong way by construction; `--model` R-free tracks its own zero-information floor, which shifts ~22x more than R-free itself over the same sweep; and per-shell agreement with XDS is biased, because XDS never divides by partiality, so "divide less" moves us toward it mechanically - measured to put the optimum ~1.4x too low. Anomalous difference density at known scatterer sites has none of those problems. It is read in units of the map's own sigma, so the uniform intensity rescale a partiality change produces cancels exactly, and it is referenced to the structure rather than to another program's partiality model. The script runs SHELXC + ANODE per arm against a model placed ONCE and then held fixed, and reports the mean site height, the off-site noise floor, and the paired per-site change between arms. Numeric arm labels turn a set of arms into a curve with a per-dataset optimum. The dataset table lives outside the repository, as rugnux_vs_xds.py already does for the battery, because dataset and sample identities are not committed. It reproduces the measurements it was built from: all nine points of three pooled curves, every per-crystal optimum, the site heights, the paired t statistics, and the adversarial control in which a model refined against the worst arm reproduces the curves to <=0.005 and the same optimum on 4/4. Four things the ad-hoc scripts it replaces got wrong, all now handled: * Keying sites on the ANODE atom label silently drops an alternate conformation sharing that label - one dataset class has 18 sulfur sites, not 17, and the uncorrected mean read 15.03 against a true 14.49. * The off-site floor skipped any peak within 1.0 A of ANY atom, so a ripple sitting on a light atom was not counted as background; requiring 1.5 A from an anomalous scatterer raises one floor from 7.12 to 9.48 sigma. * Special-position peaks are Fourier ripples, not background. Excluding them is load-bearing on 3 of 9 datasets and they are now reported in their own column rather than dropped silently. * Enantiomorph care turned out to be unnecessary - passing the merged file's screw label to ANODE while the model sits in the other hand gives byte identical peaks. What does matter is the pair whose absences are identical, I23 vs I2_13, which phaser's automatic hand test does not cover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 22:52:39 +02:00
leonarski_fandClaude Opus 5	bb7df09086	rugnux: separate the merge-time ice-ring mask from ice detection ... --detect-ice-rings did two unrelated jobs at once: flagging ice spots so indexing de-prioritises them and keeping ice reflections out of the scale fit, AND gating the merge-time mask that drops a decorrelated ice ring and re-merges. Turning it off to de-confound a merge-stage experiment therefore also changed how the data were indexed - measured, that breaks indexing outright on two of the 37 rotation battery crystals - while leaving it on lets the mask land differently between two arms of an experiment and contaminate the comparison (measured on up to 19 of 37 crystals in response to a small intensity change). Add --ice-ring-mask[=on|off], default on, gating only the merge-time mask. Verified with =off: ice-spot flagging and the scaling exclusion still log and still apply, no mask line, no second merge, and the first error model is bit-identical to the =on arm. The full pipeline and the offline --scale path reach the same verdict on the same data, as they must. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 19:00:10 +02:00
leonarski_fandClaude Opus 5	09fb8e0306	Bragg integration: clip the background ring high side instead of trimming it ... The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A symmetric trim is not a consistent estimator of the mean of a right-skewed (Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true mean at every level, and with ~50 signal pixels in the r1 disk that under-subtraction adds ~5 counts to every partial on every frame. Measured two independent ways on four rotation datasets - stored background_mean against a plain ring mean over the same pixels on reflection-free frames, and directly on apertures that provably hold no reflection. Empty-aperture pedestal, counts: plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for monochromatic data. It rejects the same one-sided contamination the trim was there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds ~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The trim stays reachable with --background-trim for back compatibility; setting either estimator clears the other, so they can never stack. --integrator boxsum does not take the clip (matching what the shipped clip already did), so it now uses the plain ring mean unless --background-trim is given. The intensities get measurably more accurate: per-shell agreement with an independent processing of the same images improves on 14 of 16 crystals (weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR - absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and CC1/2 in the outer shell improves on 14/7. EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by information-free counts, so both get worse when the bias is removed; neither is evidence against this change. That fingerprint is exactly how the trimmed mean was accepted in the first place. Known cost: over the 37-crystal rotation battery the de-novo space-group count goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal whose merge fails the absolute-sanity gate under either background (R_meas 63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the very operator being scored, so its operator CC is diluted by construction. No other crystal changes space group, and twin protection is not weakened - the H-ratio veto that refuses genuinely twinned crystals gets MORE decisive (1.63 -> 1.84, 2.83 -> 3.99). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 18:03:11 +02:00
leonarski_fandClaude Opus 5	fb55645b81	Revert "rugnux: fit the profile radius from the strongest spots too" ... Reverts the profile-radius part of 457b1bfd1; the comparison-script and mosaicity-column changes from that commit are kept. The cap was validated on the rotation battery, which cannot test it: the profile radius feeds `ewald_dist_cutoff` in IndexAndRefine, and that is read only by the STILLS predictors (BraggPrediction/BraggPredictionGPU). The rotation predictors gate on the mosaicity window instead and never look at it. So "no space-group changes, 36 of 37 crystals bit-identical" showed the quantity is inert for rotation, not that capping it is safe - and the one regime where it does act was never exercised. Validating it needs the serial-stills battery, which is a much larger exercise. Until then the arbitrary constant is not worth carrying in a code path nobody measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 21:35:17 +02:00
leonarski_fandClaude Opus 5	5eb386e333	docs: describe the per-frame geometry smoothing ... Goes in §10.3 next to the per-frame scale and mosaicity smoothing, since it is the same mechanism applied for the same reason, and trims the changelog line to one sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 20:23:19 +02:00
leonarski_fandClaude Opus 5	017f64690c	rugnux: smooth the per-frame geometry before scaling ... Geometry is re-refined independently on every frame, against that frame's spots alone - as few as a dozen on a sparse crystal, where XDS fits its equivalent to about sixty times more data. Measured over ten datasets the per-frame orientation carries two components: a slow drift that is real, with rugnux and XDS agreeing to R^2 0.83-0.88 on the two crystals that genuinely slip by 1.5 and 0.54 degrees, and a fast jitter that is fit noise, scaling with spots-per-frame at exponent -0.79 where counting noise alone would give -0.5. The jitter is worth 1-8% on merged intensities, 24% on the sparsest crystal. It cannot be fixed by refining less. Turning per-image refinement off entirely loses six space groups and a whole crystal, and even a 624-spot-per-frame crystal collapses; dropping the beam-centre terms holds the space groups but is worse on 31 of 37 crystals. The freedom is earning its keep, so keep it and suppress only the band that cannot be physical - a crystal does not re-orient and snap back from one frame to the next. So smooth the orientation in frame order after integration and recompute each partial's delta_phi, and hence its partiality, from the smoothed lattice. Batching at integration time was not an option: frames are processed independently and the online path depends on that. This runs before the GPU upload, so the device path picks it up with no separate kernel. The window is chosen per dataset by leave-one-out cross-validation, because the two components vary far too much for one number - drift spans 0.018 to 1.288 degrees and jitter 0.005 to 0.221, so any fixed window over-smooths one crystal while under-smoothing another. Chosen windows range from +-1 to +-20 frames. It is capped: cross-validation scores how well neighbours predict a frame's orientation, which on a barely-drifting crystal keeps improving with width, but the per-frame fit is also absorbing a real per-frame systematic and smoothing too wide destroys it - uncapped, one crystal chose +-60 and lost 16% of its ISa. Battery over 37 crystals: space groups unchanged at 34 matching XDS, R_meas better on 31 and worse on 6, low-resolution R_meas 30/7, ISa 26/10, high-resolution CC1/2 23/12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 20:06:41 +02:00
leonarski_fandClaude Opus 5	457b1bfd1d	rugnux: fit the profile radius from the strongest spots too ... Same defect as the mosaicity in 2c94f3013, in the same file's sibling fit. The profile radius is an RMS of the excitation error over whatever spots were kept, and weaker spots sit further off the Ewald sphere, so it grows with the depth of the list: measured over a 150 -> unlimited spot budget it rises ~60%, and on a clean dataset as much as on a hard one, so this is general rather than something one awkward crystal provoked. That made it a function of --max-spots, which is an indexing budget, rather than of the crystal. Its one consumer treats it as a membership gate (ewald_dist_cutoff is twice it) where reflections at the cutoff carry near-zero partiality and are removed downstream anyway, so the integrated data barely notices: with the mosaicity already pinned, the partial count moves 0.2% across a 34% change in the radius. It is also reported per image as a diagnostic, though, and a number that slides with an unrelated setting is misleading to anyone comparing two runs - and it would stop being benign the moment anything used it as a width rather than a gate. Battery over 37 crystals: no space group changes, 36 of 37 bit-identical, no failures, one crystal marginally better. Also in the comparison script: report XDS's mosaicity next to rugnux's. XDS has two and they are not interchangeable - CORRECT.LP's REFLECTING_RANGE_E.S.D. is post-refined, while INTEGRATE.LP's per-batch SIGMAR is its integration-stage estimate, and the two differ by up to 2.3x. The XDS cell now prints both as postrefined|MLE so a per-image estimate is compared against the one measured the same way. Fixes a scoping bug in the same addition where every crystal read the last directory's INTEGRATE.LP. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 10:52:30 +02:00
leonarski_fandClaude Opus 5	2c94f3013e	rugnux: fit the mosaicity from the strongest spots only ... The per-image mosaicity MLE ran over the whole indexed spot list, so it rode on --max-spots, which is an indexing budget. A spot is detected when I_full * R(tau) clears the finder threshold, so selecting by intensity censors on R(tau): a deeper list holds proportionally more large-|tau| partially recorded spots and the fit widens with it. Raising the budget 250 -> 1000 widened sigma_M 0.059 -> 0.075 deg on a rotation dataset whose measured rocking width says 0.054. That is not cosmetic. An over-wide mosaicity mis-states every partiality in scaling: forcing the mosaicity across that range moved the merge error model from b 0.039 / ISa 26 to b 0.167 / ISa 6, and the space-group search lost a genuine 422 with it, merging the crystal in 222 instead. Cap the fit at the strongest 250 spots. FilterSpotsByCount leaves the list strongest-first, so this selects exactly the spots a smaller --max-spots would, and the mosaicity becomes invariant: 0.0538 deg at 250, 500, 1000 and 2000 spots, with the correct space group at each. Trimming or down-weighting the tau tail does not work - the censoring is multiplicative in R(tau), so it widens the whole distribution rather than adding a tail. Battery over 37 crystals: exactly one change, the demoted crystal repaired (33 space groups matching XDS -> 34). 23 of 37 are bit-identical, never reaching 250 spots. Unaffected elsewhere: the default spot count is 250, and stills have no goniometer so they return before the fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 07:34:40 +02:00
leonarski_fandClaude Opus 5	f25fea7024	rugnux: keep 1000 spots per image instead of 250 ... Offline reprocessing is not bound by the online spot budget, and the cap is applied at the end of SpotAnalyze, so it is exactly the spot list the indexer and the per-image refinement see. jfjoch_viewer already sends 1000, so the two front ends now agree on the same file. Measured as a paired A/B over the 37-crystal rotation battery, de novo, with the resolution and Friedel setting matched to the XDS reference, both arms from the same binary bar this constant: R_meas low shell 16 better 0 worse 19 unchanged R_meas 14 better 4 worse 17 unchanged ISa 14 better 6 worse 15 unchanged CC1/2 6 better 3 worse 26 unchanged Low-resolution R_meas is a clean sweep. Around half the battery is bit-identical: those frames never reach 250 spots, so the cap never bound. Wall clock is unchanged (10m00s vs 10m44s, uncontrolled for page cache). Known cost, and the reason this is its own commit: one crystal in the battery reproducibly loses symmetry, tetragonal 422 -> orthorhombic 222, doubling its asymmetric unit. Its R_meas and ISa "improve" there, but that is what merging in too low a symmetry always does, and the lower symmetry then admits a merohedral indexing ambiguity. An intermediate cap of 500 demotes it too, so it buys none of the safety. This is the known point-group-decision-moves-with-data-amount fragility of the space-group search rather than an argument for starving the indexer of spots - the search is the thing to fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 22:08:21 +02:00
leonarski_fandClaude Opus 5	2473e03cf7	rugnux: default --spot-sigma to 4.0, the value the viewer already uses ... The two front ends disagreed on the fixed-threshold spot finder: rugnux started from 3.0, jfjoch_viewer from the SpotFindingSettings default of 4.0, so the same file processed either way could give different spots. Inert on the default path - the adaptive finder derives its threshold from each image's own per-resolution-ring noise and never reads signal_to_noise_threshold (only ImageSpotFinderCPU/GPU and DetModuleSpotFinder do). It changes behaviour only under --no-adaptive-spots, and there it now matches the viewer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 22:07:58 +02:00
leonarski_fandClaude Opus 5	cd0c43c65c	docs: add the packaging and documentation work to the rc.161 change list ... Covers the Debian package rename and the DKMS scripts, the writer's --verbose, the version plumbing, and the documentation pass - the release-contents page, the corrected repository URLs and package names, and the rugnux, viewer and writer pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:53:48 +02:00
leonarski_fandClaude Opus 5	cfcb84aece	docs: bring the tool pages back in line with the programs ... rugnux gained --model - R-free and 2Fo-Fc/Fo-Fc maps against an atomic model, and with it the resolution of the enantiomorph and of a merohedral indexing ambiguity - without the page ever mentioning it. It was the only option missing; the two lists now agree in both directions, checked against the usage the binary prints. The viewer page still said results are never saved and that no Windows package exists. Both have been false for a while: the Processing panel runs full rugnux jobs on the open dataset, writes _process.h5 and the merged reflections, registers each run as a selectable view so runs can be compared, and can hand out the equivalent command line for a cluster; and the installer is published with every release. The mask menu also loads TIFFs now, and the View menu has layout presets. The writer page documented -R for the root directory, which is the back-compatibility alias for -d, and an HTTP status interface that no longer exists - status reaches the broker over the writer notification socket, and a writer is stopped with a signal. The test page pointed at .gitlab-ci.yml and at jfjoch_offline_process, which is not a binary any more; the CrystFEL fixture pointed at HDF5DatasetWriteTest, which is not either. The broker page linked ../broker/redoc-static.html, which MyST resolved by copying the 700 kB file into _downloads/ rather than using the copy already in _static. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:53:37 +02:00
leonarski_fandClaude Opus 5	46930142dd	docs: write down what a release ships ... Nothing said what is in a release or what it needs of the machine it lands on: that the Linux binaries are built -march=x86-64-v3 and the Windows ones /arch:AVX, so each has a CPU floor; that the portable .tgz is built on RHEL 8 for its glibc; that the Windows installer is MSVC (Visual Studio 2026), CUDA 13.3, Qt 6.11 and carries the Qt runtime; and above all what the CUDA variants need. Only cuFFT is linked dynamically, and it has no link-time dependency on the driver library, so a CUDA build starts on a machine with no NVIDIA GPU at all and falls back to the CPU path - as long as cuFFT can be loaded, which the .tgz and the installer arrange by shipping it and the distribution packages arrange through the distribution's own CUDA packages. Collected into a new page rather than scattered over the install instructions. The repository page had the RHEL 9 rows pointing at el8 paths under the wrong slsdet number, no rows at all for the two slsdet9 repositories the pipeline uploads, a driver package named jfjoch-driver where it is jfjoch-driver-dkms, and a note that RPMs are unsigned from before the pipeline started uploading them with sign=true. The FPGA page had a paragraph that stopped mid-sentence, in the middle of a link, and a section describing a firmware build triggered by commit message. The firmware is stable and carried from version to version now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:53:19 +02:00
leonarski_fandClaude Opus 5	8712160adb	scripts: point the generated Python client at gitea, and make the scripts runnable ... The openapi-generator invocation still passed --git-host=git.psi.ch and a user id of jungfraujoch, from before the move to gitea.psi.ch/mx/jungfraujoch. Those properties are not cosmetic: they become the source URL in the generated README and pyproject, so the published client documentation - docs/python_client/README.md, which is copied out of the generated tree - told readers to pip install from a host that no longer answers. Regenerating with the corrected flags changes those two lines and nothing else, verified against the committed tree. update_version.sh, make_doc.sh and gen_python_client.sh were all mode 644, so the "run update_version.sh" the documentation asks for fails on the shebang. CMake and the CI both work around it by invoking them through bash. make_doc.sh builds a throw-away venv in the working tree and deletes it on the last line, which set -e skips whenever pip or sphinx fails - so a failed docs build left tmp_venv/ behind. Delete it from a trap instead, and ignore it along with the default output directory and the sdist directory gen_python_client.sh creates. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 20:52:29 +02:00
leonarski_fandClaude Opus 5	ceb92fc4cc	docs: bring the rc.161 change list up to date ... Covers the GPU decode work (fused un-transpose, the memory it frees, corrupt-chunk detection, large bitshuffle blocks, host fallback), the two reproducibility fixes (integer ring statistics, striped geometry-refinement sampling), the connection-teardown and written-extent fixes, and the thread-count, max_hkl and compression-time repairs. Folded into the existing entries where they belong rather than added as new ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 16:58:03 +02:00
leonarski_fandClaude Opus 5	b6c4a59d69	docs: record the stored-format break, and say how the adaptive finders actually accumulate ... The per-image image-scale B factor was dropped from the CBOR stream and from the written HDF5, which is a change for anything reading those files, but the changelog listed it only under the OpenAPI breaking changes. The GPU adaptive finder test claimed both finders sum the rings in double. The CPU one does; the GPU one stages a block's contribution in float before reducing across blocks in double, deliberately, to keep the hot loop's shared footprint down. Say so, and say what follows from it - detection compares integer pixel values, so a threshold that crosses an integer flips every pixel of that value in the ring at once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:14:24 +02:00
leonarski_fandClaude Opus 5	bec7e2e922	image_preprocessing: fuse the bitshuffle inverse with preprocessing, and verify the decode ... The device decoder was byte-exact on every valid input - 994 production-compressed images, 927 hand-built LZ4 blocks covering engineered (offset, matchlen) pairs across the overlap branch boundary, 18000 repeat decodes, sanitizer-clean - and an audit against LZ4_decompress_generic could not construct a valid block it mis-decodes. What it did not do was notice when the input was NOT valid, and that mattered more than it looks: the decode buffers are reused frame to frame, so a block that stopped early left the PREVIOUS image in place, and in the bitshuffled layout the untouched tail is the most significant byte-plane. A corrupt chunk therefore did not look like a missing corner. It looked like thousands of real pixels several powers of two too bright, fed to spot finding with no diagnostic, where the host decoder had raised an error. So the kernel now flags a block that fails to reach its declared length while consuming exactly its payload, and the host turns that into an exception once the caller has synchronised. Reads are clamped against the end of the payload as well as the output, both length chains are bounded exactly as read_variable_length bounds them, the two offset bytes are bounded, and LZ4's parsing restrictions are enforced. On the host side a block size that is not a multiple of 8 elements is rejected (it made the un-transpose read uninitialised shared memory), the block count is bounded by what the chunk could hold before it becomes an allocation (twelve header bytes could demand hundreds of MB of pinned memory, permanently, per worker), trailing bytes are rejected, and the stream is synchronised before any throw that happens after work is queued. An image of fewer than 8 elements is all verbatim tail and now decodes rather than throwing. When the device route fails for any reason the host decoder gets its turn, so it costs speed rather than the acquisition. The lanes cooperate on the copies and a later match can read bytes another lane wrote, which since Volta needs an explicit __syncwarp(); it worked only because ptxas happened to reconverge at the post-dominator. The prototype's offset == 1 and power-of-two fast paths are also restored - the shipped kernel ran a runtime modulo, an emulated 32-bit division per output byte, on the path its own comment calls the common case. The un-transpose is now fused with preprocessing. One thread owns one group of 8 elements across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8 complete elements and emits 8 finished int32 pixels with the mask, the error marker, the saturation cap and the statistics applied. The decompressed image is never materialised: 0.623 -> 0.411 ms/frame at 18 Mpx, 0.523 -> 0.340 with 8 concurrent workers. Staging nothing in shared memory also drops the 48 kB ceiling, which had made any file whose bitshuffle blocks exceed it a hard failure; 64 kB blocks now decode. gpu_compressed is sized from the chunk with grow-on-demand instead of from the uncompressed size - it was reserving ~73 MB per worker to hold ~4 MB. Measured on a 1630x1553 uint32 rotation set at -N 32, peak GPU memory falls 3756 -> 3084 MiB; the same model gives ~144 MB per worker on an 18 Mpx frame. Decoding on the device also stopped reporting a decompression time, which blanked the broker's compression plot trace and filled /entry/profiling/compressionTime with NaN. The decoder brackets the decode with CUDA events and reports it again. Tests: a differential fuzz suite against the CPU decoder - incompressible and highly compressible data, engineered offsets, a size sweep hitting every rem%8 value twice, all six element sizes, an 18 Mpx frame, decoder reuse, concurrency, hand-built LZ4 blocks across the overlap boundary, 26 foreign bitshuffle block sizes from 128 B to 64 kB, corrupt payloads and malformed containers, with a coverage report that proves which LZ4 paths were reached rather than assuming it. Plus the fused path held byte for byte against ImagePreprocessorCPU, statistics included, and against the host-upload path on the same frame. Battery: 37 crystals, every merged number identical to the host-decode run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:13:55 +02:00
leonarski_fandClaude Opus 5	47277674fa	docs: split the breaking API changes out of the rc.161 change list ... Earlier releases put breaking changes in their own paragraph after the bullets (rc.139, rc.29) rather than as one item among them. Follow that: the OpenAPI changes now sit under their own heading below the list, with the client-side action in the lead line, and are listed one per change instead of run together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:46:57 +02:00
leonarski_fandClaude Opus 5	737cbde3ff	docs: tighten the changelog and the decoding section for release ... The rc.161 changelog had grown entries several hundred words long and listed the same area three or four times over. Collapse them by subject - spot finding, resolution limits, space-group search, scaling, performance, correctness - and hold each to one line, keeping the actionable detail in the breaking API entry. Add the performance work that had not been written up: device-side image decoding and the parallel first-pass candidate-cell refinement. Section 0 of the CPU analysis document was the longest thing in it after two core algorithm sections, and most of that was a profiling narrative and the measurements that motivated the change rather than a description of what runs. Cut it to the two kernels, the host-side block scan and the fallback rule. The attribution stays; it is also in the reference list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:44:55 +02:00
leonarski_fandClaude Opus 5	6e4c0ce202	image_preprocessing: decode bitshuffle+LZ4 on the GPU ... The pipeline decompressed each image on the host and uploaded the result. On an 18 Mpx rotation dataset that made the host-to-device copy the bottleneck of the whole per-image loop: nsys puts the copies at 78% of the loop against 39% for every kernel combined - 3600 transfers of 72.4 MB - and they ran at only 12.5 GB/s of an available 27-28 because the host-side decompression was itself saturating host memory bandwidth. The GPU was mostly waiting. So the compressed chunk goes across instead, about 4 MB rather than 72 MB, and is decoded on the device. That removes the transfer and the host decompression that was throttling it, in one change. Measured on an idle machine, a run goes from 45.11 s to 24.97 s - 1.81x - with the merged output unchanged. THE APPROACH IS JON WRIGHT'S (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group 2021, and github.com/jonwright/ bslz4decoders. The kernels here are ours, but the idea and the demonstration that it is worth doing are his. Cited in docs/ACKNOWLEDGEMENT.md and in the new section 0 of docs/CPU_DATA_ANALYSIS.md. Two kernels mirror the CPU decoder. LZ4 runs one WARP per bitshuffle block: every lane parses the same sequence stream (a broadcast read, no divergence) and the literal and match copies are split across the 32 lanes so the stores coalesce; an overlapping match is treated as a pattern of period offset sourced from bytes that already precede the write position, which keeps it parallel rather than a serial byte loop. One thread per block instead measured 13x slower. The bitshuffle inverse then un-transposes each byte-plane through shared memory and interleaves the planes back into elements. Only BSHUF_LZ4 is decoded on the device. The zstd variants have no device decoder, and neither has an uncompressed or float image; Supports() returns false for those and the caller decompresses on the host exactly as before. The fallback is explicit, so a format we cannot decode on the device is a slower path and never a wrong answer. Tests hold the device decoder against the CPU one byte for byte, on data from the production compressor, for every element size the detectors emit - including the 8-bit DECTRIS modes, which take bitshuf_decode_block's separate elem_size == 1 branch - plus a many-block frame, the formats it must decline, and malformed containers, which must throw rather than run off a buffer. Battery: 37 crystals, no failures, identical to the host-decode run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 00:16:36 +02:00
leonarski_fandClaude Opus 5	d55f3257ed	ci: regenerate the published API artefacts, and build Linux with LTO ... update_version.sh had not been run for the adaptive spot-finding schema change. Running it leaves the C++ server model and the TypeScript client byte-identical to what the generators produced directly, but it also regenerates two artefacts the direct calls do not touch and which are tracked: the Python client's published documentation and the Redoc bundle. Both now carry adaptive_threshold and false_pixels_per_frame. LTO joins -march in the CI flags, which is why MARCH_CMAKE_FLAGS is now LINUX_CMAKE_FLAGS - it no longer describes only the architecture. Measured on rugnux against an otherwise identical build: 7-10% fewer retired instructions and a 9% smaller binary, but only ~1.5% off the wall clock, because the pipeline is GPU- and I/O-bound. It costs about 3x on an incremental rebuild (9.8 s -> 30.1 s for one file plus link), so it stays out of CMakeLists and out of a developer's edit cycle: CI builds from scratch and ships the result, paying the link once. It links against CUDA with no special handling, and both CI images already put gcc-toolset-13 on PATH, which -flto=auto requires. MSVC is left alone: its LTO is a different flag (/GL + /LTCG) and nothing here measured it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 20:21:50 +02:00
leonarski_fandClaude Opus 5	90ab67f376	broker: expose adaptive spot finding over the API ... The self-calibrating detection threshold was reachable from rugnux and the viewer but not online: spot_finding_settings carried no adaptive_threshold, so the receiver always ran the fixed-threshold finder and the fused GPU engine sat unused behind it. adaptive_threshold and false_pixels_per_frame are now part of the schema, both optional so an existing client that sends neither is unaffected, wired through OpenAPIConvert in both directions and surfaced in the frontend panel, where turning the mode on greys out the count threshold it replaces and reveals the operating point it uses instead. The C++ server model and the TypeScript client are regenerated from the spec; the Python client is generated but not tracked. Enabling it is REFUSED where spots are found on the FPGA - the JUNGFRAU and EIGER workflows - rather than accepted and ignored, because a detection setting that silently had no effect cannot be told apart from one that did. The DECTRIS/SIMPLON workflow, which analyses images in software, accepts it. Verified against a running broker: adaptive_threshold true is rejected with that message and leaves the stored settings untouched, while false and omitted both succeed. It stays off by default online, unlike rugnux and the viewer. The broker serves both workflows and the default has to be the one that works on either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 20:01:34 +02:00
leonarski_fandClaude Opus 5	e7be5447d3	receiver: stop copying every frame back from the device on the Lite path ... The Lite workflow built its analysis with the fused GPU engine disabled, which is also what decides whether the preprocessed image is copied device-to-host after every frame. So on a machine with a GPU the online path was moving the whole image back - 72 MB on a large detector, every frame, per worker - for a host reader that does not exist on that path. It was left off deliberately when the fused engine was added, to keep the online path unchanged in that commit, and never revisited. Nothing depends on it: the FPGA workflow uses a different analysis class, and strong-pixel values are read through a device gather rather than from the host image. Turning it on changes no result, and cannot: adaptive detection is unreachable online, because the REST schema exposes no way to enable it, so the classic GPU finder runs either way. Measured anyway, both engines on the same frames across five datasets including very weak ones: 2400 frames, 638260 spots, not one difference - identical lists, identical indexing rate, identical merge statistics to every printed digit. On a large detector with eight workers the median per-image cost falls from 94 to 59 ms and preprocessing from 21 to 6 ms; throughput rises from about 48 to 55 Hz. No percentile regresses, which is what matters for a service - the ninetieth improves from 128 to 74 ms and the tail with it. Spot finding gets faster too, because the large copy no longer contends with the device gather. Correct two statements while here. The flag's comment and the data-analysis document both said the online receiver uses the CPU adaptive finder; online never runs an adaptive finder at all, and the copy the flag really controls was not mentioned. That copy would be better expressed as what it is - whether a host engine will read the image, which the constructor already knows - rather than inferred from which spot finder is wanted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 15:16:18 +02:00