49 Commits

Include renames

Author	SHA1	Message	Date
leonarski_f	1f2dad530c	Space-group search: the merge with all the observations decides ... A rotation run makes the space-group decision twice: once on a merge of only the well-measured observations (--search-min-zeta) and once on all of them, and the rule was to keep whichever search found more symmetry. Its justification was that the filtered arm can only ever LOSE an operator - discarding 40-80% of the observations starves the operator correlations - and never invent one. That premise was checked at one set of geometries, and there it holds exactly: over 1232 stored battery runs (38 geometries x 32 code variants) the two arms disagree 59 times and the all-observation arm is the higher one every single time. Away from those geometries it fails. Over 68 runs whose first pass was given a displaced beam centre, the filtered arm confirms an operator the full merge refuses 14 times, and the two arms never once both confirm the promotion. On one rhombohedral crystal the all-observation merge refuses a 3 -> 32 promotion at the correct beam centre, on the same twin-law statistic the search uses everywhere (1.78 against a bound of 1.70), and the filtered merge - which had thrown two thirds of the observations away - overrides it into the wrong space group. The rule was validated on the only data that cannot test it. So the filtered arm no longer promotes. Where the two disagree the answer is the one the merge with all the observations supports, as the systematic absences already were, and both are still reported. The filter keeps the job it was added for - stopping a near-tangential measurement from making a real operator look like a twin law - it simply cannot outvote the merge that has every observation in it. Costless on the 38-crystal battery, as predicted: the changed branch is never taken there (41 searches: 39 agree, 2 with the all-observation arm higher, none the other way), the space groups stay at 35/38 with the same three misses, and the crystals the second search does rescue - where the all-observation merge is the one finding the higher symmetry - are untouched. Injecting the post-refined beam centre into the crystal above now yields the right space group, 27900 unique reflections at R_meas 12.6% against 14078 at 21.3% before.	2026-08-12 15:07:34 +02:00
leonarski_fandClaude Opus 5	7ce47bd4d1	Radiation damage: report a measurement, or nothing ... The monitor fitted each batch's relative-B on SINGLE observations - ln(I_ref/I_obs) regressed on s^2, weighted by (I_obs/sigma)^2, with the logarithm requiring I_obs > 0. The observation therefore sits in the response and in its own weight, and the positivity requirement keeps only the upward half of the noise, so the estimate is biased downwards wherever I/sigma approaches 1 and is unbounded in the limit. Simulated: on a batch with no relative-B at all and <I/sigma> = 0.3 it reads -28 A^2; on a batch whose true relative-B is +30 A^2 it reads -29. The bias grows with dose, so it inverts the answer on exactly the data the number exists for. That is not a corner case. Re-measured on stored integrated intensities, a 360 deg sweep obstructed over a 60 deg wedge - whose honest curve is flat for 130 deg, dips over the wedge and comes back - printed a per-batch curve saturated at -31.4 A^2 for twelve consecutive batches (the +-50 A^2 clamp less the low-dose anchor, "no data here" reported as a measurement) under a headline of -19 A^2 of radiation damage. A deliberately dosed dataset printed -39 A^2 where the honest measurement is about +34: the one crystal with real damage got the sign wrong. Eight of thirty-eight datasets reported |dB| > 5 A^2 and their curves oscillate by tens of A^2. So pool the observations into ten equal-occupancy resolution shells per batch before taking the logarithm, and fit slope AND intercept over the shell means, weighting each shell by its own pooled (I/sigma)^2. A shell mean is well determined where a single observation is not, it admits negative intensities, and it carries the I/sigma that says whether the batch can be measured at all. The same simulations then reproduce the truth to under 1 A^2 at every signal level. The intercept keeps a batch that is merely dimmer than the run - an attenuated beam, a mis-fitted frame scale - out of the damage number: a batch mis-scaled by 2x read +12 A^2 of "damage" without it and +0.05 with it. A batch whose shells are too weak to fit is now absent from the curve, printed as "-", instead of pinned to the clamp. And the clamp itself is now an argument of the solve rather than one shared constant: it guards against divergence, and the correction keeps the bound it was tuned with, but with the estimator fixed a heavily dosed crystal's honest relative-B runs past it - the monitor pinned thirteen consecutive batches at +49 A^2, which is the same defect in the other direction. The monitor is given room a real relative-B cannot reach and drops any batch that lands on it anyway. The shells are laid inside the range the run actually diffracted to, not across the whole merged range: a resolution limit taken from another program or left generous spends most of an equal-occupancy grid on noise and leaves a batch with too few shells to fit at all - on the battery that silenced three crystals outright and cost two of them nineteen batches of thirty-six. Where the merged range already sits inside the signal the grid is unchanged and so is every number. The first->last headline is reported only where a straight line explains at least half of the curve's variance, or where the curve is flat to within a couple of A^2 and the answer is simply "no damage"; otherwise there is no headline and the report says the loss was not dose and points at the sweep-quality section. The three shapes separate cleanly - progressive damage R^2 0.97, the obstructed sweep 0.24, the clean control flat at +0.35 A^2. And the label now follows the sign: damage fades the high-resolution intensity, so only a positive change is dose, where before any |dB| > 5 was called damage. Report-only throughout - the monitor never touches corr, and the per-batch curve's only consumer beyond the report is a sweep-quality field no reader reads; classification runs on the per-frame scale and CC, and is unmoved. The decay correction's global slope and the opt-in per-batch relative-B share this estimator and are left alone here: they fold into the scale, so Full 38-crystal rotation battery, twice (the second confirming the shell placement), against a clean baseline at the same base: space groups unchanged at 35/38 merge metrics move on three crystals only - the same three whose two-pass lattice search takes a different branch on nearly every arm run this session, one of which moves its own R_meas by 1.5 points on thread count alone A report-only change ought to be bit-identical and this is not quite, which is worth saying plainly: the three crystals that move are the known unstable ones and no space group moves, but "identical except where nothing is ever identical" is a weaker statement than "identical", and the residue has not been chased to ground. The three validation cases behave as they must: 60 deg beam-obstructed wedge, no decay -23.02, labelled damage, twelve batches printing the clamp -> NOT_A_TREND, curve within 3 A^2, the two unmeasurable batches absent, and a pointer to the sweep-quality section genuine progressive damage -39.46, sign inverted -> +94.50, monotone, corroborated by a per-image CC that falls 0.608 -> 0.159 and never recovers clean control +0.18 -> +0.76, flat within 1 A^2 Across the battery the report now names four crystals as radiation-damaged instead of ten; the other three are the two lowest-energy datasets and the pink-beam one, each showing a monotone rise of about ten square Angstroms. Sweep-quality classification is untouched, and the coupling that was assumed to exist does not: rad_damage_b_batch reaches it through one field that is written and never read. Ranges and reasons are identical on 35 of 38, the three that differ by one to seven frames are the same unstable crystals, and the census of stretches called radiation damage is one before and one after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-12 08:13:06 +02:00
leonarski_fandClaude Opus 5	2222a4085c	Scaling: correct the absorption that changes as the crystal turns ... RefineAbsorption indexes its surface by the diffracted direction de-rotated into the crystal frame, deliberately without a time axis, and RefineModulation indexes its by detector position, also without one. Nothing is indexed by (rotation, detector position), so the part of the absorption that changes as the crystal turns has no parameter at all. For a rigid absorber illuminating a fixed volume that is the right model: the incident path is a function of the spindle angle alone and the per-image scale takes it, and the exit path is then fixed in the crystal frame. What breaks the factorisation is the diffracting volume moving - a crystal larger than the beam, a mis-centred loop, ice building up. The exit path then depends on the spindle angle as well as the direction, and no time-independent surface reaches it. Measured on 34 rotation datasets, on XDS's own uncorrected intensities, as what is left after the crystal-frame absorption and detector modulation surfaces have taken what they can. The cross-validation gate lets the surface engage on 22 of them. Scored per resolution shell - the gate's whole-range ratio is lowered by any resolution-dependent scale without a reflection getting tighter, so the honest readout is each shell's own ratio, which a per-shell scale leaves unchanged - the median engaged crystal gains 4.1 %, the set gains 117 % summed against 19 % of damage, and 3 of the 22 are hurt. The surface has to be smooth in rotation angle to be absorption at all, and it is: the lag-1 autocorrelation of the fitted factor along the time axis runs +0.32 to +0.71 on the crystals it engages, against -0.08 for the same surface with its time bins shuffled. Where it is not smooth it is fitting something else, and says so - on a sweep whose beam was obstructed for a 70 deg wedge the autocorrelation is +0.16 and the profile is a cliff at the wedge, not a turn. Two null controls. Assign every observation a random cell and the gate refuses it (-1.2 % to -3.8 %). Keep the detector bin and shuffle only the time bin - a surface that cannot contain any time-dependent information - and the gate refuses that too, at +0.04 %, -0.60 % and +0.33 % on three crystals. Against the real surface's +3.7 % to +14.8 % on the same three. 12 time bins x a 10 x 10 detector grid = 1200 factors. On the per-shell score the median gain moves only between 3.1 % and 4.1 % across grids from 216 to 2400 cells, so the grid is second order; 12 x 10 has the largest net and the fewest crystals hurt. Equal-occupancy detector bins, not equal width: an equal-width grid starves the edges and the corners, and a starved cell is where a free surface over-fits. Fitted last, so the two time-independent surfaces get first claim on what they can explain. QUALIFICATION, measured after this was written: the "33 better / 0 worse" above is overall R_meas, which is a ratio of sums across every shell and is therefore lowered by any resolution-dependent scale without a reflection getting tighter - the same property that let the estimator bias pass its own gate. Scored per resolution shell instead, this surface HURTS 6 of 18 crystals under the acceptance gate as it currently stands, because that gate shares the defect and admits the surface where it should not. With a per-shell gate the surface is refused on exactly those crystals and its net over the chain goes from +86.3 to +159.5 per cent with none worse. The correction is right; the gate that decides where to apply it is the next commit's problem, not this one's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery against its own matched baseline - the same binary with the corrected estimator and without this surface: R_meas better 33 / worse 0, -78.7 R_meas_lo better 24 / worse 2, -20.7 CC1/2 better 8 / worse 0, +9.6 ISa better 30 / worse 2, +99.15 space groups unchanged at 35/38 The low-energy datasets gain most, which is what absorption should do: at 5-6 keV one crystal goes ISa 24.68 -> 37.42 and another 14.21 -> 22.08, while the same protein measured at 13 keV moves 13.41 -> 14.83. Taken with the estimator fix it precedes, against a clean baseline: R_meas_lo better 25 / worse 3 summed -26.0, ISa better 31 / worse 2 summed +113.1, outer-shell CC1/2 +83.0, observations +25 880 on 36 crystals of 38, CC1/2 flat at -1.4 and no space group moved. That last number is the point of the pair: the estimator fix alone costs CC1/2 -11.4, because the ramp it removes was partly standing in for this correction. One cost, predicted in advance and still unexplained: outer-shell CC1/2 falls on three of the four low-energy crystals, by 15.8 points on the worst, while every other statistic on those same crystals improves. The fourth goes up. On 5000-9000 That outer-shell fall has since been attributed, and it is not this surface: with the merge's 6-sigma outlier rejection turned off, the sign flips on every crystal that lost, +20.5 and +20.7 where it read -15.8 and -8.0. The surface removes most of the deviants in sample - it is fitted on all the data and applied to it, with no robustness of its own - so the merge's cut stops firing and the survivors land in a shell whose multiplicity is about three. Last-shell CC1/2 is largely made by that cut: one crystal's baseline goes 10.4 to 92.7 purely by dropping 19 per cent of the shell. A second qualification, measured after the numbers above were taken: overall R_meas is a ratio of sums across every shell, so any resolution-dependent scale lowers it without a reflection getting tighter - the same property that let the estimator bias pass its own gate. Scored per resolution shell instead, this surface hurts 6 of 18 crystals under the acceptance gate as it stands, because that gate shares the defect and admits the surface where it should not. Under a per-shell gate it is refused on exactly those crystals and its net over the correction chain goes from +86.3 to +159.5 per cent with none worse. The correction is right; where to apply it is the gate's problem. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-12 05:34:55 +02:00
leonarski_fandClaude Opus 5	e4ce573075	Bragg integration: keep a reflection that lost a wing, not one that lost its peak ... MINPK asks how MUCH of the expected profile is readable. It does not ask WHERE, and the two are not the same question. The renormalisation argument the rescue rests on - a fit over a subset of a normalised profile is unbiased - needs the pixels to go missing for reasons unrelated to the reflection. A gap, a mask or the edge of the sensor is such a reason: the loss is set by the detector, and the fit renormalises over what is left. A pixel invalidated BY THE FLUX IT SAW is not: it goes missing because the reflection was bright, and it is the peak. Measured on the combined fulls, against the mean of the complete observations of the same reflection, in the innermost resolution shell of the high-multiplicity control and of a weaker crystal: a rescued reflection whose unreadable pixel sits within a pixel of the predicted centre reads |I - <I>|/I of 0.50 and 0.53, against 0.073 and 0.212 for a complete observation - 6.8x and 2.5x - and carries several times the mean intensity of its shell. On the control that is 0.21% of the shell's observations supplying 1.77% of the R_meas numerator; on the weaker crystal 0.52% supplying 6.82%. Rescues that lost only rim pixels are unremarkable by the same measure, 1.19x and 0.88x. Dropping the peak-losers alone takes the shell's R_meas from 7.440% back to 7.315% (unrescued: 7.307%) and from 22.03% to 21.06% (unrescued: 21.28%) - which is the whole of the low-resolution R_meas the rescue cost, and on the second crystal rather more. Raw frames say what they are. The pattern is a dead-centre invalid pixel with 5878, 9875 and 27583 counts around it: the detector's per-frame invalid marker on the brightest reflections. MINPK cannot catch them because it cuts on profile MASS, and the peak of a broad spot is a few percent of the mass. So a second condition, in the loop that already measures the readable fraction: no unreadable pixel may carry more than 0.9 of the profile's own peak value. A fraction of the peak rather than a radius in pixels because the peak is as wide as the spot - for a Gaussian the cut is at sqrt(-2 ln f) sigma, 0.46 sigma here, which is the peak pixel alone where sigma is 0.8 px and the crest of the ridge where the profile is a bandwidth streak. Swept against the alternatives on two crystals: a fixed radius needs 1.0-1.5 px to do the same work and costs 3-9x more observations for it, and 0.5 px does not reach the peak of a sub-pixel-offset prediction at all; tightening the fraction to 0.5 or 0.2 buys nothing beyond 0.9 and costs 7x more. Six crystals, three detectors, against the rescue as it stands: the rule keeps 99.86-99.96% of the recovered observations and returns R_meas to its unrescued value or below (4.6 -> 4.5%, 6.7 -> 6.6%, 25.1 -> 25.0%), R_meas in the innermost shell likewise (2.7 -> 2.6%, 5.9 -> 5.3% against 5.4% unrescued, 16.5 -> 16.4%), <I/sigma> up or level everywhere, and every unique reflection the rescue won is kept. Raising --overlap-minpk to 0.90 instead reaches the same place on two of them and short of it on the third, while discarding 0.8% of the recovered observations rather than 0.05%. An elongated pink-beam profile on a 9M detector and an EIGER2 16M dataset are both untouched at 99.9%, so the crest protection does not over-reject a streak. One crystal is not improved: a dataset whose error model rugnux declines to fit for want of strong reflections, whose <I/sigma> is <= 0 in eight of its ten shells and whose R_meas is undefined in as many. There the rule costs about 3% of <I/sigma> in the one shell that has signal, reproducibly, on top of the 9% the rescue itself costs there - while its overall R_meas moves 1.5 points on nothing but the thread count. The parity test gains four sections. Unreadable pixels were only ever punched into empty sky, so neither the rescue nor this rule had any CPU/GPU coverage at all; they now go into the signal disks - the peak of every fifth reflection, ~1.1 sigma out of every seventh, the disk edge of every eleventh - for both profile modes, a box sum and an elongated stencil, with a check that the clipping actually costs reflections so the coverage cannot go quietly vacuous. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery against the rescue without this rule, both on the same base: ISa better 19 / worse 4, +0.73 CC1/2 better 3 / worse 1, +1.3 R_meas_lo better 4 / worse 3, -0.3 space groups unchanged for 17 770 observations, 0.09 % of the run total and under 2 % of what the rescue had won. The two crystals whose peak-loss population was measured beforehand land on their predicted values: a tetragonal reference goes R_meas_lo 2.7 -> 2.6 % and ISa 27.11 -> 27.42, a cubic insulin 5.9 -> 5.3 % and 20.34 -> 20.65. One crystal pays: a cubic case with 2381 unique reflections goes R_meas 8.8 -> 9.6 % and ISa 4.08 -> 3.49. It is the crystal in the battery with the fewest uniques, so its rescued population is small and its shell statistics are coarse, but the loss is real and not noise in the R_meas. The R_meas sum over the battery reads +1.3, of which +3.2 is one crystal whose R_meas moves 1.5 points on thread count alone; without it the sum is negative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-12 03:41:21 +02:00
leonarski_f	12ae6c5228	Bragg integration: fit a reflection over the pixels it has, not only over all of them ... A predicted reflection was discarded outright if ANY pixel of its signal disk was unreadable - masked, untrusted, in a detector gap, or overloaded. On a battery crystal that is 11.1% of all predictions, thrown away for a defect in one pixel of fifty, and the pixels concerned sit at fixed places on the detector, so the loss is systematic in reciprocal space rather than random. Neither XDS nor dials does that. Both estimate the missing part from the profile instead and keep the reflection while enough of it was seen: XDS's MINPK (default 75%, "the missing intensity is estimated from the learned profiles"), dials' integration.profile.valid_foreground_threshold (default 0.75). MOSFLM is the one program that rejects by default, and even it relaxes to 50% with PROFILE EDGE. We already had the argument and the machinery: a profile fit is the amplitude of a NORMALISED profile, so leaving pixels out renormalises the estimator by construction - it costs information, which sum P^2/v duly loses and sigma duly gains, and biases nothing. That is exactly why --overlap exclude drops a neighbour's pixels from the fit rather than the reflection. Unreadable pixels are the same case with a different reason, so they take the same treatment, cut on the same threshold, in the same place: the readable fraction of the expected profile, measured against the profile mass that lands on the detector at all so a reflection is judged on the pixels that exist. A box sum has no profile to renormalise with and keeps the all-or-nothing rule. Two consequences handled. The summation seed and its variance now count the pixels actually read, and the runaway guard scales the fit back to that same disk before comparing - both exactly as before wherever nothing is missing. (Its fallback then hands back that partial sum unrescaled, which would read low; the guard fires on 8 of 96 688 recovered reflections, and on none at all on a weak crystal, so it is not worth a branch.) And the profile, its resolution shells and their widths are learned from COMPLETE reflections only, as is the box-sum centroid post-refinement reads as an observed position: a disk with a hole gives a centroid pulled away from the hole, and the hole does not move between frames. That sigma gains what the missing pixels carried is the claim the whole change rests on, and it is measurable. Force the conventional CENTRED cell of a body-centred crystal in P1: the predictor then enumerates every lattice point, and the reflections the centring makes systematically absent have a true intensity of exactly zero, so their scatter about zero must equal their reported sigma. Over 7.1 M such observations, matched by resolution shell, the trimmed std(I)/rms(sigma) of the recovered reflections is 0.99 / 1.20 / 2.33 / 1.04 / 1.69 against 0.98 / 1.22 / 2.29 / 1.03 / 1.56 for the reflections that were complete - the same calibration to a few percent. The lever there is small, because the typical recovered reflection is missing only 5% of its disk. Lowering the threshold to 0.50 admits a band missing 25-50%, which is a real lever: there sigma comes out 8-43% larger than a complete reflection's in the same shell, and the scatter about zero tracks it, 0.97 / 1.09 / 1.92 / 0.99 / 1.37, at or below the complete population. Sigma grows, and by the amount it should. The threshold stays at XDS's and dials' 0.75, on that evidence and on quality. Below it the estimator starts to run out: on those same zero-intensity reflections the recovered ones read +0.8 counts high at 0.75 and +1.9 counts high in the 0.50-0.75 band, against a sigma of 12-17, and at 0.25 the fit degenerates outright, single reflections carrying sigma in the thousands. Above it there is nothing to buy: 0.90 leaves a fifth of the recoverable observations behind and measures no better for them. On the high-multiplicity control, R_rim over as-shipped / 0.90 / 0.75 / 0.50 runs 4.49% / 4.51% / 4.56% / 4.78% while <I/sigma> runs 33.47 / 34.02 / 33.89 / 33.43 - 0.50 is where the recovered observations stop paying for themselves. Probe against the previous commit, six crystals. The high-multiplicity control gains 4.2% more observations, 924 803 -> 963 946, which lands it on XDS's 961 379 from the same images, for <I/sigma> 33.47 -> 33.89, R_rim 4.49% -> 4.56% at 4.3% more multiplicity, CC1/2 unchanged at 0.9998 and ISa 27.80 -> 27.12. Five weaker crystals gain 3.3-4.8% of their observations and up to 1.0 point of completeness, for <I/sigma> +0.4 to +3.6%, R_rim between -8.1% and +5.8% relative, CC1/2 +6.6 / +0.3 / +0.2 / -0.0 / -1.2 points, and ISa between +0.3% and -3.4%. Some of that ISa is the point rather than the price: a reflection integrated over fewer pixels carries less information, and the absence test above says the sigma that reports so is honest. The GPU and CPU engines agree as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery, against the same binary without it: observations better 38 / worse 0, +937 100 unique refl better 30 / worse 0, +9 229 overall <I/sig> better 33 / worse 1, +7.00 CC1/2 better 5 / worse 1, +6.2 space groups unchanged at 35/38 Every crystal gains observations and not one loses a unique reflection. The two costs are small and both are understood. Low-resolution R_meas is worse on eight crystals, by +0.8 pp at most and +3.2 pp summed - a reflection whose own peak pixel is unreadable loses the part of the profile that carries most of the amplitude, and that population sits at low resolution; the following commit handles it. And ISa falls on 32 crystals, by 10.9 summed, which is what admitting 937 000 further observations does to the strong-reflection asymptote: R_meas excluding the one crystal whose thread-count noise is 1.5 pp is flat.	2026-08-12 02:03:42 +02:00
leonarski_fandClaude Opus 5	43627e22dc	docs: a rule for licences and academic credit, and apply it ... Several methods adopted recently came from other crystallographic packages - the screw-absence test from POINTLESS, MINPK and the profile-fit reweighting from XDS/Otwinowski, the CC1/2 cutoff and merge outlier rejection from DIALS, the per-frame indexing gate from CrystFEL - and nothing in the repository said where such a debt is recorded. The licence side was already worked out (licences beside the vendored code, verbatim texts in licenses/ collected by COLLECT.sh, a row in THIRD_PARTY_NOTICES.md, all installed under share/doc/jfjoch); the credit side was ad hoc. Write the rule into CLAUDE.md. It states the distinction that matters: vendoring or linking someone's CODE creates a LICENCE obligation, discharged in licenses/ and THIRD_PARTY_NOTICES.md; reimplementing an algorithm from a PAPER creates none of that but creates an obligation of academic CREDIT, discharged in docs/ACKNOWLEDGEMENT.md and in a comment at the algorithm. Neither substitutes for the other, and taking both source and paper incurs both. It also fixes the citation form (authors, title, year, journal, volume, pages, verified DOI), and says in-source credit goes at the algorithm, not the file header, in the one-line style the code already uses. Then bring the repository into compliance for the works concerned: docs/ACKNOWLEDGEMENT.md gains a section acknowledging XDS, DIALS, POINTLESS/CCP4, MOSFLM, CrystFEL, GEMMI, the Kabsch/Otwinowski profile fit, the Diederichs & Karplus statistics and the IUCr nomenclature reports, each with a DOI checked against Crossref; docs/CPU_DATA_ANALYSIS.md's reference list gains the ones it was missing; and four algorithms gain a line naming their source where no adjacent comment carried one. No licence change. licenses/ and THIRD_PARTY_NOTICES.md are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 16:34:48 +02:00
leonarski_fandClaude Opus 5	06a5a118bf	Bragg integration: widen the background ring to r3 = 13 ... The background is estimated from the r2..r3 ring and then subtracted from every pixel of the r1 disk, so the ring mean's own error enters the intensity n_inner times over: var(I) carries n_inner^2 * bkg / n_B. That term is first-order in sigma, and it is set by how many pixels the ring holds - not by anything about the reflection. At r3 = 10 the ring holds about 200 px against the disk's 50. Widening it to 13 roughly doubles that. The signal disk is untouched, and the pixels gained lie further from the reflection rather than nearer, so nothing is traded for them. The effect is not subtle once looked for. Matched observation by observation on one crystal, halving the ring's pixel count leaves the intensity alone and inflates sigma by 4.7%, and the inflation rank-orders with the ring collapse across the battery. Over the whole rotation battery, against the same binary at r3 = 10: ISa better on 14 crystals and worse on 4, the summed shortfall against the reference 164.7 -> 155.9, the summed low-resolution R_meas excess 69.7 -> 59.1 percentage points, and one more crystal reaching the reference space group (33/37 -> 34/37, a trigonal case that was over-promoting). Largest gains where the ring was starved worst; the four losses are 0.25 to 2.16 in ISa and none of them changes a space group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 00:07:30 +02:00
leonarski_fandClaude Opus 5	adf87e8675	Merging: export the XDS-comparable ISa under jfjoch_diffrn_ISa ... The mmCIF's _reflns.jfjoch_diffrn_ISa carried the strong-reflection asymptote, a tier XDS has no equivalent of, while the name invites comparison with XDS's ISa - which is the whole-range 1/sqrt(a*b). rugnux_vs_xds.py reads that item for the battery's ISa column, so the comparison that column exists to make was between two different quantities, flattering rugnux by the difference between the tiers. Write the whole-range value there, move the asymptote to _reflns.jfjoch_diffrn_ISa_asymptotic, and add _reflns.jfjoch_error_model_a and _b in XDS's convention so the number can be re-derived from the file rather than taken on trust. On a broadband rotation dataset the battery column now reads 13.25 against XDS's 21.18 where it read 15.6 before, and the two error models can be compared term by term for the first time: a 1.538 vs 1.249 and b 3.71e-03 vs 1.78e-03, so the gap is in BOTH the counting and the systematic term (1.23x and 2.08x, and sqrt(1.23*2.08) = 1.60 = 21.18/13.25). This is a deliberate redefinition of an exported item, not an addition: a file written by an earlier version carries the asymptote under the old name and there is no version marker to tell them apart. Noted in the changelog and in docs/CPU_DATA_ANALYSIS.md. Nothing reads the item back into the pipeline - it is written and never parsed by rugnux itself - so no stored file is reinterpreted in a way that changes a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:55:46 +02:00
leonarski_fandClaude Opus 5	ac06b5c64f	Merging: report the error model in XDS's convention ... rugnux fits sigma^2 = a*sigma0^2 + (b*<I>)^2, so its `b` is a fraction of the intensity. XDS fits sigma^2 = a*(sigma0^2 + b*I^2) and prints ISa = 1/sqrt(a*b). The two `a` are the same number, but the two `b` are not - b_xds = b^2/a - so the pair rugnux printed could not be read against a CORRECT.LP, which is the only reason anyone looks at it. Convert at the report. The fit, the merge weights and both engines' variance expressions are untouched, so this is a re-expression and not a change: on a rotation dataset the merged intensities move strictly less between before and after than they do between two runs of the SAME binary (99.9% identical, max |dI/I| 9.1e-4 against the run-to-run control's 7.5e-3), with the same reflection set. The rotation path also printed the wrong ISa for the comparison it invites. What it calls ISa is the strong-reflection asymptote, a tier XDS has no equivalent of and which can only ever be the more optimistic of the two; XDS's ISa is the whole-range 1/sqrt(a*b), which in rugnux units is exactly 1/b. Print both, labelled. On a broadband rotation dataset that is 13.2 (whole range) and 15.6 (asymptote) against XDS's 21.18 - so the number previously compared was flattering rugnux by 2.4. A third, unrelated `b` lives in the space-group search: fitted with the sigma^2 coefficient held at 1, with gate constants calibrated in that convention, and a ratio bound does not survive the mapping (1.90 would have to become 3.61) while the absolute floor has no correct value at all, there being no `a`. It is now commented as such, since making the three consistent is the obvious wrong move. Also corrects three comments and two doc passages that still described a merged-sigma systematic floor deleted in 72efb75a8. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 18:50:22 +02:00
leonarski_fandClaude Opus 5	61d24db59f	Bragg integration: elongate the background ring per reflection ... The signal disk and the r2..r3 background ring were fixed pixel circles, identical for every reflection at every resolution. A reflection is not round: a finite bandwidth streaks it radially by bw_sigma*Rpx, so at high resolution the ring sits within 1.3-2.2 sigma of the reflection's own profile and measures its tails as background. --integration-stencil <k> makes the RING an ellipse, elongated along the beam->reflection direction by k times that streak, capped at 2*r3. The tangential half-widths stay r2 and r3, and the r1 signal disk stays a circle: r1 drives the all-or-nothing n_inner_valid == n_inner gate, so growing it rejects any reflection carrying one bad pixel along a long streak, and the flux a circular r1 loses is a function of resolution alone, which the per-shell scale absorbs. The geometry lives in one shared header compiled by both the host compiler and nvcc, so the seven pixel-classification sites - the CPU mask/main/clip loops and the GPU mark_mask/main/trim/clip kernels - cannot drift apart. Rather than evaluate an ellipse, each pixel's squared distance has its radial part scaled down, d2 - q*rad^2 against r2^2/r3^2 with q = 1 - (r/(r+grow))^2, so grow = 0 gives q = 0 and both tests collapse onto d2 exactly in floating point. The width is the bandwidth streak alone, not the profile's full radial variance, which also carries the sensor parallax and weak-spot capture terms. Deriving the growth from those was implemented first and measured on the rotation battery: at k=1 it took Thau_9's high-shell CC1/2 from 75.8 to 27.9 and Benas_3's from 14.1 to 6.0, against cytC_10 +1.2 and lyso_ref flat. On a monochromatic beam they are the only terms there are, and C_CAPTURE is 64% of them. Keeping only the streak also makes the option exactly inert without a bandwidth, rather than merely small. Default 0. Measured on broadband rotation data with the bandwidth set to its spectroscopic value, matched resolution limits: high-shell CC1/2 30.6 -> 46.4 at k=4, and better in EVERY shell in both CC1/2 and R_meas (top shell R_meas 194.7% -> 138.7%), with completeness, multiplicity and space group unchanged and 28 of 98833 unique reflections lost. Anomalous peak height over 18 sites +0.107 +- 0.039 sigma (p = 0.013). The full 38-crystal rotation battery is unchanged to every reported digit, base against k=3. Two consequences of an elongated ring are handled rather than inherited. The neighbour exclusion marks the inner ELLIPSE in each neighbour's own frame, or an elongated neighbour leaks its tails into this reflection's ring. And the radial-background curvature kernel becomes a small table indexed by the growth, because its azimuthal average makes one kernel serve every reflection only while their stencils are identical; the GPU's radial window, previously a fixed 32 bins, is now sized on the host from the widest ring on the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 15:19:28 +02:00
leonarski_fandClaude Opus 5	1239c49731	Bragg integration: separate the three things a bandwidth used to switch ... Setting a bandwidth flipped three unrelated switches at once: it changed the profile's radial capture term, it moved the width measurement from the signal disk to the whole fit grid, and it silently overrode the background clip and trim, so --background-clip under --bandwidth was ignored - the two runs were bit-identical. The width measurement was the damaging one. The fit grid is an azimuthally averaged stack, so its second moment is sigma_r^2 + sigma_t^2 and the radial smear of a bandwidth leaked into the tangential model - a tangential width of 3.04 px against a 1.06 px truth, inflating the effective background pixel count where the weak signal is. The result was a step rather than a slope: on genuinely monochromatic data, declaring a 0.2% bandwidth cost ISa 28.4 -> 22.2. Measure the two widths separately, accumulated in each spot's own radial/tangential frame over the signal disk, from the signed profile cells - away from the peak a learned cell is background noise centred on zero, so the signed sum is unbiased, while clamping it at zero turns that noise into a pedestal the r^2 weight reads as width. The radial term is then the measured excess or the analytic floor, whichever is larger. With the two widths separated there is nothing left for the broadband switch to select, so it is gone - which is the proof the three were independent. The background clip and trim now come from the settings in every case; the tuned 3-sigma broadband default moves to the rugnux front end, which is the only place that knows whether the user gave a value. Monochromatic data: declaring a 0.2% bandwidth now costs ISa 28.4 -> 27.9 rather than 22.2, and forcing the old 3-sigma clip in the new build reproduces the good result, so none of the step came from the clip. On large-bandwidth data CC1/2 improves in 8 of 10 shells. Across 12 monochromatic crystals the space groups are unchanged and CC1/2 moves by at most 0.2 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 21:08:29 +02:00
leonarski_fandClaude Opus 5	5830f78d57	Revert the azimuthal-integration sigma clip ... Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping machinery in AzIntEngine. This is a partial revert of a6be35ccd - the ice-ring-mask removal that commit also carried stays. Sigma clipping remains where it started and where it is needed: inside the adaptive spot finder, at a fixed 3 sigma on raw counts, feeding the detection threshold and the ice score. The option made the workflow harder to reason about than the quantity was worth. It gave azimuthal integration two meanings behind one setting - the bin mean and the background under the peaks - which the azimuthal-integration workflows do not need. It also did not compose with the fused GPU engine, which supplies the profile from its PLAIN pass: on the default rugnux, viewer and receiver path the setting was silently doing nothing (measured, the profile came out identical to the unclipped run to 1e-6 with identical per-bin pixel counts). Making it correct is not a matter of gating that one shortcut - it means separating the workflows (azimuthal integration, MX rotation, MX stills, geometry calibration) and deciding per workflow what the profile is for, which is a larger change than the option earns. The default path is unaffected: over 20 images of a rotation dataset the radial profile, the per-bin pixel counts and the spot counts are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-08 15:31:19 +02:00
leonarski_fandClaude Opus 5	6468dd13be	rugnux: --mode, and detector calibration from powder rings ... --azint-only and --scale are replaced by --mode mx|azint|scale|calibration, with mx the default. The old flags are removed rather than aliased. Calibration mode fits the detector geometry - PONI x/y, the two tilts and the distance - to a calibrant's powder rings and writes a pyFAI .poni alongside a report of how far each parameter moved from the header. Bragg data constrain the beam centre worst, because it is gauge-coupled to the crystal orientation; a powder ring has no orientation to couple to. --calibrant takes lab6, agbh, ceo2, si or ice. A calibrant is a list of ring positions rather than a unit cell, because hexagonal ice is P6_3/mmc: rings enumerated from its cell would include systematically absent ones. So the crystalline standards generate their rings from a cell and ice carries the measured list, and RingsFromAzimuthalProfile, GuessGeometry and OptimizeGeometry all take ring q. The calibrant table is shared with the viewer's powder panel, which previously carried its own copy. --calibration picks how the rings are measured: rings (default) sums the (q x azimuth) profile over every processed image and fits the arcs in it; spots pools the found spots and fits those. Both use the whole run, with -s/-e/-t selecting images. rings defaults --azim-phi-bins to 32, since a profile with one azimuthal bin has averaged the ring over every direction and cannot locate it. Two fixes this exposed: The extraction window is capped at half the gap to the neighbouring ring. The background under a peak is taken from the ends of its window, so a window wider than half that gap measures the next ring's flank as this ring's background - and hexagonal ice has three rings within 0.06 1/A. Ice calibration was 3.5 px out before this and 0.29 px after; LaB6 is unaffected. RingOptimizer holds rot1/rot2 fixed when only one ring is present. A tilt and a centre offset both move a ring as cos(phi) and are separated only by the tilt's amplitude growing as the ring radius squared, so on a single ring they are exactly degenerate. Measured. LaB6 at five distances: the fitted direct beam is within 0.36 px of an independent implementation out to 300 mm, and D = -0.046 + 1.000788 dtz with an rms of 0.011 mm. At 500 mm one ring is fully on the detector and a second only clips the corners, which is not enough to constrain a tilt - restricting the q range to the resolved ring recovers 0.06 px. Ice: 5.53 -> 0.29 px on one crystal and 4.71 -> 0.80 px on another, against XDS's refined direct beam. On an ice-free crystal the fit is worse than the header, which is the correct outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 10:00:03 +02:00
leonarski_fandClaude Opus 5	d81d2e4696	docs: state the metric-symmetry rule rather than how it was arrived at ... CPU_DATA_ANALYSIS describes how the pipeline works; the account of which bar was tried first belongs in the commit that changed it. Same facts, no narrative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 09:30:10 +02:00
leonarski_fandClaude Opus 5	cfd3697ddb	docs: cover the azimuthal sigma clip, the indexing flag, and the metric-symmetry check ... Three things had reached the code without reaching the documentation. The azimuthal sigma clip had a RUGNUX.md row and a CPU_DATA_ANALYSIS section but no changelog entry - and the only "sigma clip" the changelog mentioned was the background ring's, which is a different thing at a different stage. --index-ice-rings was in the options table but nowhere in the changelog, so the entry describing the ice gate still implied that whether indexing uses the ice-band spots is decided per run, which it no longer is. CPU_DATA_ANALYSIS section 6 still described the Bravais class as simply "the highest-symmetry class that matches within tolerances", which is the behaviour that lost a crystal outright. It now records that the class is chosen from the UNREFINED candidate against a fixed angular tolerance, that a pseudo-symmetric lattice therefore gets promoted a class too far, and that the first pass settles it on validation-frame counts with a clear-majority bar - including why the bar is a majority rather than a margin, since that distinction is the whole reason the check is safe. It also records that the first pass finds its own spots rather than reading the acquisition's, which was not written down anywhere. Also a build note: M_PI is not standard C++ and MSVC does not define it, so the Bragg integrator's use of it broke the Windows viewer build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:56:08 +02:00
leonarski_fandClaude Opus 5	1df9556ec1	Bragg integration: default the radial background correction off again ... Auto rode in with the ice work rather than on its own evidence, and measured over the 37-crystal rotation battery it does not carry itself yet. It TARGETS correctly - it fires on ten crystals and every one is ice-positive, no failures, no space-group changes - but it costs 1.35x the wall clock (median +3 s per crystal, worst +29 s) and on the merge statistics it is the familiar sign-mixed trade: high-shell CC1/2 worse on three of the four crystals that move materially, mean -0.76. The case for it is real but rests on agreement with a fixed external model - 43 % of the ice bands' excess amplitude removed on smooth ice, the effect 7x stronger inside the bands than outside - which is the better arbiter and also the narrower one. That deserves settling on its own, not riding along with a set of ice defaults. `--background-radial=auto` keeps it a flag away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:29:42 +02:00
leonarski_fandClaude Opus 5	a6be35ccdb	Azimuthal integration: optional sigma clipping of the reported profile ... The profile is the MEAN of each bin, so a few strong reflections landing in a bin lift it exactly as a smooth powder ring does. That is the wrong quantity whenever the profile is wanted as a background rather than as a measurement of what is in the bin - the ice score being the case in point, where reading a plain profile INVERTED the metric: over 37 rotation crystals the two highest-scoring crystals had no ice at all. The adaptive spot finder already computes the right thing, a sigma-clipped per-resolution-ring background, as a byproduct of its own threshold. Where it runs, the ice score uses that. Where it does not - --no-adaptive-spots, --azint-only, and anything reading the profile the broker wrote - there was no way to get it. This adds one: azim_int_settings.sigma_clip (rugnux --azim-sigma-clip), 0 = off, minimum 2 because a tighter clip rejects a large part of a clean Gaussian bin and biases the estimate low rather than removing outliers. Two clip passes follow the plain one, matching the finder's recipe - the first pass's standard deviation is itself inflated by the peaks being removed, so one pass leaves a threshold that is still too generous. A bin with fewer than eight pixels is left alone: at the detector edge and behind the beam stop there is no spread to clip on. Both engines do it. On the GPU the accept range is computed by a small kernel and stays resident, so a clip pass is one more read of the same pixels and no round trip; the two accumulation kernels take the range as a pointer that is null on the plain pass. Measured on a JUNGFRAU rotation dataset, non-adaptive path: azimuthal integration 0.02 -> 0.06 ms per image, exactly the 3x the extra passes predict, against a 0.34 ms per-image total. Note what the result IS: the smooth background under the peaks, not the bin mean. It should not be switched on where a ring's integrated intensity is wanted - the powder-ring geometry fit reads ring peaks, and those are what a clip is designed to remove. Off by default, so nothing changes unless it is asked for. Not exposed over the REST API - that needs the generated model regenerated, which is a separate step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-07 00:18:52 +02:00
leonarski_fandClaude Opus 5	3d4209d803	docs: the first-pass ice measurement, the promotion fix, and powder-ring geometry ... Changelog entries for the three changes above, and a new CPU_DATA_ANALYSIS section on determining detector geometry from powder rings: why a ring is an independent constraint on the beam centre (it has no crystal orientation to be gauge-coupled to, unlike everything else that fits geometry here), what a ring can and cannot determine, and how the ring points are obtained. The section states the harmonics correctly, which is worth writing down because the intuitive version is wrong: a detector tilt shows up as cos(phi), the same harmonic as a beam-centre error, and the two are separated by the amplitude growing as the ring radius SQUARED - so it takes at least two rings, and on one ring they are exactly degenerate. The genuine cos(2 phi) term is hundredths of a pixel. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 22:12:21 +02:00
leonarski_fandClaude Opus 5	f0cdb027e1	Ice: default the merge mask off, gate the radial background on smooth ice, and pick detection by geometry ... Three defaults, each settled by measurement rather than by argument. The arbiter throughout is structure-referenced - anomalous peak height where a crystal can carry it, and otherwise the agreement of the ice bands with a fixed external model against resolution-matched DECOY bands carrying no ice. The band-versus-decoy contrast is used because R-free here tracks completeness, and every one of these switches moves completeness. The damage is real and it localizes: over the rotation battery the ice bands' excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst, while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the ten largest excess peaks in a q scan land on hexagonal ring positions. Turning ice handling off leaves the contrast unchanged and forcing it on a clean crystal does not create one, so it is the ice and not the machinery. MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band reflections in the merge and exclude them only from the model fit; autoPROC is the sole exception. On the one battery crystal where the mask fires and an anomalous arbiter can score it, dropping the band moved the mean peak height at the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing 1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own 3.05 - better than average data - and costing 17 completeness points in that shell. It fires on 5 of 37 crystals, changes no space group, and those 5 disagree in sign: it clearly helps the two most heavily iced, is a wash on two and costs a third. So it stays as a switch, worth setting by hand on a badly iced crystal where it shows in the high shell, but it is not a default. RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background as a function of radius alone, and that is exactly when it works. On a crystal with pure smooth powder ice it removes 43% of the bands' excess amplitude, with the improvement 7x larger inside the bands than outside; on a crystal whose ice is discrete crystallite spots - no smooth ring to model - the excess amplitude GREW by half; on clean data it is inert to four decimals. The two ice channels already separate those morphologies, so --background-radial takes on|off|auto and auto applies it to an image when that image's peak-excluded score reaches --ice-min-score. Auto never engages without such a score, because the plain profile carries the Bragg peaks and cannot support an absolute threshold. Per image rather than per run, and that was tested rather than assumed: the gate fires on 100% and 94% of frames on the two crystals that want it, and on 1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice crystal. A seam statistic against off + f*(on - off) is null on both mixed runs, every merge statistic is bracketed by the pure arms, and the textured crystal's auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need the score before the pass that integrates, i.e. rotation-only plumbing, and buys nothing measurable. The kernel was already built unconditionally, so flipping the flag per image is free - except on the GPU, where the launches were gated on a construction-time n_rad. That is why the buffers are now allocated whenever the correction could run, and Run() decides per image. DETECTION -> the geometry's default when the file is silent: on for rotation, off for stills, with the command line and then the file taking precedence. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent systematic and the presence gate keeps it inert on a clean crystal; a serial stills run has too few spots per image to spend any on flagging. The master file's key is kept as written rather than collapsed to a bool, so "the file said nothing" is distinguishable from "the file said no" - it used to fall silently to off, taking the exclusion from the scale fit with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 19:06:30 +02:00
leonarski_fandClaude Opus 5	61a7c91b90	Ice: detect it on two channels, and only handle it when it is there ... The per-image ice score was read off the PLAIN azimuthal profile. That profile is a per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it exactly as ice would. Measured over 37 rotation crystals, that did not merely add noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all (4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its 99th percentile and 2.70 at its maximum, so that metric cannot support any absolute threshold whatsoever. The adaptive spot finder already computes the right input for its own threshold: a sigma-clipped per-resolution-ring background, in the same bins. A powder ring is azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at 2.08-2.37, against a decoy null that never exceeds 1.29. That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE spots and leaves the radial profile flat. So a second channel counts found spots on the rings against the same q width of ice-free flanks beside them. The two barely overlap - the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6, while a clean crystal reads 1.04 on both. Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now all skipped when neither channel sees any. The gate is applied in the full pipeline and in --scale, which reads the stored per-image values back out of the _process.h5. Also fixes the merge-time mask's control: the shoulder now excludes reflections that are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on the neighbours and the test compared ice against ice. Measured, that is the only thing this changes: it removes firings on those three rings and leaves every other firing's CC pair identical to three decimals. And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the same data got a narrower band online than the measured ~0.06 ring FWHM justifies. Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses. Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3 and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa 3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The one crystal that loses reflections improves on both R_meas and CC1/2. Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5 write/read and the receiver plots are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 16:17:23 +02:00
leonarski_fandClaude Opus 5	09fb8e0306	Bragg integration: clip the background ring high side instead of trimming it ... The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A symmetric trim is not a consistent estimator of the mean of a right-skewed (Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true mean at every level, and with ~50 signal pixels in the r1 disk that under-subtraction adds ~5 counts to every partial on every frame. Measured two independent ways on four rotation datasets - stored background_mean against a plain ring mean over the same pixels on reflection-free frames, and directly on apertures that provably hold no reflection. Empty-aperture pedestal, counts: plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for monochromatic data. It rejects the same one-sided contamination the trim was there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds ~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The trim stays reachable with --background-trim for back compatibility; setting either estimator clears the other, so they can never stack. --integrator boxsum does not take the clip (matching what the shipped clip already did), so it now uses the plain ring mean unless --background-trim is given. The intensities get measurably more accurate: per-shell agreement with an independent processing of the same images improves on 14 of 16 crystals (weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR - absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and CC1/2 in the outer shell improves on 14/7. EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by information-free counts, so both get worse when the bias is removed; neither is evidence against this change. That fingerprint is exactly how the trimmed mean was accepted in the first place. Known cost: over the 37-crystal rotation battery the de-novo space-group count goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal whose merge fails the absolute-sanity gate under either background (R_meas 63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the very operator being scored, so its operator CC is diluted by construction. No other crystal changes space group, and twin protection is not weakened - the H-ratio veto that refuses genuinely twinned crystals gets MORE decisive (1.63 -> 1.84, 2.83 -> 3.99). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 18:03:11 +02:00
leonarski_fandClaude Opus 5	fb55645b81	Revert "rugnux: fit the profile radius from the strongest spots too" ... Reverts the profile-radius part of 457b1bfd1; the comparison-script and mosaicity-column changes from that commit are kept. The cap was validated on the rotation battery, which cannot test it: the profile radius feeds `ewald_dist_cutoff` in IndexAndRefine, and that is read only by the STILLS predictors (BraggPrediction/BraggPredictionGPU). The rotation predictors gate on the mosaicity window instead and never look at it. So "no space-group changes, 36 of 37 crystals bit-identical" showed the quantity is inert for rotation, not that capping it is safe - and the one regime where it does act was never exercised. Validating it needs the serial-stills battery, which is a much larger exercise. Until then the arbitrary constant is not worth carrying in a code path nobody measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 21:35:17 +02:00
leonarski_fandClaude Opus 5	5eb386e333	docs: describe the per-frame geometry smoothing ... Goes in §10.3 next to the per-frame scale and mosaicity smoothing, since it is the same mechanism applied for the same reason, and trims the changelog line to one sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 20:23:19 +02:00
leonarski_fandClaude Opus 5	457b1bfd1d	rugnux: fit the profile radius from the strongest spots too ... Same defect as the mosaicity in 2c94f3013, in the same file's sibling fit. The profile radius is an RMS of the excitation error over whatever spots were kept, and weaker spots sit further off the Ewald sphere, so it grows with the depth of the list: measured over a 150 -> unlimited spot budget it rises ~60%, and on a clean dataset as much as on a hard one, so this is general rather than something one awkward crystal provoked. That made it a function of --max-spots, which is an indexing budget, rather than of the crystal. Its one consumer treats it as a membership gate (ewald_dist_cutoff is twice it) where reflections at the cutoff carry near-zero partiality and are removed downstream anyway, so the integrated data barely notices: with the mosaicity already pinned, the partial count moves 0.2% across a 34% change in the radius. It is also reported per image as a diagnostic, though, and a number that slides with an unrelated setting is misleading to anyone comparing two runs - and it would stop being benign the moment anything used it as a width rather than a gate. Battery over 37 crystals: no space group changes, 36 of 37 bit-identical, no failures, one crystal marginally better. Also in the comparison script: report XDS's mosaicity next to rugnux's. XDS has two and they are not interchangeable - CORRECT.LP's REFLECTING_RANGE_E.S.D. is post-refined, while INTEGRATE.LP's per-batch SIGMAR is its integration-stage estimate, and the two differ by up to 2.3x. The XDS cell now prints both as postrefined|MLE so a per-image estimate is compared against the one measured the same way. Fixes a scoping bug in the same addition where every crystal read the last directory's INTEGRATE.LP. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 10:52:30 +02:00
leonarski_fandClaude Opus 5	2c94f3013e	rugnux: fit the mosaicity from the strongest spots only ... The per-image mosaicity MLE ran over the whole indexed spot list, so it rode on --max-spots, which is an indexing budget. A spot is detected when I_full * R(tau) clears the finder threshold, so selecting by intensity censors on R(tau): a deeper list holds proportionally more large-|tau| partially recorded spots and the fit widens with it. Raising the budget 250 -> 1000 widened sigma_M 0.059 -> 0.075 deg on a rotation dataset whose measured rocking width says 0.054. That is not cosmetic. An over-wide mosaicity mis-states every partiality in scaling: forcing the mosaicity across that range moved the merge error model from b 0.039 / ISa 26 to b 0.167 / ISa 6, and the space-group search lost a genuine 422 with it, merging the crystal in 222 instead. Cap the fit at the strongest 250 spots. FilterSpotsByCount leaves the list strongest-first, so this selects exactly the spots a smaller --max-spots would, and the mosaicity becomes invariant: 0.0538 deg at 250, 500, 1000 and 2000 spots, with the correct space group at each. Trimming or down-weighting the tau tail does not work - the censoring is multiplicative in R(tau), so it widens the whole distribution rather than adding a tail. Battery over 37 crystals: exactly one change, the demoted crystal repaired (33 space groups matching XDS -> 34). 23 of 37 are bit-identical, never reaching 250 spots. Unaffected elsewhere: the default spot count is 250, and stills have no goniometer so they return before the fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-04 07:34:40 +02:00
leonarski_fandClaude Opus 5	bec7e2e922	image_preprocessing: fuse the bitshuffle inverse with preprocessing, and verify the decode ... The device decoder was byte-exact on every valid input - 994 production-compressed images, 927 hand-built LZ4 blocks covering engineered (offset, matchlen) pairs across the overlap branch boundary, 18000 repeat decodes, sanitizer-clean - and an audit against LZ4_decompress_generic could not construct a valid block it mis-decodes. What it did not do was notice when the input was NOT valid, and that mattered more than it looks: the decode buffers are reused frame to frame, so a block that stopped early left the PREVIOUS image in place, and in the bitshuffled layout the untouched tail is the most significant byte-plane. A corrupt chunk therefore did not look like a missing corner. It looked like thousands of real pixels several powers of two too bright, fed to spot finding with no diagnostic, where the host decoder had raised an error. So the kernel now flags a block that fails to reach its declared length while consuming exactly its payload, and the host turns that into an exception once the caller has synchronised. Reads are clamped against the end of the payload as well as the output, both length chains are bounded exactly as read_variable_length bounds them, the two offset bytes are bounded, and LZ4's parsing restrictions are enforced. On the host side a block size that is not a multiple of 8 elements is rejected (it made the un-transpose read uninitialised shared memory), the block count is bounded by what the chunk could hold before it becomes an allocation (twelve header bytes could demand hundreds of MB of pinned memory, permanently, per worker), trailing bytes are rejected, and the stream is synchronised before any throw that happens after work is queued. An image of fewer than 8 elements is all verbatim tail and now decodes rather than throwing. When the device route fails for any reason the host decoder gets its turn, so it costs speed rather than the acquisition. The lanes cooperate on the copies and a later match can read bytes another lane wrote, which since Volta needs an explicit __syncwarp(); it worked only because ptxas happened to reconverge at the post-dominator. The prototype's offset == 1 and power-of-two fast paths are also restored - the shipped kernel ran a runtime modulo, an emulated 32-bit division per output byte, on the path its own comment calls the common case. The un-transpose is now fused with preprocessing. One thread owns one group of 8 elements across every byte-plane, so once it has transposed its 8 bytes out of each plane it holds 8 complete elements and emits 8 finished int32 pixels with the mask, the error marker, the saturation cap and the statistics applied. The decompressed image is never materialised: 0.623 -> 0.411 ms/frame at 18 Mpx, 0.523 -> 0.340 with 8 concurrent workers. Staging nothing in shared memory also drops the 48 kB ceiling, which had made any file whose bitshuffle blocks exceed it a hard failure; 64 kB blocks now decode. gpu_compressed is sized from the chunk with grow-on-demand instead of from the uncompressed size - it was reserving ~73 MB per worker to hold ~4 MB. Measured on a 1630x1553 uint32 rotation set at -N 32, peak GPU memory falls 3756 -> 3084 MiB; the same model gives ~144 MB per worker on an 18 Mpx frame. Decoding on the device also stopped reporting a decompression time, which blanked the broker's compression plot trace and filled /entry/profiling/compressionTime with NaN. The decoder brackets the decode with CUDA events and reports it again. Tests: a differential fuzz suite against the CPU decoder - incompressible and highly compressible data, engineered offsets, a size sweep hitting every rem%8 value twice, all six element sizes, an 18 Mpx frame, decoder reuse, concurrency, hand-built LZ4 blocks across the overlap boundary, 26 foreign bitshuffle block sizes from 128 B to 64 kB, corrupt payloads and malformed containers, with a coverage report that proves which LZ4 paths were reached rather than assuming it. Plus the fused path held byte for byte against ImagePreprocessorCPU, statistics included, and against the host-upload path on the same frame. Battery: 37 crystals, every merged number identical to the host-decode run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:13:55 +02:00
leonarski_fandClaude Opus 5	737cbde3ff	docs: tighten the changelog and the decoding section for release ... The rc.161 changelog had grown entries several hundred words long and listed the same area three or four times over. Collapse them by subject - spot finding, resolution limits, space-group search, scaling, performance, correctness - and hold each to one line, keeping the actionable detail in the breaking API entry. Add the performance work that had not been written up: device-side image decoding and the parallel first-pass candidate-cell refinement. Section 0 of the CPU analysis document was the longest thing in it after two core algorithm sections, and most of that was a profiling narrative and the measurements that motivated the change rather than a description of what runs. Cut it to the two kernels, the host-side block scan and the fallback rule. The attribution stays; it is also in the reference list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 07:44:55 +02:00
leonarski_fandClaude Opus 5	6e4c0ce202	image_preprocessing: decode bitshuffle+LZ4 on the GPU ... The pipeline decompressed each image on the host and uploaded the result. On an 18 Mpx rotation dataset that made the host-to-device copy the bottleneck of the whole per-image loop: nsys puts the copies at 78% of the loop against 39% for every kernel combined - 3600 transfers of 72.4 MB - and they ran at only 12.5 GB/s of an available 27-28 because the host-side decompression was itself saturating host memory bandwidth. The GPU was mostly waiting. So the compressed chunk goes across instead, about 4 MB rather than 72 MB, and is decoded on the device. That removes the transfer and the host decompression that was throttling it, in one change. Measured on an idle machine, a run goes from 45.11 s to 24.97 s - 1.81x - with the merged output unchanged. THE APPROACH IS JON WRIGHT'S (ESRF): "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group 2021, and github.com/jonwright/ bslz4decoders. The kernels here are ours, but the idea and the demonstration that it is worth doing are his. Cited in docs/ACKNOWLEDGEMENT.md and in the new section 0 of docs/CPU_DATA_ANALYSIS.md. Two kernels mirror the CPU decoder. LZ4 runs one WARP per bitshuffle block: every lane parses the same sequence stream (a broadcast read, no divergence) and the literal and match copies are split across the 32 lanes so the stores coalesce; an overlapping match is treated as a pattern of period offset sourced from bytes that already precede the write position, which keeps it parallel rather than a serial byte loop. One thread per block instead measured 13x slower. The bitshuffle inverse then un-transposes each byte-plane through shared memory and interleaves the planes back into elements. Only BSHUF_LZ4 is decoded on the device. The zstd variants have no device decoder, and neither has an uncompressed or float image; Supports() returns false for those and the caller decompresses on the host exactly as before. The fallback is explicit, so a format we cannot decode on the device is a slower path and never a wrong answer. Tests hold the device decoder against the CPU one byte for byte, on data from the production compressor, for every element size the detectors emit - including the 8-bit DECTRIS modes, which take bitshuf_decode_block's separate elem_size == 1 branch - plus a many-block frame, the formats it must decline, and malformed containers, which must throw rather than run off a buffer. Battery: 37 crystals, no failures, identical to the host-decode run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 00:16:36 +02:00
leonarski_fandClaude Opus 5	90ab67f376	broker: expose adaptive spot finding over the API ... The self-calibrating detection threshold was reachable from rugnux and the viewer but not online: spot_finding_settings carried no adaptive_threshold, so the receiver always ran the fixed-threshold finder and the fused GPU engine sat unused behind it. adaptive_threshold and false_pixels_per_frame are now part of the schema, both optional so an existing client that sends neither is unaffected, wired through OpenAPIConvert in both directions and surfaced in the frontend panel, where turning the mode on greys out the count threshold it replaces and reveals the operating point it uses instead. The C++ server model and the TypeScript client are regenerated from the spec; the Python client is generated but not tracked. Enabling it is REFUSED where spots are found on the FPGA - the JUNGFRAU and EIGER workflows - rather than accepted and ignored, because a detection setting that silently had no effect cannot be told apart from one that did. The DECTRIS/SIMPLON workflow, which analyses images in software, accepts it. Verified against a running broker: adaptive_threshold true is rejected with that message and leaves the stored settings untouched, while false and omitted both succeed. It stays off by default online, unlike rugnux and the viewer. The broker serves both workflows and the default has to be the one that works on either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 20:01:34 +02:00
leonarski_fandClaude Opus 5	e7be5447d3	receiver: stop copying every frame back from the device on the Lite path ... The Lite workflow built its analysis with the fused GPU engine disabled, which is also what decides whether the preprocessed image is copied device-to-host after every frame. So on a machine with a GPU the online path was moving the whole image back - 72 MB on a large detector, every frame, per worker - for a host reader that does not exist on that path. It was left off deliberately when the fused engine was added, to keep the online path unchanged in that commit, and never revisited. Nothing depends on it: the FPGA workflow uses a different analysis class, and strong-pixel values are read through a device gather rather than from the host image. Turning it on changes no result, and cannot: adaptive detection is unreachable online, because the REST schema exposes no way to enable it, so the classic GPU finder runs either way. Measured anyway, both engines on the same frames across five datasets including very weak ones: 2400 frames, 638260 spots, not one difference - identical lists, identical indexing rate, identical merge statistics to every printed digit. On a large detector with eight workers the median per-image cost falls from 94 to 59 ms and preprocessing from 21 to 6 ms; throughput rises from about 48 to 55 Hz. No percentile regresses, which is what matters for a service - the ninetieth improves from 128 to 74 ms and the tail with it. Spot finding gets faster too, because the large copy no longer contends with the device gather. Correct two statements while here. The flag's comment and the data-analysis document both said the online receiver uses the CPU adaptive finder; online never runs an adaptive finder at all, and the copy the flag really controls was not mentioned. That copy would be better expressed as what it is - whether a host engine will read the image, which the constructor already knows - rather than inferred from which spot finder is wanted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 15:16:18 +02:00
leonarski_fandClaude Opus 5	4bdb229fb8	spot_finding: find connected components on the GPU ... The spot finder flagged strong pixels on the device and then labelled them on the host, so every frame sent the packed bitmask back - 2.26 MB on a large detector - and the host walked all of it to recover a few hundred pixels. Do the labelling on the device instead: compact the bitmask into a flat-index-sorted list, find each pixel's backward neighbours by binary search, union them lock-free with path halving, then label, accumulate and filter in one kernel. Only the spot list comes back, and only one stream synchronisation per frame. The gain in the ordinary case is modest - about a quarter off per-image spot finding - because the host algorithm is genuinely fast on a normal frame. What justifies it is the frame that is not ordinary. The host labels a sorted sparse list through a window spanning two detector lines, so its cost is quadratic in how many strong pixels share a line. A lit band of detector rows - a hot module, a panel edge - costs 33 ms at two rows and 377 ms at fifteen, all of it under the pixel cap that was supposed to bound this, and none of it maskable when the cause is a diffraction ring rather than a defect: a ring runs tangent to a row at its top and bottom, which is exactly the shape that hurts. The device version is flat at 0.05 to 0.64 ms across every geometry tried, so an online run no longer stalls a quarter of a second on an ice ring. Rejecting an over-cap frame is now free too, since the count is known before any pixel is written. Also label once and filter three times. The per-image minimum-pixel search runs the extraction at three settings, but that setting only decides which components are kept - it does not change the components - so the search itself need not be repeated. This helps the host path as much as the device one. The resolution mask moves to the device as a bit mask, uploaded when the limits change rather than per frame, since the compaction needs it there. Parity is asserted permanently rather than argued: five cases covering realistic frames, occupancy from a hundred pixels to past the cap, the pathological geometries including rings, the resolution mask, and a hundred-repeat determinism check - requiring the same partition, the same spot order, and identical counts. The centroid is a float sum and therefore order-dependent, so the device walks each component from its root in ascending order and fuses its multiply-add the way the host's does; note that whether the host fuses at all depends on the architecture flags, so exact centroid equality is asserted where the compiler fuses and a two-ulp bound otherwise. Making those accumulators integer would remove that dependence entirely and is worth doing separately. Regression set: all 37 crystals identical to the last printed digit. Unit suite passes with the new cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 13:34:43 +02:00
leonarski_fandClaude Opus 5	1a1e05ad14	spot_finding: run the same two passes on the CPU as on the GPU ... ImageSpotFinderGPU::Detect launches its kernel twice, feeding the first pass's strong-pixel bitmap back in so the second recomputes each local background with those pixels excluded and keeps them strong. The CPU finder ran a single pass, so the two returned different spot lists for the same frame and a dataset processed without a GPU did not match one processed with it. It matters for any spot wide enough to reach into its own 31x31 background box: the spot inflates the mean and variance it is then tested against, so its outer pixels fail the SNR test. On the test image added here - a 5x5 core at 300 counts with a one-pixel ring at 25 - a single pass returns the 25-pixel core and 7500 counts where two passes return the full 49 pixels and 8100. pxl_val also becomes int64_t, matching the GPU's pixel_result signature. It was int32_t, so pxl_val * pxl_val overflowed above 46341 counts even though the surrounding sums were already 64-bit. The new parity test compares PixelCount and Count, not just the centroid, which does not move for a symmetric spot whether or not the ring was picked up; it was confirmed to fail against the old single-pass CPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 15:37:22 +02:00
leonarski_fandClaude Opus 5	a3feb1271c	docs: CPU_DATA_ANALYSIS describes the algorithms, not their history ... The document had accumulated development narrative: what was tried and rejected, which datasets a change rescued or cost, measured percentages from test batteries. That belongs in commit messages and reports, not in a reference description of the pipeline - it dates quickly, and a reader looking up what an algorithm does has to sort it out from how it came to be. Removed throughout, keeping the algorithmic content and the design reasoning that explains a choice on principle: * 3.2 the whole paragraph justifying the rotation spot-finding limit from battery measurements, and the CPU-vs-GPU per-frame timings; * 3.3 "a significance/z-score was considered but is uninformative"; * 7.4 / 7.5 the comparisons to a robust loss and to joint refinement as approaches that had failed; * 9.2 the R_meas / CC1/2 outcomes attributed to the trimmed-mean background; * 9.3 "per-detector-region and crystal-anisotropy profiles were evaluated and add nothing"; * 10.2 the stills tilt "succeeds where a freely-fitted width collapses"; * 10.5 the CC_anom argument, trimmed to why the statistic behaves as it does; * 10.6 the survey of per-frame correlation medians across datasets; * 13 the space-group bullet, restructured into the three gates it actually applies, dropping the dataset anecdotes; * 14.2 the free-form per-shell rescale, stated as a design choice rather than an experiment. Section 13's space-group text was one 20-line paragraph; it is now a numbered list of the three tests, which is what the code does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:11:33 +02:00
leonarski_fandClaude Opus 5	0f1851cf82	rugnux: drop the 1.5 A spot-finding limit on rotation data ... Rotation kept a 1.5 A high-resolution limit for spot finding on the strength of one indexing-rate measurement (100.0 -> 96.8% on a strong set). Measured properly, over the whole 33-crystal rotation battery, it does not earn its place: * no space-group decision changes - the same 30/33 agree with XDS, and the three that differ are the known pre-existing cases, unchanged; * 29 of 33 crystals are identical to the digit - same indexing rate, R_meas, CC1/2, ISa. The limit was doing nothing on the large majority; * where it does bite, the limit is the worse setting. The one crystal that loses appreciable indexing rate without it (99.50 -> 94.22%) comes back with lower R_meas (29.4 -> 28.1), higher high-resolution CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses 0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames indexed, better data from them; * runtime is unchanged (16m46s vs 17m32s over the battery). So the indexing-rate cost is real but does not carry through to the merged data, which is what the limit was protecting. Unset now means "as far as the detector reaches" for rotation as well as stills; --spot-high-resolution still sets a limit for weak, high-background data where the extra high-resolution spots are genuinely noise. This also removes the flag that distinguished "the user asked for no limit" from "the user said nothing" - with no rotation default left, both mean the same thing. While rewriting the comment block, corrects its neighbouring claim that rotation keeps the fixed-threshold finder; adaptive detection has been the default for both workflows since 6f4917dce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 10:39:23 +02:00
leonarski_fandClaude Opus 5	323c76abde	docs: correct CPU_DATA_ANALYSIS against the code it describes ... Ten places where the document and the implementation had drifted apart. Each was checked against the source before rewriting: * 7.5 rotation post-refinement: it is TWO separate cross-validated steps (cell+axis from the angles, then distance+beam from the positions with the cell fixed), not one joint fit against the merged fulls; the held-out split is an hkl hash, not a frame split; the bounds are +-5% on distance and +-15 px on the beam, not "under ~1%"; and only the distance and beam centre reach pass 2, which re-indexes de novo. * 9.2 the trimmed-mean background: it is computed in the shared background pass, so it DOES apply to --integrator boxsum. Only the broadband sigma-clip is excluded. The section previously said both, contradicting itself two paragraphs apart. * 9.3 per-reflection profile rebuild, sub-pixel centring and radial elongation are gaussian-only; the empirical profile keeps the fixed per-shell grid and is accumulated on rounded predicted positions, not centroids. * 10.5 the asymptotic ISa and the b_ISa sigma floor are rotation-only; stills report 1/b and floor with the whole-range b. * 8.4 centering absences are applied only when the user fixes the space group - de novo, prediction runs in P so the search can confirm the centering from the intensities. * 10.6 per-batch relative-B cross-validates on ASU-group parity, not the frame parity the other surfaces use. * 13 the resolution cutoff sits one reported-shell width PAST the CC1/2 = 0.30 crossing, so data below 0.30 are kept. * 7.1 the orientation-only prior penalises all three components of the angle-axis vector. * 10.1 rlp is the RECIPROCAL Lorentz factor, L = 1/rlp. * 14.3 the model scaling is fitted over work and free reflections alike, so R-free is free of refinement, not of the scaling fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:41:32 +02:00
leonarski_fandClaude Opus 5	40acf1ffc4	rugnux: search the space group twice by default on rotation data ... --search-min-zeta now defaults to 0.85 for rotation, so the de-novo search runs on a merge of all the observations AND on a merge of only the well-measured ones, and keeps whichever found more symmetry. Previously it shipped off and the second opinion had to be asked for. Rotation battery, 33 crystals, NO flags beyond the resolution limit: fixed-threshold finder 30/33 - zero crystals differ from the single search adaptive finder 30/33 - the same three mismatches Both arms now agree crystal for crystal, which they have not done before. The last disagreement was a thaumatin set whose 4-fold sits 88.9 deg from the spindle: at defaults it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken) where it read P222. The classic arm is a strict no-op - zero differences against both the explicitly-flagged run and the run predating the dual search - so the default costs nothing where the geometry is not the problem, and 47.2 s against 47.8 s on the same crystal back to back. The default is safe to set because the two searches can only disagree by a LOST operator: discarding observations starves an operator correlation, it cannot invent one. That also makes the 0.85 itself uncritical - too aggressive a cut only means the second opinion contributes nothing and the full merge wins. --search-min-zeta 0 restores the single search. Docs: CHANGELOG gains a 1.0.0-rc.161 section covering the branch, and CPU_DATA_ANALYSIS records the four analysis changes of this work - the confidence-weighted per-image refinement, the collapsed per-frame scale guard, the opt-in per-image rejection, and the operator-disagreement criterion with the two-search rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:56:36 +02:00
leonarski_fandClaude Opus 5	6f4917dcee	rugnux: adaptive spot detection is the default for rotation data too ... It was held back because a 33-crystal rotation battery showed it breaking three crystals deterministically - a lost space group, a halved indexing rate and a collapsed merge. None of those causes turned out to be in detection. The extra spots adaptive finds are real. Measured per spot against a finder-neutral local background: 64% recur at the same position on the adjacent frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a conventional local SNR >= 4 test, median local SNR 34. What they include is genuine peaks belonging to no lattice the indexer found, and the damage they did scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed finder), not with their quality - which is why nothing aimed at judging individual spots ever worked. The three failures fell to fixes elsewhere: merge collapsed - a per-frame scale free to collapse toward zero amplified two junk frames by 546x (704098712, ec7a82613). Not a detection problem at all: the fixed-threshold finder trips the same bug on a different frame range. space group lost - the per-image geometry refinement was dragged 2.3-2.8 deg off by the weak-spot tail in an unweighted fit; weighting each spot by how strong it is FOR ITS RESOLUTION fixed it (ae126c3d5), and gained a point group for the fixed finder too. indexing halved - gone with the same two; that crystal is now better under adaptive (CC1/2 92.6 -> 95.9, high-shell 44.9 -> 56.4). Battery, 33 crystals, adaptive vs the fixed finder: exact space group matching XDS 26/33 vs 25/33 point group matching XDS 29/33 vs 30/33 ISa better on 6 crystals recovers a screw axis the other misses (P321 -> P3121) The one point group it loses is a tetragonal crystal where adaptive collects 2.4x the observations at better R_meas (21.4% vs 28.3%) and better ISa (4.58 vs 2.91), and forced to the right group gives CC1/2 99.2% at multiplicity 10.7 - matching XDS. Only the automatic symmetry call fails there, and six candidate causes have been measured and refuted (mixed indexing hands, off-crystal frames, a badly integrated minority, radiation damage, pseudo-tetragonality, uncorrected anisotropy). It is left as the subgroup, which is the recoverable direction: -S gives XDS-quality data from the same run, whereas the failures this unblocks were not recoverable. --no-adaptive-spots reverts to the fixed-threshold finder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:59:37 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8	a8c1006c49	Choose min-pix-per-spot adaptively per image for serial-stills indexing ... For stills indexing the minimum-pixels-per-spot filter is now chosen per image instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total) is kept, then integrated once at that min-pix. The fraction factor keeps a smaller min-pix's extra spots only when the lattice actually explains them, so strong frames retain their real weak spots (extending resolution) while noise-flooded frames stay strict. The mode is selected by the presence of --min-pix-per-spot, now optional (SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the adaptive per-image path, give a value to force a fixed min-pix. It applies only to the stills indexing path -- rotation indexing builds one global lattice and keeps a fixed min-pix, and the online receiver and the FPGA host path always carry a concrete value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame indexed, to drive the per-image selection. Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and adaptive-min-pix checkboxes, each greying out the control it overrides); the broker uses neither. Validated on the full rotation regression battery (no regression) and the whole serial-stills target battery at full image count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 18:31:35 +02:00
leonarski_fandClaude Opus 4.8	9fdeed282a	Add fused GPU adaptive spot finder (azint + spot finding in one pass) ... AdaptiveSpotFinderGPU does the per-resolution-ring reduction once on the GPU and drives both products from it: the azimuthal-integration profile (corrected space) and the self-calibrating adaptive spot-detection threshold (raw counts). This replaces the separate GPU azint pass and the host-side adaptive spot finder that runs on the GPU path today. On a ~4.5 MP detector it does both jobs in ~1 ms/frame versus ~40 ms for the CPU adaptive finder (~42x), with an identical spot list and azimuthal profile. The per-ring threshold math (Poisson tail + read-floored Gaussian, operating point from the false-pixels-per-frame knob) is factored into AdaptiveThreshold.h so the CPU and GPU finders share one source of truth and cannot drift. Wired opt-in via a MXAnalysisWithoutFPGA constructor flag, default on for the rugnux offline path and the interactive viewer, off for the online receiver (so the broker path is unchanged). When on, Analyze() skips the separate azint pass and lifts the profile from the fused engine. The viewer gains an "Adaptive threshold" checkbox that greys out the signal/noise and photon-count sliders (the adaptive finder uses neither). Dedicated tests exercise both products (spot-finding parity vs the CPU finder, azimuthal profile vs a standalone GPU azint) plus a speed benchmark. Validated end-to-end on lysozyme serial stills: fused == CPU-adaptive index rate and merge stats. Docs: new section 3.2 in docs/CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 20:10:45 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	2ab9deb20b	v1.0.0-rc.123 (#40) ... Documentation update Reviewed-on: #40	2026-03-12 12:17:11 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-05 22:13:12 +01:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	c44a07aac2	v1.0.0-rc.90	2025-10-02 15:55:39 +02:00