41 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	e4ce573075	Bragg integration: keep a reflection that lost a wing, not one that lost its peak ... MINPK asks how MUCH of the expected profile is readable. It does not ask WHERE, and the two are not the same question. The renormalisation argument the rescue rests on - a fit over a subset of a normalised profile is unbiased - needs the pixels to go missing for reasons unrelated to the reflection. A gap, a mask or the edge of the sensor is such a reason: the loss is set by the detector, and the fit renormalises over what is left. A pixel invalidated BY THE FLUX IT SAW is not: it goes missing because the reflection was bright, and it is the peak. Measured on the combined fulls, against the mean of the complete observations of the same reflection, in the innermost resolution shell of the high-multiplicity control and of a weaker crystal: a rescued reflection whose unreadable pixel sits within a pixel of the predicted centre reads |I - <I>|/I of 0.50 and 0.53, against 0.073 and 0.212 for a complete observation - 6.8x and 2.5x - and carries several times the mean intensity of its shell. On the control that is 0.21% of the shell's observations supplying 1.77% of the R_meas numerator; on the weaker crystal 0.52% supplying 6.82%. Rescues that lost only rim pixels are unremarkable by the same measure, 1.19x and 0.88x. Dropping the peak-losers alone takes the shell's R_meas from 7.440% back to 7.315% (unrescued: 7.307%) and from 22.03% to 21.06% (unrescued: 21.28%) - which is the whole of the low-resolution R_meas the rescue cost, and on the second crystal rather more. Raw frames say what they are. The pattern is a dead-centre invalid pixel with 5878, 9875 and 27583 counts around it: the detector's per-frame invalid marker on the brightest reflections. MINPK cannot catch them because it cuts on profile MASS, and the peak of a broad spot is a few percent of the mass. So a second condition, in the loop that already measures the readable fraction: no unreadable pixel may carry more than 0.9 of the profile's own peak value. A fraction of the peak rather than a radius in pixels because the peak is as wide as the spot - for a Gaussian the cut is at sqrt(-2 ln f) sigma, 0.46 sigma here, which is the peak pixel alone where sigma is 0.8 px and the crest of the ridge where the profile is a bandwidth streak. Swept against the alternatives on two crystals: a fixed radius needs 1.0-1.5 px to do the same work and costs 3-9x more observations for it, and 0.5 px does not reach the peak of a sub-pixel-offset prediction at all; tightening the fraction to 0.5 or 0.2 buys nothing beyond 0.9 and costs 7x more. Six crystals, three detectors, against the rescue as it stands: the rule keeps 99.86-99.96% of the recovered observations and returns R_meas to its unrescued value or below (4.6 -> 4.5%, 6.7 -> 6.6%, 25.1 -> 25.0%), R_meas in the innermost shell likewise (2.7 -> 2.6%, 5.9 -> 5.3% against 5.4% unrescued, 16.5 -> 16.4%), <I/sigma> up or level everywhere, and every unique reflection the rescue won is kept. Raising --overlap-minpk to 0.90 instead reaches the same place on two of them and short of it on the third, while discarding 0.8% of the recovered observations rather than 0.05%. An elongated pink-beam profile on a 9M detector and an EIGER2 16M dataset are both untouched at 99.9%, so the crest protection does not over-reject a streak. One crystal is not improved: a dataset whose error model rugnux declines to fit for want of strong reflections, whose <I/sigma> is <= 0 in eight of its ten shells and whose R_meas is undefined in as many. There the rule costs about 3% of <I/sigma> in the one shell that has signal, reproducibly, on top of the 9% the rescue itself costs there - while its overall R_meas moves 1.5 points on nothing but the thread count. The parity test gains four sections. Unreadable pixels were only ever punched into empty sky, so neither the rescue nor this rule had any CPU/GPU coverage at all; they now go into the signal disks - the peak of every fifth reflection, ~1.1 sigma out of every seventh, the disk edge of every eleventh - for both profile modes, a box sum and an elongated stencil, with a check that the clipping actually costs reflections so the coverage cannot go quietly vacuous. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery against the rescue without this rule, both on the same base: ISa better 19 / worse 4, +0.73 CC1/2 better 3 / worse 1, +1.3 R_meas_lo better 4 / worse 3, -0.3 space groups unchanged for 17 770 observations, 0.09 % of the run total and under 2 % of what the rescue had won. The two crystals whose peak-loss population was measured beforehand land on their predicted values: a tetragonal reference goes R_meas_lo 2.7 -> 2.6 % and ISa 27.11 -> 27.42, a cubic insulin 5.9 -> 5.3 % and 20.34 -> 20.65. One crystal pays: a cubic case with 2381 unique reflections goes R_meas 8.8 -> 9.6 % and ISa 4.08 -> 3.49. It is the crystal in the battery with the fewest uniques, so its rescued population is small and its shell statistics are coarse, but the loss is real and not noise in the R_meas. The R_meas sum over the battery reads +1.3, of which +3.2 is one crystal whose R_meas moves 1.5 points on thread count alone; without it the sum is negative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-12 03:41:21 +02:00
leonarski_f	12ae6c5228	Bragg integration: fit a reflection over the pixels it has, not only over all of them ... A predicted reflection was discarded outright if ANY pixel of its signal disk was unreadable - masked, untrusted, in a detector gap, or overloaded. On a battery crystal that is 11.1% of all predictions, thrown away for a defect in one pixel of fifty, and the pixels concerned sit at fixed places on the detector, so the loss is systematic in reciprocal space rather than random. Neither XDS nor dials does that. Both estimate the missing part from the profile instead and keep the reflection while enough of it was seen: XDS's MINPK (default 75%, "the missing intensity is estimated from the learned profiles"), dials' integration.profile.valid_foreground_threshold (default 0.75). MOSFLM is the one program that rejects by default, and even it relaxes to 50% with PROFILE EDGE. We already had the argument and the machinery: a profile fit is the amplitude of a NORMALISED profile, so leaving pixels out renormalises the estimator by construction - it costs information, which sum P^2/v duly loses and sigma duly gains, and biases nothing. That is exactly why --overlap exclude drops a neighbour's pixels from the fit rather than the reflection. Unreadable pixels are the same case with a different reason, so they take the same treatment, cut on the same threshold, in the same place: the readable fraction of the expected profile, measured against the profile mass that lands on the detector at all so a reflection is judged on the pixels that exist. A box sum has no profile to renormalise with and keeps the all-or-nothing rule. Two consequences handled. The summation seed and its variance now count the pixels actually read, and the runaway guard scales the fit back to that same disk before comparing - both exactly as before wherever nothing is missing. (Its fallback then hands back that partial sum unrescaled, which would read low; the guard fires on 8 of 96 688 recovered reflections, and on none at all on a weak crystal, so it is not worth a branch.) And the profile, its resolution shells and their widths are learned from COMPLETE reflections only, as is the box-sum centroid post-refinement reads as an observed position: a disk with a hole gives a centroid pulled away from the hole, and the hole does not move between frames. That sigma gains what the missing pixels carried is the claim the whole change rests on, and it is measurable. Force the conventional CENTRED cell of a body-centred crystal in P1: the predictor then enumerates every lattice point, and the reflections the centring makes systematically absent have a true intensity of exactly zero, so their scatter about zero must equal their reported sigma. Over 7.1 M such observations, matched by resolution shell, the trimmed std(I)/rms(sigma) of the recovered reflections is 0.99 / 1.20 / 2.33 / 1.04 / 1.69 against 0.98 / 1.22 / 2.29 / 1.03 / 1.56 for the reflections that were complete - the same calibration to a few percent. The lever there is small, because the typical recovered reflection is missing only 5% of its disk. Lowering the threshold to 0.50 admits a band missing 25-50%, which is a real lever: there sigma comes out 8-43% larger than a complete reflection's in the same shell, and the scatter about zero tracks it, 0.97 / 1.09 / 1.92 / 0.99 / 1.37, at or below the complete population. Sigma grows, and by the amount it should. The threshold stays at XDS's and dials' 0.75, on that evidence and on quality. Below it the estimator starts to run out: on those same zero-intensity reflections the recovered ones read +0.8 counts high at 0.75 and +1.9 counts high in the 0.50-0.75 band, against a sigma of 12-17, and at 0.25 the fit degenerates outright, single reflections carrying sigma in the thousands. Above it there is nothing to buy: 0.90 leaves a fifth of the recoverable observations behind and measures no better for them. On the high-multiplicity control, R_rim over as-shipped / 0.90 / 0.75 / 0.50 runs 4.49% / 4.51% / 4.56% / 4.78% while <I/sigma> runs 33.47 / 34.02 / 33.89 / 33.43 - 0.50 is where the recovered observations stop paying for themselves. Probe against the previous commit, six crystals. The high-multiplicity control gains 4.2% more observations, 924 803 -> 963 946, which lands it on XDS's 961 379 from the same images, for <I/sigma> 33.47 -> 33.89, R_rim 4.49% -> 4.56% at 4.3% more multiplicity, CC1/2 unchanged at 0.9998 and ISa 27.80 -> 27.12. Five weaker crystals gain 3.3-4.8% of their observations and up to 1.0 point of completeness, for <I/sigma> +0.4 to +3.6%, R_rim between -8.1% and +5.8% relative, CC1/2 +6.6 / +0.3 / +0.2 / -0.0 / -1.2 points, and ISa between +0.3% and -3.4%. Some of that ISa is the point rather than the price: a reflection integrated over fewer pixels carries less information, and the absence test above says the sigma that reports so is honest. The GPU and CPU engines agree as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery, against the same binary without it: observations better 38 / worse 0, +937 100 unique refl better 30 / worse 0, +9 229 overall <I/sig> better 33 / worse 1, +7.00 CC1/2 better 5 / worse 1, +6.2 space groups unchanged at 35/38 Every crystal gains observations and not one loses a unique reflection. The two costs are small and both are understood. Low-resolution R_meas is worse on eight crystals, by +0.8 pp at most and +3.2 pp summed - a reflection whose own peak pixel is unreadable loses the part of the profile that carries most of the amplitude, and that population sits at low resolution; the following commit handles it. And ISa falls on 32 crystals, by 10.9 summed, which is what admitting 937 000 further observations does to the strong-reflection asymptote: R_meas excluding the one crystal whose thread-count noise is 1.5 pp is flat.	2026-08-12 02:03:42 +02:00
leonarski_fandClaude Opus 5	d8029524e7	Scaling: never let an observation's own fluctuation set its weight ... A weighted mean is only unbiased while the weights are independent of the values being averaged. The IUCr's own nomenclature report (Schwarzenbach et al., Acta Cryst A45 (1989) 63-75) puts it directly: weights in averaging "should not be based on the counting statistics of the individual observations whose estimated variances are biased and result in larger weights for accidentally low intensities". Two places in the rotation pipeline were doing exactly that, and between them they drove whole resolution shells of merged intensity negative. 1. The profile fit computed its non-signal variance as var_bkg = max(0, 1/den - max(0, I) + bkg-estimate term) The point of a separate var_bkg is that it does NOT move with the reflection's own fluctuation, and 1/den - I is the quantity that does not: 1/den is the fit variance taken at the fitted intensity and grows with it roughly one for one. Clamping the subtrahend at zero left a down-fluctuated reflection's own deflated variance standing as its background variance. Measured over 6.9 M partials of one weak rotation dataset, var_bkg/bkg came out at 3.7-5.4 for observations with I < 0 against 11.4-13.7 for I > 0 - the down-fluctuated half of every reflection carried a variance ~2.7x too small and was weighted up by the same factor, first in the 3D combine and then again in the merge. Removing the clamp makes var_bkg flat in I (~13 x bkg across the whole range). 2. The merge then weighted each combined full by 1/sigma_full^2, and sigma_full is by construction a function of the full's own answer: the combine's variance carries a corr*max(0, F) signal term, so every full with F <= 0 got the smallest variance the model allows while the strongest quartile got 2.26x more. The merge now rebuilds that variance at the reflection's mean instead, from a linear model var(I) = var_bkg + var_per_I * I that the combine measures and stores on the full. This mirrors MergeOnTheFly::CorrectedSigma, whose comment already claimed to mirror the rotation combine. Verified against an estimator that cannot see the fluctuation - summing the partials and dividing by the summed partiality, the classical construction every other program uses (Greenhough & Suddath, J. Appl. Cryst. 19 (1986) 400-409, via Leslie, Acta Cryst D55 (1999) 1696-1702: profile fitting biases the individual partials but not their sum). Reproducing the merge on dumped observations, the shipped weighting sat ~1.9 sigma below that reference in the noise shells; the two changes recover most of it, and every intensity-independent weighting scheme agrees with the reference once (1) is in. Four-crystal probe, XDS resolution limits, branch fingerprint identical on all four (so none of these is a two-pass branch flip): weak cubic case last shell <I/sig> -1.6 -> +0.2 (XDS +0.10), last shell R_meas 478% -> 250% (XDS 246%), overall <I/sig> 6.1 -> 7.5 (XDS 7.18), R_meas 18.3% -> 18.1%, CC1/2_hi 38.2% -> 43.7% tetragonal case outer shells <I/sig> -0.4/-0.8/-0.9/-1.0 -> +1.8/+1.2/ +0.9/+0.4, R_meas 184%/595%/7614%/nan -> 95%/119%/135%/232% (the nan was the shell mean crossing zero), R_meas 33.3% -> 32.9%, CC1/2_hi 38.3% -> 56.5% trigonal case R_meas 13.0% -> 12.5%, CC1/2_hi 14.4% -> 16.5% strong control unchanged to every printed digit but ISa Cost: ISa falls (17.2 -> 14.0 and 16.7 -> 14.9 on the two mid-strength cases, 28.3 -> 27.8 on the control). Strong reflections are untouched by (1) - their partials are all positive, so var_bkg is bit-identical - but the joint a/b fit redistributes: honest weak sigmas lower a, and b rises to keep the strong bins fitted. The median reduced chi^2 improves (1.25 -> 1.14, 1.35 -> 1.28) so the new split describes the scatter better, but ISa is the one headline metric that moves the wrong way and it should be watched over the full battery. The integrator change is shared, so the stills merge sees it too; there it feeds GetExpectedVarianceMerge, which had been handed the same contaminated var_bkg. That path is untested here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 17:40:15 +02:00
leonarski_fandClaude Opus 5	43627e22dc	docs: a rule for licences and academic credit, and apply it ... Several methods adopted recently came from other crystallographic packages - the screw-absence test from POINTLESS, MINPK and the profile-fit reweighting from XDS/Otwinowski, the CC1/2 cutoff and merge outlier rejection from DIALS, the per-frame indexing gate from CrystFEL - and nothing in the repository said where such a debt is recorded. The licence side was already worked out (licences beside the vendored code, verbatim texts in licenses/ collected by COLLECT.sh, a row in THIRD_PARTY_NOTICES.md, all installed under share/doc/jfjoch); the credit side was ad hoc. Write the rule into CLAUDE.md. It states the distinction that matters: vendoring or linking someone's CODE creates a LICENCE obligation, discharged in licenses/ and THIRD_PARTY_NOTICES.md; reimplementing an algorithm from a PAPER creates none of that but creates an obligation of academic CREDIT, discharged in docs/ACKNOWLEDGEMENT.md and in a comment at the algorithm. Neither substitutes for the other, and taking both source and paper incurs both. It also fixes the citation form (authors, title, year, journal, volume, pages, verified DOI), and says in-source credit goes at the algorithm, not the file header, in the one-line style the code already uses. Then bring the repository into compliance for the works concerned: docs/ACKNOWLEDGEMENT.md gains a section acknowledging XDS, DIALS, POINTLESS/CCP4, MOSFLM, CrystFEL, GEMMI, the Kabsch/Otwinowski profile fit, the Diederichs & Karplus statistics and the IUCr nomenclature reports, each with a DOI checked against Crossref; docs/CPU_DATA_ANALYSIS.md's reference list gains the ones it was missing; and four algorithms gain a line naming their source where no adjacent comment carried one. No licence change. licenses/ and THIRD_PARTY_NOTICES.md are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 16:34:48 +02:00
leonarski_fandClaude Opus 5	6f7b136ec2	Bragg integration: a shared signal pixel belongs to the nearer reflection ... Nothing kept a neighbour's flux out of a reflection's own signal disk. The union mask keeps neighbour cores out of the BACKGROUND ring, but the r1 disk was read whole, so on a dense pattern a crowded reflection measures part of its neighbour as its own. Ownership is decided once per image into a per-pixel (quantised distance, reflection) key written with an atomic minimum, so the nearest predicted centre wins whatever order the writes arrive in and the lowest index breaks a tie. `--overlap exclude`, now the default, drops the pixels a nearer neighbour owns from the profile fit. A profile fit is the amplitude of a normalised profile, so leaving pixels out renormalises the estimator by construction and the reflection stays unbiased rather than being discarded; the summation-fallback guard is scaled back to the disk the box-sum seed actually read, so it still compares like with like. `--overlap reject` is the XDS MINPK alternative - drop the reflection when less than `--overlap-minpk` of its expected profile is cleanly its own. A box sum has no profile to renormalise with, so `exclude` is a no-op there and only `reject` acts on it. Widening the split - keeping a pixel only where no other centre is within its distance PLUS a margin - was built and measured, and it is worse monotonically: the residual bias of the pixels that were kept grows from +0.072 to +0.209 in ln intensity at 0 to 3 px of margin. What the margin removes is the reflection's own profile, not the neighbour's tail, so the plain nearest-centre split is the rule. Measured on the full 38-crystal rotation battery against the same binary with the treatment off: ISa better 15 / worse 8, summed shortfall against XDS 39.7 -> 28.1. Three of the losses are the two-pass loop taking its other branch - their median mosaicity moves between the two known attractors - rather than the change under test; excluding those it is better 15 / worse 5 and the shortfall goes 31.3 -> 14.4. The two crowded crystals gain 38% and 52% of their ISa, one of them passing XDS. High-shell CC1/2 over the 35 crystals that neither flipped branch nor carry a collapsed error model is better 7 / worse 7. Space groups unchanged at 35/38. The owner map is built only when a treatment is asked for and costs 1.1% of the battery's wall clock - 23% on a genuinely crowded crystal, nothing where no two predictions touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 05:02:31 +02:00
leonarski_fandClaude Opus 5	06a5a118bf	Bragg integration: widen the background ring to r3 = 13 ... The background is estimated from the r2..r3 ring and then subtracted from every pixel of the r1 disk, so the ring mean's own error enters the intensity n_inner times over: var(I) carries n_inner^2 * bkg / n_B. That term is first-order in sigma, and it is set by how many pixels the ring holds - not by anything about the reflection. At r3 = 10 the ring holds about 200 px against the disk's 50. Widening it to 13 roughly doubles that. The signal disk is untouched, and the pixels gained lie further from the reflection rather than nearer, so nothing is traded for them. The effect is not subtle once looked for. Matched observation by observation on one crystal, halving the ring's pixel count leaves the intensity alone and inflates sigma by 4.7%, and the inflation rank-orders with the ring collapse across the battery. Over the whole rotation battery, against the same binary at r3 = 10: ISa better on 14 crystals and worse on 4, the summed shortfall against the reference 164.7 -> 155.9, the summed low-resolution R_meas excess 69.7 -> 59.1 percentage points, and one more crystal reaching the reference space group (33/37 -> 34/37, a trigonal case that was over-promoting). Largest gains where the ring was starved worst; the four losses are 0.25 to 2.16 in ISa and none of them changes a space group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-11 00:07:30 +02:00
leonarski_fandClaude Opus 5	bf80935b80	Bragg integration: do not let the radial correction outlive its kernel table ... The kernel table is sized and built only where the correction can ever run - explicitly on, or auto, which is the same condition the GPU allocates its radial buffers under. BackgroundRadial(true) on any other engine therefore asked the CPU to correct with a single CIRCULAR kernel for rings that may be elongated, while the GPU, having no buffers, did not correct at all: a wrong kernel on one engine and silence on the other, from the same call. Only the auto path calls it today, so it was unreachable, but the setter is public and the invariant it depends on is not local to it. Remember whether the table was built and refuse to raise the flag otherwise. Also treat a zero stencil cap as "uncapped" rather than "no growth". The engine always sets max_grow, so this changes nothing that runs; it makes a caller that forgets it fail loudly instead of silently disabling the feature. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 16:38:37 +02:00
leonarski_fandClaude Opus 5	61d24db59f	Bragg integration: elongate the background ring per reflection ... The signal disk and the r2..r3 background ring were fixed pixel circles, identical for every reflection at every resolution. A reflection is not round: a finite bandwidth streaks it radially by bw_sigma*Rpx, so at high resolution the ring sits within 1.3-2.2 sigma of the reflection's own profile and measures its tails as background. --integration-stencil <k> makes the RING an ellipse, elongated along the beam->reflection direction by k times that streak, capped at 2*r3. The tangential half-widths stay r2 and r3, and the r1 signal disk stays a circle: r1 drives the all-or-nothing n_inner_valid == n_inner gate, so growing it rejects any reflection carrying one bad pixel along a long streak, and the flux a circular r1 loses is a function of resolution alone, which the per-shell scale absorbs. The geometry lives in one shared header compiled by both the host compiler and nvcc, so the seven pixel-classification sites - the CPU mask/main/clip loops and the GPU mark_mask/main/trim/clip kernels - cannot drift apart. Rather than evaluate an ellipse, each pixel's squared distance has its radial part scaled down, d2 - q*rad^2 against r2^2/r3^2 with q = 1 - (r/(r+grow))^2, so grow = 0 gives q = 0 and both tests collapse onto d2 exactly in floating point. The width is the bandwidth streak alone, not the profile's full radial variance, which also carries the sensor parallax and weak-spot capture terms. Deriving the growth from those was implemented first and measured on the rotation battery: at k=1 it took Thau_9's high-shell CC1/2 from 75.8 to 27.9 and Benas_3's from 14.1 to 6.0, against cytC_10 +1.2 and lyso_ref flat. On a monochromatic beam they are the only terms there are, and C_CAPTURE is 64% of them. Keeping only the streak also makes the option exactly inert without a bandwidth, rather than merely small. Default 0. Measured on broadband rotation data with the bandwidth set to its spectroscopic value, matched resolution limits: high-shell CC1/2 30.6 -> 46.4 at k=4, and better in EVERY shell in both CC1/2 and R_meas (top shell R_meas 194.7% -> 138.7%), with completeness, multiplicity and space group unchanged and 28 of 98833 unique reflections lost. Anomalous peak height over 18 sites +0.107 +- 0.039 sigma (p = 0.013). The full 38-crystal rotation battery is unchanged to every reported digit, base against k=3. Two consequences of an elongated ring are handled rather than inherited. The neighbour exclusion marks the inner ELLIPSE in each neighbour's own frame, or an elongated neighbour leaks its tails into this reflection's ring. And the radial-background curvature kernel becomes a small table indexed by the growth, because its azimuthal average makes one kernel serve every reflection only while their stencils are identical; the GPU's radial window, previously a fixed 32 bins, is now sized on the host from the widest ring on the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-10 15:19:28 +02:00
leonarski_fandClaude Opus 5	1239c49731	Bragg integration: separate the three things a bandwidth used to switch ... Setting a bandwidth flipped three unrelated switches at once: it changed the profile's radial capture term, it moved the width measurement from the signal disk to the whole fit grid, and it silently overrode the background clip and trim, so --background-clip under --bandwidth was ignored - the two runs were bit-identical. The width measurement was the damaging one. The fit grid is an azimuthally averaged stack, so its second moment is sigma_r^2 + sigma_t^2 and the radial smear of a bandwidth leaked into the tangential model - a tangential width of 3.04 px against a 1.06 px truth, inflating the effective background pixel count where the weak signal is. The result was a step rather than a slope: on genuinely monochromatic data, declaring a 0.2% bandwidth cost ISa 28.4 -> 22.2. Measure the two widths separately, accumulated in each spot's own radial/tangential frame over the signal disk, from the signed profile cells - away from the peak a learned cell is background noise centred on zero, so the signed sum is unbiased, while clamping it at zero turns that noise into a pedestal the r^2 weight reads as width. The radial term is then the measured excess or the analytic floor, whichever is larger. With the two widths separated there is nothing left for the broadband switch to select, so it is gone - which is the proof the three were independent. The background clip and trim now come from the settings in every case; the tuned 3-sigma broadband default moves to the rugnux front end, which is the only place that knows whether the user gave a value. Monochromatic data: declaring a 0.2% bandwidth now costs ISa 28.4 -> 27.9 rather than 22.2, and forcing the old 3-sigma clip in the new build reproduces the good result, so none of the step came from the clip. On large-bandwidth data CC1/2 improves in 8 of 10 shells. Across 12 monochromatic crystals the space groups are unchanged and CC1/2 moves by at most 0.2 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 21:08:29 +02:00
leonarski_fandClaude Opus 5	3d3fb0e58b	Bragg integration: stop rectifying the fitted intensity into its own variance ... The profile fit weights each pixel by 1/v with v = max(bkg, floor) + max(0, I)*P, where I is the fit's own current estimate. Rectifying it means that at true zero the plug-in is E[max(0,I)] = 0.4*sigma rather than 0, and with sum(P^3)/sum(P^2)^2 = 4/3 for a Gaussian the reported sigma comes out about 0.2 counts too large - always, additively. That is nothing at sigma ~ 7 counts and 11% at sigma ~ 2, so it only shows on data measured against roughly one background count. Clamp the whole weight instead of the intensity: v = max(bkg + I*P, bkg/2). Simulation of the real integrator gives claimed/true sigma 0.92-1.01 at zero intensity across backgrounds 0.02-2.0 ct/px and 1.000-1.007 above I = 30, where the clamp never binds. Dropping the signal term entirely instead (v = max(bkg, floor)) is exact at zero and wrong everywhere else - 1.91 at I = 5, 4.29 at I = 30, 13.3 at I = 300 - and a test built on systematically absent reflections cannot see that, because it only measures zero. Removing the clamp altogether overshoots and biases the intensity, since a downward fluctuation shrinks v at the peak and over-weights it. The pixel variance floor was 1/12, documented as the rounding of a continuous energy. That does not describe a photon counter: measured on raw frames at 0.065-0.082 ct/px, var/mean is 1.042-1.045, i.e. Poisson with no digitisation term, and a digitisation term would be additive rather than a floor. What the floor really protects is the background estimate, which a small ring can read as exactly zero, so it belongs at the resolution of that estimate, ~1/n_bkg. At 1/12 it multiplied the reported variance by floor/bkg below 0.083 ct/px - a factor of two at 0.04. Set to 0.01. Measured on systematically absent reflections, whose true intensity is zero, as std(I)/rms(sigma) binned by background - not std(I/sigma), which is deflated by the correlation between the plug-in sigma and the reflection's own fluctuation. On 2.78 M absent observations at 0.16-3 ct/px the ratio goes 1.04-1.07 to 0.99-1.00. On 2.58 M at 0.005-0.6 ct/px, decomposed: the clamp carries it above 0.08 ct/px, the floor below it. Intensities move 0.4%; this changes sigma, not I. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 21:08:29 +02:00
leonarski_fandClaude Opus 5	f60768d49c	Bragg integration: drop the 2% sigma floor and carry the background variance ... Two changes to the same variance chain; they are in one commit because the second exists to remove an assumption the first was breaking, and separating them leaves a tree that is correct only by luck. The reported sigma was floored at 2% of the intensity, a per-partial I/sigma cap of 50. It applied only to the box-sum seed, never to the profile fit, so the shipped default was unaffected - but the combine back-derives each partial's non-signal variance as sigma^2 - I, and a floored sigma makes that quantity mean nothing. It then read corr^2 * (0.0004 I^2 - I), which is not a background variance. Measured on --integrator boxsum: the reported sigma understated the true scatter by up to 16x at I ~ 21000 counts per partial, and pooled_I amplified a 1 ct/px background drift into an 11.5% intensity error on the strongest reflections. What the floor stood in for - that at high intensity the error is systematic rather than counting - is already carried downstream, twice: the fitted b in v = a*sigma^2 + (b*I)^2, measured from the data rather than assumed, and SigmaWithSystematicFloor on the merged sigma. The floor was that idea applied one level too early with a hardcoded b of 0.02. It arrived without a test or a setter and was unreachable from the CLI, the API and the config. The merge now takes the non-signal variance the integrator actually measured instead of inverting sigma^2 = I + N. That identity is exact for a box sum once the floor is gone and was never exact for a profile fit, whose sigma^2 = 1/den + (wsum/den)^2 * bkg_var is formed against a fitted intensity. The value is carried through BraggFitResult, Reflection and Obs, both engines, both merges, and the process-file round trip; files written before this change are read with the term absent, which is what they had. Battery, 37 crystals, paired: space groups unchanged, reflection sets unchanged, median delta zero on R_meas and CC1/2. --integrator boxsum on the reference crystal goes ISa 8.9 -> 20.2 with a 0.947 -> 1.032. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-09 19:10:57 +02:00
leonarski_fandClaude Opus 5	b71e8c6a56	Bragg integration: use the project's PI, not M_PI, in the radial kernel ... M_PI is not standard C++. MSVC defines it only when _USE_MATH_DEFINES is set before <cmath>, so the radial background kernel's azimuth loop does not compile there: error C2065: 'M_PI': undeclared identifier error C2737: 'phi': const object must be initialized (cascade from the first) GCC and Clang define it anyway, which is why the Linux build stayed green. image_analysis is viewer-reachable, so it has to build under MSVC. common/JFJochMath.h already carries a constexpr PI for exactly this reason - its comment names this case - so use that. Same value to the last digit, so the integration results are unchanged; the CPU/GPU parity test passes unaltered (9002 assertions). This was the only M_PI left in the viewer-reachable tree. The remaining uses are in tests/, which Windows does not build (JFJOCH_VIEWER_ONLY is forced there). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 20:52:33 +02:00
leonarski_fandClaude Opus 5	f0cdb027e1	Ice: default the merge mask off, gate the radial background on smooth ice, and pick detection by geometry ... Three defaults, each settled by measurement rather than by argument. The arbiter throughout is structure-referenced - anomalous peak height where a crystal can carry it, and otherwise the agreement of the ice bands with a fixed external model against resolution-matched DECOY bands carrying no ice. The band-versus-decoy contrast is used because R-free here tracks completeness, and every one of these switches moves completeness. The damage is real and it localizes: over the rotation battery the ice bands' excess amplitude reaches +9.6% on a smooth-ice crystal and +35% on the worst, while a clean control sits at +0.6% (z +0.45). On the worst crystal, nine of the ten largest excess peaks in a q scan land on hexagonal ring positions. Turning ice handling off leaves the contrast unchanged and forcing it on a clean crystal does not create one, so it is the ice and not the machinery. MERGE-TIME RING MASK -> OFF. It deletes reflections, which no other program does by default - AIMLESS, DIALS, xia2, XDS and CrystFEL all keep ice-band reflections in the merge and exclude them only from the model fit; autoPROC is the sole exception. On the one battery crystal where the mask fires and an anomalous arbiter can score it, dropping the band moved the mean peak height at the known sites by -0.001 +- 0.018 sigma, 2% of the site height, while removing 1149 unique reflections whose mean I/sigma was 3.62 against the dataset's own 3.05 - better than average data - and costing 17 completeness points in that shell. It fires on 5 of 37 crystals, changes no space group, and those 5 disagree in sign: it clearly helps the two most heavily iced, is a wash on two and costs a third. So it stays as a switch, worth setting by hand on a badly iced crystal where it shows in the high shell, but it is not a default. RADIAL BACKGROUND -> AUTO, gated per image. The correction models the background as a function of radius alone, and that is exactly when it works. On a crystal with pure smooth powder ice it removes 43% of the bands' excess amplitude, with the improvement 7x larger inside the bands than outside; on a crystal whose ice is discrete crystallite spots - no smooth ring to model - the excess amplitude GREW by half; on clean data it is inert to four decimals. The two ice channels already separate those morphologies, so --background-radial takes on|off|auto and auto applies it to an image when that image's peak-excluded score reaches --ice-min-score. Auto never engages without such a score, because the plain profile carries the Bragg peaks and cannot support an absolute threshold. Per image rather than per run, and that was tested rather than assumed: the gate fires on 100% and 94% of frames on the two crystals that want it, and on 1.5% of frames - 32 blocks, 23 of them single frames - on the textured-ice crystal. A seam statistic against off + f*(on - off) is null on both mixed runs, every merge statistic is bracketed by the pure arms, and the textured crystal's auto arm lands on `off` rather than on `on`'s harm. A run-level gate would need the score before the pass that integrates, i.e. rotation-only plumbing, and buys nothing measurable. The kernel was already built unconditionally, so flipping the flag per image is free - except on the GPU, where the launches were gated on a construction-time n_rad. That is why the buffers are now allocated whenever the correction could run, and Run() decides per image. DETECTION -> the geometry's default when the file is silent: on for rotation, off for stills, with the command line and then the file taking precedence. A rotation sweep sits on the same rings for the whole run, so ice there is a coherent systematic and the presence gate keeps it inert on a clean crystal; a serial stills run has too few spots per image to spend any on flagging. The master file's key is kept as written rather than collapsed to a bool, so "the file said nothing" is distinguishable from "the file said no" - it used to fall silently to off, taking the exclusion from the scale fit with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 19:06:30 +02:00
leonarski_fandClaude Opus 5	0e23fd3ab9	Bragg integration: propagate the background-estimate uncertainty, add an opt-in radial background correction ... Two independent pieces in the same code path. The background-estimate variance was never propagated. A reflection's background comes from a finite ring of n_b pixels, so subtracting it adds var(B)/n_b per signal pixel - sqrt(1 + n_d/n_b) = 1.109 with the shipped stencil. Both engines omitted it, which is exactly the 1.11-1.19 gap measured between the off-ring scatter and the reported sigma. Three lines each; it affects every dataset, not only iced ones. The radial correction is new and OFF by default (--background-radial). The signal disk and the background ring are concentric, so for any background LINEAR in position <B>_ann == <B>_disk identically and a plane fit buys nothing; the leading error is the CURVATURE of the radial background, which on a sharp ice ring reaches +26 counts on a single reflection. Since every reflection uses the same stencil, that error is a fixed kernel over radial offset - one short dot product per reflection and no extra pixel reads. Validated on empty apertures before any C++: mean |bias| over 9 bands / 3 crystals 4.33 -> 0.79 counts with the scatter unchanged. Three things it cost a battery each to learn, all now in the code: - the radial curve must be accumulated from CLIPPED annulus pixels, inside the clip pass, or it carries neighbour tails and zingers (so it is inert under --integrator boxsum, which has no clip pass); - the GPU version was a 1.8x slowdown from atomicAdd contention on a small radial array - staged in shared memory per block it now costs nothing measurable; - it is battery-NEUTRAL as a default, because the reflections whose bias it fixes are the ones the ice handling already excludes. Hence off by default. CPU/GPU parity extended with two radial sections: 9002 assertions. Also fixes a latent French-Wilson quadrature collapse: j_max = I + 8 sigma on a fixed 400-point grid degenerates to a single cell once sigma >> 50 <I>, giving F = 0.1 sqrt(sigma) with sigmaF -> 0. Harmless today, but any sigma-inflation scheme detonates it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 15:44:13 +02:00
leonarski_fandClaude Opus 5	09fb8e0306	Bragg integration: clip the background ring high side instead of trimming it ... The r2..r3 background ring was averaged with a 10% SYMMETRIC trimmed mean. A symmetric trim is not a consistent estimator of the mean of a right-skewed (Poisson) sample: on a clean Poisson ring it sits ~0.1 ct/px BELOW the true mean at every level, and with ~50 signal pixels in the r1 disk that under-subtraction adds ~5 counts to every partial on every frame. Measured two independent ways on four rotation datasets - stored background_mean against a plain ring mean over the same pixels on reflection-free frames, and directly on apertures that provably hold no reflection. Empty-aperture pedestal, counts: plain mean -0.03..-0.20, 10% symmetric trim +5.05..+6.34, 4 sigma clip +0.02..+0.54. Replace it with a high-side-only sigma clip at mean + n*sqrt(mean), n = 4 for monochromatic data. It rejects the same one-sided contamination the trim was there for - better, in fact: a 40 px neighbour core at +100 ct shifts the trim by +10.1 ct/px, because a symmetric trim collapses once contamination exceeds ~10% of the ring, versus +0.009 ct/px at 4 sigma. False rejection on a clean ring is 0.04-0.39%. Broadband data keep their tuned 3 sigma clip unchanged. The trim stays reachable with --background-trim for back compatibility; setting either estimator clears the other, so they can never stack. --integrator boxsum does not take the clip (matching what the shipped clip already did), so it now uses the plain ring mean unless --background-trim is given. The intensities get measurably more accurate: per-shell agreement with an independent processing of the same images improves on 14 of 16 crystals (weighted -0.0347, outermost shell 12/4), the outermost-shell R_meas NUMERATOR - absolute scatter, not a denominator effect - falls 13.5% median on 16/5, and CC1/2 in the outer shell improves on 14/7. EXPECT <I/sigma> TO FALL AND EDGE R_meas TO RISE. Both are inflated by information-free counts, so both get worse when the bias is removed; neither is evidence against this change. That fingerprint is exactly how the trimmed mean was accepted in the first place. Known cost: over the 37-crystal rotation battery the de-novo space-group count goes 34 OK / 3 DIFF to 33 / 4. The single regression is a two-lattice crystal whose merge fails the absolute-sanity gate under either background (R_meas 63.5%, CC1/2 72.2%) and which carries an unresolved indexing ambiguity on the very operator being scored, so its operator CC is diluted by construction. No other crystal changes space group, and twin protection is not weakened - the H-ratio veto that refuses genuinely twinned crystals gets MORE decisive (1.63 -> 1.84, 2.83 -> 3.99). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 18:03:11 +02:00
leonarski_fandClaude Opus 5	4b1c611bdf	image_analysis: query the current device, and upload the resolution mask on the engine stream ... Two leftovers from earlier fixes of the same shape. BraggIntegrationEngineGPU still read device 0's shared-memory size to decide whether its profile grid fits; workers are pinned round-robin across GPUs, so on a heterogeneous node that check can pass on a different card than the one the kernel launches on. SpotExtractorGPU still uploaded its default resolution mask with a pageable copy on the NULL stream, which is not ordered against the engine stream now that streams are created non-blocking. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 14:14:24 +02:00
leonarski_fandClaude Opus 5	13aa20a528	bragg_integration: grow the GPU reflection arrays with slack ... EnsureCapacity resized its 13 device arrays to exactly the current image's predicted-reflection count, so every image that set a new record freed and reallocated all of them. cudaMalloc and cudaFree take a device-wide lock in the CUDA driver, so those images stalled every other worker: sampling the worker threads during the per-image loop found 21-24 of 32 parked in cuMemAlloc_v2 or cuMemFree_v2, all called from this one function, and the running maximum makes 32 workers do far more allocator work than one does. Grow by half again instead. All transfers and kernel launches are sized by the per-image reflection count rather than by the capacity, and the member is already documented as holding at least that many, so over-allocating changes no result. On an 18 Mpx rotation set the integration stage drops from 1.37 to 1.25 ms per image at 32 workers; merged statistics, error model and adopted space group are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-03 01:02:07 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	cc3eb8352c	v1.0.0-rc.148 (#58) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. * jfjoch_broker: Improve azimuthal integration (add <I^2> calculation) * jfjoch_broker: Fixes around indexing, aiming to handle multi-lattice crystals (work in progress, it is not fully integrated) * jfjoch_writer: Save mean(I), stddev(I), and count(I) for each azimuthal bin Reviewed-on: #58	2026-06-08 08:30:35 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_f	bb9f5c715f	v1.0.0-rc.135 (#44) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. * Multiple small bug fixes scattered across the whole code base. (detected with GPT-5.4) * jfjoch_viewer: Improve image render performance Reviewed-on: #44 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-16 11:59:59 +02:00
leonarski_f	928789a67c	v1.0.0-rc.130 (#37) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Rotation indexer has two retries if failes * jfjoch_broker: Rotation indexer handles small number of rotation images (like test shot) * jfjoch_broker: Integration calculates background mask based on R2 radius * jfjoch_process: HDF5 files are not saved by default Reviewed-on: #37	2026-03-06 14:38:56 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00
leonarski_f	27496b8207	v1.0.0-rc.122 (#29) ... This is an UNSTABLE release. * jfjoch_broker: Add thresholding to prefer shorter vectors after FFT * jfjoch_broker: Add experimental mosaicity estimation for rotation experiments (this is work in progress) * jfjoch_viewer: Display file opening errors * jfjoch_viewer: When loading files over DBus add retry/back-off till the file is available Reviewed-on: #29 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-12-16 15:27:40 +01:00
leonarski_f	f44c6520a8	v1.0.0-rc.107 (#12) ... This is an UNSTABLE release. * jfjoch_viewer: Minor polishing of new functionality * jfjoch_broker: User NaN for empty azimuthal bins Reviewed-on: #12 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2025-11-24 07:07:32 +01:00
leonarski_f	fb9462f67b	v1.0.0-rc.92	2025-10-23 09:05:11 +02:00
leonarski_f	061152279c	v1.0.0-rc.91	2025-10-20 20:43:44 +02:00
leonarski_f	14f5051692	1.0.0-rc.89	2025-10-01 22:54:40 +02:00
leonarski_f	99b7dc07f7	1.0.0-rc.87	2025-09-30 20:43:53 +02:00
leonarski_f	5d9d2de4a4	v1.0.0-rc.81	2025-09-21 19:27:51 +02:00
leonarski_f	20cd8dc4ac	v1.0.0-rc.76	2025-09-11 14:37:16 +02:00
leonarski_f	c67337cfe1	v1.0.0-rc.72	2025-09-08 20:28:59 +02:00
leonarski_f	fa95858008	v1.0.0-rc.69	2025-08-17 21:21:20 +02:00
leonarski_f	a32cec3404	v1.0.0-rc.67	2025-08-11 11:13:59 +02:00
leonarski_f	0d1b388f4d	v1.0.0-rc.50	2025-06-20 09:52:26 +02:00