42 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	ffbf38d2ab	Rotation scaling: guard the per-frame scales whatever else is switched on ... The protection against a per-frame scale collapsing toward zero lived inside ComputeSmoothGWindow, so it only existed when smooth-G did: --smooth-g=0, a dataset whose oscillation width is unknown, and any caller that never sets a smoothing range - the viewer among them - merged with no guard at all. A collapsed G multiplies that frame's intensities by 1/G and its sigmas by the same factor, so nothing downstream can see it; the merge's n-sigma cut scales with the number that is wrong. Pull it out into ReplaceCollapsedScales, called unconditionally right after the partial scaling loop, and let the smooth-G window assume what it now guarantees instead of computing its own median and floor. The fulls guard built its median from every frame including those never fitted - those sit at the combine's corr = 1, so a run with many unfitted frames dragged the median toward 1 and the floor with it. It also reported the absolute amplification where the message says "below the run median". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 17:51:44 +02:00
leonarski_fandClaude Opus 5	a95aca382c	Remove dead code left behind by recent changes ... None of this has a reader: - ScalingSettings::scaling_regularize and its setter/getter - ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read), with the timing that only fed the latter - JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the comment claiming the retry uses it - the retry keys off initial_fit_done_ - JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include in a file that uses none - an unused gemmi::Op accumulator in the spindle-angle helper - <random> in Merge.{h,cpp}, from before the half-set split became a hash - an orphaned comment describing the Ceres B-factor residual deleted in 014e43a4c, and two trailing comments that had collided on one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:29:42 +02:00
leonarski_fandClaude Opus 5	f90969ea21	Stills partiality: do not adopt a tilt from a failed solve ... The Ceres summary was discarded, so a solve that diverged or aborted left its last iterate in psi and that tilt was written onto the partiality of every reflection of the crystal. Restore the tilt the crystal came in with and stop refining it; the scale fit alone is still a usable model, which is what the other three early returns in this function fall back to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:14:26 +02:00
leonarski_fandClaude Opus 5	3e8a994d2e	Stills scaling: leave an image unscaled when its scale collapses ... SolveScaleIRLS returns whatever it converged to and both writers accept any G > 0, so a fit that collapsed to ~1e-3 multiplies that image's intensities by a thousand. Nothing downstream notices, because the sigmas are multiplied by the same factor and the merge's n-sigma outlier test is therefore blind to it - only a total collapse self-heals, by overflowing corr to inf. The rotation path refuses a per-frame scale this far below its neighbours; the stills path had no guard. Judge each image against the median of the images that did scale, and put a collapsed one back to G = 1 - the same state as an image with too few reflections to fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:13:51 +02:00
leonarski_fandClaude Opus 5	af97ad61e3	Rotation scaling: the search-only filters must not outlive their pass ... Two faults in the same block, both of which let a search pass corrupt the production merge that follows it. The device's corr was only copied back to the host for the diagnostic dump, but the |zeta| filter runs on the host and then uploaded the whole host array - so on a CUDA build it wrote the values ingested BEFORE scaling over the scaled and smoothed corr the device had just computed. With the rotation default --search-min-zeta 0.85 that means the space-group search was deciding the symmetry from an unscaled merge. Copy corr back first, and upload once after both filters instead. Zeroing corr also has no owner: it is how an observation leaves the merge, but the only thing that ever rewrites it is the scaling loop, which skips frames it cannot fit. A frame left with too few well-measured reflections therefore kept its dropped observations at zero for the rest of the object's life - and the final production merge re-uses the same object without re-ingesting. Snapshot corr before the filters and restore it at the start of the next pass, so each pass decides for itself and the final merge keeps everything, as documented. The frame rejection (--min-image-cc) is now applied on the host for both paths; its separate device path did nothing whenever the CPU combine was in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:12:13 +02:00
leonarski_fandClaude Opus 5	64c8117a96	Space-group search: absences always come from the all-observation merge ... The two-arm search is meant to use the Lorentz-filtered merge for the point group only - systematic absences live in the weak reflections a |zeta| cut removes, and reading them off the filtered arm is what cost four crystals their screw axes. That is what the code comment and CPU_DATA_ANALYSIS both say, but the filtered-arm-wins branch kept its whole result, screws and centering included. Let a search be pinned to a point group decided elsewhere (fixed_point_group) and re-run Stage B on the all-observation merge when the filtered arm rescues the point group. The point group is passed as its symmorphic representative, not by name: gemmi calls both P321 and P312 "32". Reporting that representative also lets the ambiguity check see two arms that disagree about which 2-folds are real - by name they looked identical - and the advice it prints now names a space group -S can actually be given. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:54:02 +02:00
leonarski_fandClaude Opus 5	1355d6b2aa	Space-group search: a screw threshold needs a row with controls behind it ... The row-relative cut scales the "too strong to be absent" threshold by the axial row's own median E^2, floored at the plain value - so it can only raise the bar, and a row whose control class holds a single strong reflection sets it from that one reflection. That direction invents screws: a genuine 4_2 whose 00l happen to be observed only at l=4n reads its l=4n+2 reflections as absent and ranks 4_1/4_3 above the truth. Require three controls before the row may set the scale; below that the row keeps the plain cut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:52:32 +02:00
leonarski_fandClaude Opus 5	72cd91ccc3	Space-group search: judge a promotion against the worst of its tied parents ... A candidate can have several confirmed subgroups of the same order - 422 has both 4 and 222 - and on a twinned crystal the rival is not a harmless alternative: a P4 crystal twinned by 2[100] has 222 confirmed too, and 222 CONTAINS the twin laws, so its own merge b is already ballooned. The H test already answers to every tied parent; the systematic-b veto and rescue took whichever one the enumeration happened to list first (222 before 4, by space- group number), which disabled the veto on exactly the case it exists for. Take the smallest parent b, which is the conservative direction for both tests. The refusal message also quoted the raw parent b rather than the floored value the veto actually compared against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:52:05 +02:00
leonarski_fandClaude Opus 5	609650b061	Rotation merging: keep the systematic sigma floor when ISa is unmeasurable ... The cap that refuses to report an impossible ISa was zeroing the asymptotic b itself, and that same value is the floor passed to SigmaWithSystematicFloor - where zero means "no floor". So on the degenerate low-multiplicity fit the guard is written for, instead of capping merged I/sigma at 100 it removed the cap entirely. Report the asymptote as unmeasured, keep the fitted value for the floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:50:30 +02:00
leonarski_fandClaude Opus 5	386f10ad07	Merging: mirror the negative-intensity Poisson guard on the GPU ... 6be94f2be stopped subtracting a negative intensity's Poisson term from the background variance, but only in the host Combine(). The CUDA combine is the path that actually runs: Run() selects it whenever a device is present and no observation dump was asked for, so the correction never took effect on a normal run, and a --dump-observations run merged differently from a normal one - the two are meant to be identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:49:37 +02:00
leonarski_fandClaude Opus 5	6be94f2be0	Merging: do not subtract a negative intensity's Poisson term ... The expected-variance weights decompose an observation's sigma^2 into a background part and a Poisson signal part, then rebuild the signal part at the reflection's merged mean. The decomposition subtracted corr*I with I taken as-is, so a negative I ADDED to the background part: an observation at I = -1.5 with sigma^2 = 1 came out with a base variance of 2.7 rather than 1. That inflates the variance of precisely the down-fluctuated observations the correction exists for. Below about one photon they are then under-weighted and the merged mean is biased high - the same direction of error, in the same regime, that weighting by the observation's own sigma produces. Subtract max(0, I) instead: a negative intensity has no Poisson signal to remove. Both users of the decomposition are fixed - the stills merge, where expected-variance weighting is now the default, and the rotation combine it was mirrored from, which had it first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:36:40 +02:00
leonarski_fandClaude Opus 5	157698d31e	Space-group search: report the order of the point group it chose ... The two-arm search compared its arms by the order of the space group each had picked, but Stage B leaves best_space_group unset whenever no candidate is eligible - no candidate had enough observed absences to trust. That is not rare on the Lorentz-filtered arm, and for a systematic reason: the filter removes the badly-measured observations, which is where the weak systematically-absent reflections are. An arm that confirmed 422 but stopped short of naming a space group therefore scored order 0 and lost to an arm supporting P2, and the demotion was logged as "taking the higher symmetry" - the comparison and the message both wrong, in the one direction the design says cannot happen. Carry the point-group order in the result, set from the order Stage A actually adopted, and compare on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:33:29 +02:00
leonarski_fandClaude Opus 5	d28db19ab1	Space-group search: judge a screw axis against its own axial row ... A reflection the group predicts absent counted as a violation when I/sigma > 3 AND E^2 = I/<I>(shell) > 0.3. Neither half survives contact with real data: * merged sigma is floored at b|I|, so merged I/sigma saturates at ISa for nearly every reflection - the I/sigma half is an on/off switch keyed on ISa vs 3, not a per-reflection test. On one crystal the absent class read <I/s> 4.10 against 3.73 for the present class while being genuinely extinct; * <I>(shell) decays with resolution while a systematically-absent reflection keeps a small NON-decaying residual (background / profile leakage), so absent reflections drift over an absolute E^2 cut at high resolution. That cost a tetragonal 42_12 crystal its 4_1: 18 of its 47 absent 00l crossed the cut, all beyond 3.7 A, at absolute intensities identical to the low-resolution ones correctly judged absent, while their l=4n row-mates sat 20-60x higher at the same resolution. A screw extinguishes only the reflections that lie ON its axis, so the fair yardstick is the rest of that same row. The threshold is now 0.3 * max(1, median E^2 of the reflection's own row), the row being the gcd-reduced reciprocal-space direction and the control class the same-row reflections the group predicts present. Floored at 1, so it only ever relaxes: a screw can be recovered by it, never lost. Per row, not pooled. A 4_1 along c and a 2_1 along a are separate conditions with separate controls; pooling let the weak a/b rows (median E^2 ~0.5) set the threshold for a strong c row (8.4) and the rescue never fired. The candidate table now reports the screw evidence (median E^2 of the absent class and of its rows) - the <I/s> columns are the centering evidence and say nothing about screws, for the sigma-floor reason above. Rotation battery, 33 crystals: 31 decisions bit-identical, the 42_12 crystal recovers its 4_1 (0 violations, row E^2 8.4 vs absent 0.12), and one crystal with a long axis and heavy 00l overlap moves to a 4_1 group at exactly 10.0% violations - marginal, and its sister crystal of the same form sits at 13.3% and does not move. Real screws now span 0-9.3% violations, so max_absent_violation_fraction cannot be tightened below 0.10 without risking a genuine one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 08:27:34 +02:00
leonarski_fandClaude Opus 5	25458265d3	Space-group search: ask twice - all observations, and only the well-measured ones ... --search-min-zeta rescues a point group that the full merge cannot confirm, but used on its own it is a trade: on the crystal it was built for it recovers the correct 422, and on four others it costs the space group outright, because discarding 40-80% of the observations starves operator correlations that were perfectly healthy. Both ways of applying it - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade the SAME crystals, so the cut itself is the problem, not where it is applied. Filip's observation makes it one-way: every disagreement between the two is a LOST operator, never an invented one. Discarding observations can starve a correlation; it cannot manufacture symmetry that is not there. So run the search on both merges and keep whichever found MORE symmetry, and the failure mode disappears - each arm rescues the other exactly where it fails. crystal all observations Lorentz-filtered adopted thaumatin (weak) 222 422 422 tetragonal lysozyme 422 222 422 cubic insulin x3 23 2 / 222 23 The filtered merge is used ONLY to rescue the point group. The screw and centering determination always comes from the merge with all the observations, because systematic absences are decided by the WEAK reflections and the filter throws most of them away. Preferring the filtered arm on a tie is not a conservative choice, it is a wrong one: it cost four crystals their screw axes (P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa. Where the two find the same ORDER but different symmetry, nothing can prefer one, so the run says so: it names both space groups, states that the data do not decide, reports which one processing continued in, and gives the flag to force the other. Two candidates of the same order imply different molecular replacement searches, and trying both is cheap next to reprocessing - much cheaper than a confident wrong answer. Rotation battery, 33 crystals, both spot finders: fixed-threshold finder 30/33 - ZERO crystals differ from the single search adaptive finder 30/33 - the same three mismatches, gap CLOSED The adaptive finder now matches the fixed-threshold one exactly, which it has not done before: its last remaining loss was the thaumatin set whose 4-fold sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused in BOTH arms at all three frame ranges where it over-promotes, and at one of them the second opinion is strictly better than shipping behaviour - the full merge collapses to P1 where the filtered one finds the correct H3. Cost is the extra scale-combine-merge on already-ingested partials, with no re-integration: 47.2 s against 47.8 s on the same crystal back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:11:05 +02:00
leonarski_fandClaude Opus 5	f2b92e3f4d	rugnux: --search-min-zeta drops badly-measured observations from the symmetry search ... zeta is the sine of the angle between a reflection's rocking path and the spindle. Near 0 the reflection crosses the Ewald sphere almost tangentially, spends many frames in diffracting position and is measured worst. The de-novo space-group search asks how EQUAL an operator's paired intensities are, so its answer is dominated by whichever reflections are measured worst - and when the spindle lies in a lattice plane, an operator that permutes the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. That is not a fair comparison, and it can make a real symmetry operator look like a twin law. Measured on a thaumatin set mounted that way (its 4-fold is 88.9 deg from the spindle), the added operators' disagreement is 1.74x the parent's over pairs where both reflections have zeta < 0.85 and 1.003x - i.e. the symmetry is exact - over pairs where both are above it. The search consequently refuses the 422 promotion and merges the crystal in P222, while the same data forced to the right group give CC1/2 99.2% at multiplicity 10.7, matching XDS. With the option the de-novo pass ignores those observations (the final merge keeps everything - there completeness is the point): zeta cut observations ignored H ratio adopted 0 (off) - 1.47 P222 0.5 1620648 1.44 P222 0.7 3006013 1.34 P21212 0.85 4536724 promoted P4212 (correct point group) OFF BY DEFAULT, and it must stay off, because the same cut costs four other crystals their space group (P41212 -> P212121, I23 -> P2, I23 -> I222 twice): at 0.85 it discards 40-80% of all observations, which on a crystal whose geometry is not the problem simply starves the search. Two independent implementations - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade exactly the same crystals, so this is a property of the cut and not of where it is applied. Verified bit-identical to the previous binary when off. The companion diagnostic is already there: the run now reports how close a symmetry axis lies to the spindle, which is the geometry that makes this option worth reaching for. Implementation note for anyone tempted by the cheaper route: excluding these observations from the ASU grouping alone does NOT work. The 3D combine selects partials on corr, not on their group, so their intensity still reaches the fulls and the merged intensities are unchanged - measured, the statistic did not move by 0.03 while 67% of observations were nominally excluded. Zeroing corr is what removes an observation from the combine, the merge and the error model alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 20:03:43 +02:00
leonarski_fandClaude Opus 5	4895dc1018	Rotation: let --min-image-cc drop frames that disagree with the merged reference ... The flag was accepted on rotation data and did nothing - it is read only by the stills merge (Merge.cpp), and the CLI warned about that rather than fixing it. Meanwhile RotationScaleMerge already COMPUTES a per-frame correlation against the merged reference and writes it to the per-image table; nothing acted on it. Wire the two together. A rejected frame has its partials' corr set to 0, which is how a frame already leaves the pipeline - every consumer requires corr > 0, so the combine, the merge and the error model all drop it together. The GPU path reuses the SmoothCorr kernel with a ratio of 0, so one implementation covers both. Off by default (0), and verified bit-identical to the previous binary when off. What it catches, on the two rotation datasets that have a population to catch: a two-lattice crystal - two lattices in two physical AREAS of the sample, so the sweep passes from one to the other and whole blocks of frames measure a different crystal from the one being merged (frames 500-700 index perfectly well at a per-frame CC of 0.22 against 0.47-0.56 either side, in 11 contiguous runs). R_meas 28.6 -> 24.6%, CC1/2 93.6 -> 95.1, high-shell CC 23.4 -> 38.3. a second dataset with 9.5% of frames below CC 0.30: R_meas 24.3 -> 23.4%, CC1/2 92.6 -> 93.4. The criterion is "this frame disagrees with the merged reference", NOT "this frame is off-crystal". It happens to catch both, because a frame that measures nothing and a frame that measures a DIFFERENT crystal fail the same test, and it does not need to know which. For the two-area case that is a workaround, not a treatment: it recovers one crystal by discarding the other, where processing the two as separate sweeps would keep both. The frame-block structure is clean enough that such a split could be detected automatically. WHY THERE IS NO DEFAULT. The per-frame CC is not comparable between datasets - it is as much a measure of data quality as of frame validity. Measured medians across the battery run from 0.30 to 0.81, so one absolute bound removes 13 frames from one dataset and 584 of 1800 from another: battery at --min-image-cc 30, 33 crystals: no point group changed (30/33), four crystals clearly better (one +5.4 CC1/2 points, the two-lattice case above, and ISa gains of 1.3-4.6 on three others) - and one healthy crystal lost a third of its frames and with them its high-resolution shell (CC1/2_hi 26.2 -> 2.0). This is the same trap as an absolute bound on any per-operator or per-frame agreement statistic, and the same one the per-frame scale guard avoids by measuring against the run's own median. A principled version would cut on the SHAPE of the per-frame CC distribution - a dataset with a bad subpopulation is bimodal, a uniformly weak one is not - rather than on an absolute value. Until that exists this stays opt-in, and the per-image CC it keys on is already in the _image.dat table for anyone choosing a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:45:49 +02:00
leonarski_fandClaude Opus 5	11c7cab2e5	Space-group search: take the operator disagreement as a median, not a mean ... A merohedral twin mixes EVERY reflection with its twin mate, so it shifts the whole distribution of |I1-I2|/(I1+I2). A minority of badly measured reflections shifts only the tail. The mean cannot tell those apart; the median is blind to the second and just as sensitive to the first. Measured on real crystals, moving the statistic from the mean to the median leaves genuine promotions where they are and pushes every twin up: genuine tetragonal 1.016 -> 1.013 genuine lysozyme 1.051 -> 1.067 genuine tetragonal 1.238 -> 1.231 twin (-e 1050) 1.272 -> 1.447 twin (-e 450) 1.280 -> 1.622 twin (full) 1.441 -> 1.522 twin (-e 600) 1.427 -> 2.010 The margin around the 1.25 bound widens from 2.7% (genuine 1.238 against twin 1.272 - uncomfortably tight for a decision that cannot be undone downstream) to 17.5% (1.231 against 1.447). The bound itself does not move. Rotation battery, 33 crystals in both detection modes: no point group changed in either (30/33 and 29/33, as before), and only one crystal's numbers move at all - the one already documented as nondeterministic between repeat runs of the same binary. The synthetic twin-fraction x multiplicity grid passes unchanged. So this buys margin, not outcomes. Found while testing a different hypothesis, which the same measurement refuted: a tetragonal crystal whose 422 promotion is wrongly refused reads 1.484 by the mean and 1.472 by the median, i.e. its disagreement is distribution-wide and is NOT a badly-integrated minority. That crystal's cause is elsewhere and is not addressed here - see the note below. Its indexing-ambiguity operator (-k,-h,-l) lies INSIDE 422 but OUTSIDE 222, so the subgroup merge the search is given mixes lattices indexed in the two alternative hands. That corrupts exactly the 4-fold relationships and leaves the 2-fold ones intact - measured, the 222 step reads 0.917 and the 422 step 1.484 - and the corruption is indistinguishable from a twin law. Forcing the tetragonal group merges the two hands as equivalent and the same data give CC1/2 99.2% at multiplicity 10.7, matching XDS. The failure is worse the BETTER the frames index (99.9% vs 63.3% for the run that gets it right), because indexing more frames picks up more of both hands. So no statistic computed on a subgroup merge can arbitrate a promotion whose added operators include an indexing-ambiguity operator. Fixing that means resolving the ambiguity before the search, or detecting the coincidence and deciding another way; the operators needed to detect it are already computed (the run warns about them). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:07:06 +02:00
leonarski_fandClaude Opus 5	ec7a826136	Rotation scaling: guard the fulls refit against a collapsed per-frame scale too ... 704098712 guarded the per-frame scale on the partials, but it put the check inside ComputeSmoothGWindow - which only the partials path calls. Step 4 refits the scale from scratch on the COMBINED FULLS (Unity model, so corr is exactly 1/G), with no smoothing and no floor, and that refit was still free to collapse toward zero. It is the same failure and it is worse here, because there is no window average to dilute it: the collapsed frame's own fulls are multiplied directly. Measured on a dataset where the previous commit had already fixed the partials stage, the fulls refit put 1/G = 559x and 175x on two frames carrying 517 observations, and the merged CC1/2 read 26.2% where the intensities ENTERING that stage were fine - better, in fact, than the comparison run's in all ten resolution shells (R_meas 34.5% vs 39.5%, CC1/2 89.1% vs 83.0%). Reject a collapsed scale here as well, on the same measured criterion, and let those fulls merge unscaled - the state the combine left them in, and the same fallback the fit already uses for a frame with too few reflections. The check reads the host fulls after both the CPU loop and the GPU ScaleFulls, so one implementation covers both paths; the corrected corr is pushed back to the device exactly as the correction surfaces already do. THIS IS NOT A DETECTION-MODE PROBLEM. Over 8 configurations (both spot finders x 4 frame ranges) the separation is exact: every run with a collapsed scale had CC1/2 <= 58.5%, every run without had CC1/2 >= 74.1%, and nothing else predicted it. On one frame range it is the DEFAULT finder that collapses (CC1/2 58.5%) while the other is clean at 93.8%. The instability was never specific to the finder; it was latent in the scaling stage and either finder could trip it. Same 8 configurations, with this commit: finder A full 93.7 -> 93.7 (untouched) finder A -s 1 58.5 -> 91.3 (recovered) finder A -e 899 94.0 -> 94.0 (untouched) finder A -e 898 87.2 -> 87.2 (untouched) finder B full 26.2 -> 91.1 (recovered) finder B -s 1 93.8 -> 93.8 (untouched) finder B -e 899 8.1 -> 91.7 (recovered) finder B -e 898 74.1 -> 74.1 (untouched) Every collapse recovers; every healthy run is unchanged. The CC1/2 spread over the four frame ranges falls from 35.5 to 6.8 points for one finder and from 85.7 to 19.7 for the other - this dataset was not sampling a deep instability when its CC1/2 swung between 17 and 94 across frame ranges, it was sampling whether this bug happened to fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 23:08:50 +02:00
leonarski_fandClaude Opus 5	7040987125	Rotation scaling: do not trust a per-frame scale that has collapsed toward zero ... The per-frame scale enters every intensity as 1/G, and SolveScaleIRLS floors G at zero and nothing else. A frame whose fit is not determined by its data can return G ~ 0.002 against a run median of 0.865, and every observation it carries is then multiplied by ~500 - sigma by the identical factor, which is why no sigma-based outlier test can see it and why this looked for a long time like a partiality problem. (The 1/partiality path is in fact guarded: min_captured_fraction floors it at 0.7 by default on rotation.) The window smoothing that should have absorbed such a frame instead made it permanent. It averages log G over a window, so a scale collapsing toward zero does not merely corrupt its own frame - its logarithm drags the whole window down. Worse, where a run has a stretch of frames too sparse to fit at all, the only FITTED frames in a window can be the collapsed ones, and the geometric mean then averages the fault with itself. Measured on a multi-lattice dataset: frames 816 and 818 fitted G = 0.0023 and 0.0014 with every neighbour from 800 to 839 unfitted, so smoothing set G = 0.0018 across the whole neighbourhood - a 546x amplification. About 500 observations of 152000 (0.66%) then carried 99% of sum(I^2), and the merged CC1/2 read 17.2% where the same data with the classic finder read 93.7%. Treat a fitted scale far below the run's median as what it is - an undetermined scale, exactly like the too-few-reflections case the code already handles - rather than as a successful fit. Such frames no longer contribute to the smoothing mean, and a frame whose own scale is not credible takes the neighbourhood's, or the run's typical scale when the neighbourhood holds nothing credible either. The bound is a RATIO to the run's own median because the rotation per-frame G is not gauge-fixed: G and the group means have an exact global multiplicative degeneracy, and the fitted median drifts over 0.745-1.358 across the battery. An absolute floor would reject everything in a run that drifted low. MIN_CREDIBLE_SCALE_RATIO = 0.02 was chosen from measurement over 12 crystals in the default configuration, where the smallest legitimate min(G)/median(G) is 0.070; the failing case sat at 0.0017. It is 3.5x below anything real and 12x above the failure. Effect on the intensity tail of the failing case: max I 10224 -> 438, and the top 1000 observations' share of sum(I^2) 0.990 -> 0.421 (the classic-finder reference is 0.632, so the tail is now cleaner than the run this was compared against). Rotation battery, 33 crystals in the default configuration: ZERO crystals differ - no space group, CC1/2, high-shell CC or ISa change anywhere. The guard fires only on the pathology. It does NOT rescue that dataset: with the amplification gone its CC1/2 is 26.2% and R_meas 49.2% against the classic finder's 93.7% and 27.8%. Adaptive detection degrades those intensities for a second, independent reason that is still open. This commit removes a latent hazard for any run with a sparse stretch of frames; it is not the fix for that dataset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 22:43:58 +02:00
leonarski_fandClaude Opus 5	0cd8cb7ba3	Space-group search: name the veto that actually refused a promotion ... The refusal message fell through to the chi^2 branch whenever the systematic-b balloon veto was the binding test, so it reported a chi^2 ratio that did not justify the refusal at all - on one battery crystal it printed "merge chi^2 is 1.25x the subgroup's (bound 1.85)", i.e. a number comfortably inside its own bound, as the reason for processing in the lower symmetry. A diagnostic that names the wrong cause is worse than none: it sends the reader after the wrong statistic. Report the b test when it is what fired, with both b values and the bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 21:38:07 +02:00
leonarski_fandClaude Opus 5	3171b071e6	Space-group search: judge a promotion against its parent, not against the error model ... The point-group decision moved with the AMOUNT of data at fixed physics: a partially twinned trigonal crystal was promoted into the twin's holohedry whenever the search happened to see a larger first-pass merge, and kept its true subgroup when it saw a smaller one. Simulation over 6 noise draws with only the merge multiplicity varying: the twin is promoted 0/6 at multiplicity 2 and 6/6 at 18, while the genuine control is promoted 6/6 throughout. The cause is that every existing gate is a ratio to the merge error model - b_parent grows toward the true systematic scatter as sigma shrinks with 1/sqrt(N), while b_cand is already saturated by the twin's disagreement, so the ratio slides down through a fixed veto. The parent statistic moves with data amount and the candidate statistic does not. Gate promotions on the operator disagreement H = <|I1-I2|/(I1+I2)> instead, as the ratio of the operators a promotion ADDS to the parent group's own operators on the same reflections. There is no sigma in it, so it cannot drift with the error model, and the parent normalisation cancels data quality. Measured over 27 runs, 5 promotion types and 450-1800 images: genuine symmetry 0.862-1.219, merohedral twins 1.270-2.084. On the synthetic grid it is flat across a 9x change in multiplicity - genuine pinned at 1.00, twins 3-12x the bound - which is precisely the property the old gates lacked. chi^2 and the systematic-b stay as secondary vetoes; they protect against non-crystallographic pseudo-symmetry, which is where correlation-based scoring is weak. Pick the parent carefully: 422 has two maximal subgroups of order 4, and on a tetragonal crystal twinned by 2[100] the rival (222) is CC-confirmed too and CONTAINS the twin laws, so normalising against it hides the twin among the promotion's own real operators (ratio 8.19 against the true parent, 0.78 against the rival). Where several parents tie, judge on the most damning. Also: - Report a refused promotion instead of silently processing lower. Merging a twin in the twin's holohedry averages non-equivalent reflections into each other and cannot be undone from the output; keeping the subgroup costs only redundancy. The refusal names the group and the number that caused it. - Stop the twinning report from arguing in a circle. It ran after adoption and conditioned on the adopted group, so a promotion into a holohedral Laue class made it print "no merohedral twin law exists" - the test was conditioned on the decision it should audit. Twinning is now also measured on the subgroup merge before adoption, and the post-adoption text says when its own conclusion is not authoritative. - Compare PRIMITIVE cell volumes in the first-pass scheme tie-break. A centred setting's cell is an exact integer multiple of its primitive one (a rhombohedral lattice in hexagonal axes is exactly 3x), so the integer-supercell test fired on a pure setting difference and demoted a good scheme to a threefold-smaller merge - which is what let the twin see the small merge to begin with. Rotation battery, 33 crystals: point-group agreement 30/33 -> 29/33, one crystal moved. That crystal (P422 -> P222) is the one with the known unresolved integration defect where reflections near the rotation-axis plane are wildly mis-integrated; its symmetry mates genuinely disagree, and its lower-symmetry merge is measurably better (ISa 2.72 -> 3.63, high-shell CC 75.4 -> 86.0). The threshold was not moved to accommodate it: 1.25 sits inside the measured gap and widening it would admit real twins. Separately the tie-break improved one crystal's CC1/2 from 77.7 to 84.0. Tests: a synthetic twin-fraction x multiplicity grid, which is what the search had never had - the existing tests are noise-free and exercise only Stage B absences. A NOTE ON WHAT WAS TRIED AND REJECTED, so it is not rebuilt: the obvious "physics-anchored" statistic is the disattenuated cross-validated correlation rho = corr(I_half0(h), I_half1(Rh)) / corr(I_half0, I_half1), which is 1 for real symmetry at any data quality and 2a(1-a)/((1-a)^2+a^2) for a twin. It passes the synthetic grid perfectly and FAILS ON REAL DATA IN BOTH DIRECTIONS - five false refusals of genuine symmetry on the battery, and it waves through a twin (rho 0.998) that H refuses. The reason is that cc_half correlates the two halves of the SAME reflection and so measures only random error, while cc_cross compares DIFFERENT reflections carrying different systematic error; dividing by cc_half removes the noise and leaves a systematic floor that varies by crystal AND by operator. Genuine rho measures 0.9987 on strong data and 0.73 on weak. A synthetic generator validates a statistic's arithmetic, never its premise, and this premise - that the only departure from exact symmetry is noise - is false for every real crystal. Any per-operator agreement statistic needs a same-crystal reference; an absolute threshold on one cannot be made to work by tuning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 21:05:16 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8	014e43a4c9	Remove non-helping stills merge/scaling knobs ... Trims three opt-in stills parameters that did not improve data quality on the external-reference (PDB R-free) battery and only added code: - --partiality-uncertainty: the (1-p)/p merge-sigma term was null on all four serial-stills datasets of the battery vs their reference structures (and neutral-to-harmful at higher coefficients); removed the flag, setting and CorrectedSigma term. - --stills-modulation: the detector-plane flat-field surface was net-negative on flooded data; removed the flag, setting and MergeOnTheFly::RefineModulation (the rotation modulation in RotationScaleMerge is unaffected). - --min-indexed-fraction: every value other than the 0.20 default collapsed CC1/2; removed the override flag/setter, keeping the fixed 0.20 acceptance floor. Default behaviour is unchanged (all three were off / at their default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:40:48 +02:00
leonarski_fandClaude Opus 4.8	f72b4484e2	Remove dead per-crystal deltaCChalf rejection ... Drops the --reject-delta-cchalf flag and MergeOnTheFly::DeltaCChalfReject. The CLI value was parsed but never consumed (the method had no call site), so the flag was already a no-op. Wiring it up and testing against an external reference structure showed it is confirmation bias: on a spurious-crystal flood it raised internal CC1/2 while CCref (correlation to the true structure) fell, and it never improved R_meas. The merge weights are already correct; per-crystal merge-side rejection has no genuine lever here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 15:05:06 +02:00
leonarski_fandClaude Opus 4.8	1a2b0181a5	Add physical partiality post-refinement for stills (default on) ... Replace the frozen scalar-sigma stills partiality with a physical, refined model. Per crystal, refine an orientation tilt (dpsi_x, dpsi_y) against the running merge and recompute each reflection's partiality analytically from the refined geometry (angular Ewald-proximity model, sigma(d*) = gamma_e*d*), with the per-crystal scale G profiled out by the existing robust IRLS - no re-integration. A soft Gaussian prior on dpsi tames weak-data overfit while staying inert on strong data. The merge <-> refine loop iterates a few times. This is now the stills default via ScalingSettings::stills_partiality_refine (on). A single opt-out flag `--simple-stills` reverts to treating every reflection as a full (p=1, single pass). Retires the experimental `--still-partiality` flag. The viewer gains a "Partiality post-refinement (stills)" checkbox in Scaling settings. Validated (integrate-once / --scale): CC1/2 and R_meas both improve on three monochromatic serial-stills datasets (+2.8 / -10, +5.6 / -3.4, +2.1 / -4); neutral on a pink-beam DMM set (already-full reflections); R-free/R-work down vs a fixed model; competitive with CrystFEL partialator on matched frames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 14:45:43 +02:00
leonarski_fandClaude Opus 4.8	cbc6a85157	Debias stills merge with expected-variance weighting ... The serial-stills merge (MergeOnTheFly::CorrectedSigma) weighted each observation by 1/sigma^2 using the observation's OWN sigma. Below ~1 photon the Poisson signal part of that sigma correlates with the observation's up/down fluctuation, so the inverse-variance mean is biased low: an up-fluctuated observation acquires a larger sigma and is over-downweighted. The rotation combine (RotationScaleMerge:: process_rawrun) already avoids this by rebuilding the signal variance at the pooled estimate; the stills path did not. Decompose each observation's variance into a background/read part (kept per-observation) and a Poisson signal part, and rebuild the signal part at the reflection's expected <I>. Bit-identical when an observation sits at its reflection mean; only weak-shell weights move. Now default on, so the stills path matches the rotation path; --no-expected-variance-merge restores the old observed-sigma weighting. Validated by paired refinement (phenix, 5 free-set seeds, byte-identical free flags across arms): R-free-neutral on strong lysozyme and lower R-free on weak serial-stills data checked against an independent deposited model (6/6 seeds). The CC1/2 dip on strong data reflects precision, not accuracy. Applies to both offline rugnux and the online broker stills merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 22:50:25 +02:00
leonarski_fandClaude Opus 4.8	c52886c8ff	Guard degenerate asymptotic-ISa fit on low-multiplicity data ... On very-low-multiplicity data (e.g. EP_cs_01-24, mult ~1.4) the merge has too few symmetry equivalents to measure the asymptotic I/sigma: both the (a, b) error-model fit and the per-group strong-reflection scatter collapse toward zero, so 1/error_model_b_asymptotic either explodes to an impossibly high ISa (tiny positive b) or is left as 0. Real macromolecular data does not exceed ISa ~50, so clamp the reported asymptote at a generous cap (ISa 100) and treat anything past it as unmeasured (result.isa undetermined) rather than emitting a spurious extreme. No-op for all well-measured data (b_asy well above the cap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-21 23:02:40 +02:00
leonarski_f	67dca388bd	v1.0.0-rc.160 (#70) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-19 09:39:28 +02:00
leonarski_f	dd0bffb283	v1.0.0-rc.159 (#69) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell. * rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged. * rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set. * rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme. * rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free. * rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry. * Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md. * Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #69 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-13 13:54:03 +02:00
leonarski_f	451310f43d	v1.0.0-rc.158 (#68) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * Analysis: The azimuthal-integration solid-angle correction now follows the incidence angle to the detector normal (`cos^3` of that angle) instead of `cos^3(2*theta)`, so it is correct for a tilted detector and matches PyFAI `solidAngleArray` and MAX IV azint (unchanged for an untilted detector). Crystal geometry refinement (`XtalOptimizer`) no longer silently ignores an imported PONI `rot3` (rotation about the beam): it is applied as a fixed rotation in the residual so refinement stays consistent with the rest of the pipeline. Polarization and azimuthal binning already honoured `rot3` through the full PONI rotation. * jfjoch_viewer: Open datasets on the WSL2/UNC filesystem (paths starting `\\`); write processing outputs next to the input file, with a Browse button and independent `_process.h5` / merged `.mtz`/`.cif` toggles; and show the determined space group in the merge-statistics window. * rugnux: Accept an absolute `-o` output prefix in offline processing. * Packaging: The self-contained Linux viewer `.tgz` now bundles cuFFT, so it runs without a system CUDA toolkit (`.deb`/`.rpm` are unchanged, distro-managed). * Docs: Bring the analysis references up to date with the code. `docs/CPU_DATA_ANALYSIS.md` now reflects the unified profile-fit Bragg integration engine, multi-lattice indexing, azimuthal phi binning, the radial parallax/bandwidth profile with sub-pixel centring, the rot3d capture-fraction handling and the automatic CC1/2 resolution cutoff, and drops the descriptions of features that were never implemented (French-Wilson amplitudes, the still excitation-error partiality model); `docs/RUGNUX.md` documents the new `--resolution-cutoff`/`--resolution-cc-target`/`--resolution-shells`, `--min-captured-fraction`, `--mosaicity`, `--reference-column`, the azimuthal correction toggles and the geometry-override options, and corrects the `-N` default. The outdated in-source design notes (ICE_RING_DETECTION, BRAGG_INTEGRATION_ENGINE, NEXTGEN_INTEGRATOR) are removed.Reviewed-on: #68 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-12 19:42:29 +02:00
leonarski_f	54c0100e8e	v1.0.0-rc.157 (#67) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * rugnux: Rebrand the offline data-processing subsystem as `rugnux` and consolidate all offline analysis into the single `rugnux` binary - `jfjoch_process` is now `rugnux`, the former `jfjoch_azint` is now `rugnux --azint-only`, and `jfjoch_scale` is now `rugnux --scale` (see the new docs/NAMING.md and docs/RUGNUX.md). Scaling and merging are on by default for rotation and stills (`--no-merge` disables them), replacing the previous opt-in `-M, --scale-merge`. * rugnux: CLI fixes - default `-N` to all hardware threads, parse numeric option arguments strictly (reject non-numeric or trailing input instead of silently yielding 0), require `--wavelength > 0`, and correct the reproduced command line and `--scale` reference-cell handling. * rugnux: De-novo space-group improvements - recover genuine high symmetry and centred Bravais lattices from intensities, add an automatic CC1/2 high-resolution cutoff, and report L-test twinning statistics. * rugnux: Index weakly-diffracting low-resolution rotation data that previously failed (e.g. F-cubic crystals that diffract only to ~4 A on a detector reaching ~1.5 A). The per-frame indexing gate now measures the indexed fraction only within the resolution range the lattice actually diffracts to, so the many sub-diffraction ice/noise spots no longer make the fraction floor unreachable; the two-pass first pass tries several image-sampling schemes (spread across the whole rotation vs a consecutive wedge whose native stride keeps a reflection's rocking curve continuous, letting the FFT resolve a long axis) and keeps the one that indexes the most frames; and the de-novo space-group search no longer discards all reflections (and crashes) when every resolution shell falls below <I/sigma> = 1. * rugnux: Lower the low-resolution R-meas for strongly-diffracting rotation data - drop edge-of-sweep truncated fulls whose rocking curve was captured below `--min-captured-fraction` (default 0.7 for rotation), and report R-meas only over the observations kept by outlier rejection (matching XDS). The 0.7 default also strips the partiality-extrapolated fulls that dominate the intensity second moment on weakly-diffracting crystals, so the de-novo space-group search is no longer starved by the error-model I/sigma floor and recovers the correct symmetry (e.g. the F-cubic Benas crystals: Benas_3 -> F432, Benas_7 -> P6122, instead of P4/P1); on the reference battery every other crystal keeps its space group. * rugnux: Write the refined geometry (beam, tilt, axis) to _process.h5 and place non-standard mmCIF items under a reserved `jfjoch` prefix. * jfjoch_broker: Ordinary acquisition failures (receiver/writer/analysis problems, missed packets, writer disconnect) now return to the Idle state with an Error-severity message, so a run can be retried without an expensive re-initialisation; only failures that leave the detector in an undefined state (new JFJochCriticalException, e.g. PCIe/FPGA faults) go to the Error state and force re-initialisation. * jfjoch_broker: A synchronous /start now reports its failure to the HTTP caller instead of returning HTTP 200, and an incomplete or truncated dataset (missing packets, writer disconnect) is reported as an error rather than a "reduce frame rate" warning. * jfjoch_broker: Drop uncollected placeholder rows (number = -1) from the scan_result REST endpoint. * jfjoch_broker: Fix the inverted per-image compression ratio reported by the Lite receiver (was compressed/uncompressed instead of uncompressed/compressed). * jfjoch_broker: Bragg integration adds a quantization-noise variance floor with a box-sum fallback, and treats the type-maximum marker as an invalid pixel for unsigned image types. * jfjoch_writer: Detect file-overwrite conflicts at start for back-channel transports, and reset the writer when end-of-collection finalisation fails. * jfjoch_viewer: Preview overlays follow the geometry (resolution/ROI arcs, true beam centre, predictions, coral secondary-lattice spots, legend), add save-as-JPEG, and fix an HTTP live-follow memory leak. * Frontend: Improved aesthetics and usability, and added in-browser pixel-mask and JUNGFRAU-pedestal visualisation. * CI: Name the Windows installer jfjoch-viewer-* instead of jfjoch-*.Reviewed-on: #67 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-11 07:19:11 +02:00
leonarski_f	d6389e12da	v1.0.0-rc.156 (#66) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Major rotation (rot3d) data processing overhaul - robust profile-fit integration, Cauchy-loss scaling with optional absorption surface, de-novo indexing and space-group/centering determination fixes, and merging statistics + ISa in the mmCIF output. * jfjoch_process: Add EXPERIMENTAL ice-ring detection (--detect-ice-rings) that excludes ice reflections from scaling. * Compression: Add BSHUF_ZSTD_RLE_HUFF, make compression size-aware (drop frames that don't fit rather than aborting), and add the jfjoch_recompress tool. * jfjoch_viewer: Report "Multiple lattices detected" and grey out "Analyze dataset" on a live connection. * jfjoch_broker: Write smargon chi/phi goniometer positions to NXmx; read sensor thickness/material from HDF5 metadata. * CI: Build Windows (CUDA and non-CUDA) installers.Reviewed-on: #66 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-07-03 19:18:56 +02:00
leonarski_f	54c667190f	v1.0.0-rc.155 (#65) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_process: Remove pixelrefine option (replaced with ProfileIntegrate2D) * jfjoch_viewer: Some graphical improvements. * jfjoch_viewer: Simplify und unify data analysis settings. * jfjoch_writer: Add TCP keepalive to increase robustness if jfjoch_broker "dies" in the middle of data acquisition. Reviewed-on: #65	2026-06-25 22:01:48 +02:00
leonarski_f	75e401f0e5	v1.0.0-rc.153 (#63) ... This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing * jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files * jfjoch_broker: Add ROI calculation in non-FPGA workflow * jfjoch_broker: Fixes to TCP image pusher * jfjoch_broker: Remove NUMA bindings * jfjoch_broker: Improvements to indexing * jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values * jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors * jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs * jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results Reviewed-on: #63	2026-06-23 20:29:49 +02:00
leonarski_f	fc68a9baed	v1.0.0-rc.146 (#56) ... This is an UNSTABLE release. The release has significant modifications for data processing - in case of troubles go back to 1.0.0-rc.144. jfjoch_process: Generate a dedicated file (_process.h5), which can be used as a replacement for the _master.h5 file for a reanalyzed dataset. jfjoch_process: Improve the performance of scaling and merging, implement on the fly scaling. jfjoch_writer: All final data analysis results are repopulated in the _master.h5 file. jfjoch_scale: Dedicated tool for rescaling/merging existing data. jfjoch_viewer: Fix bugs where pixel labels where displayed on a wrong pixel. WARNING! Scaling and merging are experimental at the moment, and may not provide reasonable results for the time being. Reviewed-on: #56	2026-05-28 18:48:35 +02:00
leonarski_fandtakaba_k	230480e390	v1.0.0-rc.138 (#48) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.132. jfjoch_broker: Cleanup DECTRIS start-up code to enable a shorter start time jfjoch_broker: Allow for asynchronous start to allow overlapping detector configuration with other beamline preparations jfjoch_broker: Goniometer axis name is converted to lowercase jfjoch_broker: Fix bug, where wrong HTTP error codes were returned jfjoch_broker: Improve sigma estimation during merging (K. Takaba) --------- Co-authored-by: takaba_k <kiyofumi.takaba@psi.ch> Reviewed-on: #48 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-04-27 19:56:14 +02:00
leonarski_f	166fcdb68f	v1.0.0-rc.131 (#39) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Fix bug in saving JUNGFRAU calibration (pedestal/pedestalRMS) * jfjoch_viewer: Fix calibration (pedestal) images being open flipped * jfjoch_process: Add space group detection (EXPERIMENTAL) Reviewed-on: #39	2026-03-07 11:34:04 +01:00
leonarski_f	64002f1e29	v1.0.0-rc.129 (#36) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Significant improvements in TCP image socket, as a viable alternative for ZeroMQ sockets (only a single port on broker side, dynamically change number of writers, acknowledgments for written files) * jfjoch_broker: Delta phi is calculated also for still data in Bragg prediction * jfjoch_broker: Image pusher statistics are accessible via the REST interface * jfjoch_writer: Supports TCP image socket and for these auto-forking option Reviewed-on: #36 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-05 22:13:12 +01:00
leonarski_f	3036b1d37f	v1.0.0-rc.128 (#35) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Handle properly reuse of image buffer locations * jfjoch_broker: Fix bug in counting idle slots * jfjoch_broker: Force obtuse angle for monoclinic cells * jfjoch_process: Change scaling refinement tolerance Reviewed-on: #35 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-03-03 22:24:44 +01:00
leonarski_f	f3e0a15d26	v1.0.0-rc.127 (#34) ... This is an UNSTABLE release. The release has significant modifications and bug fixes, if things go wrong, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Default EIGER readout time is 20 microseconds * jfjoch_broker: Multiple improvements regarding performance * jfjoch_broker: Image buffer allows to track frames in preparation and sending * jfjoch_broker: Dedicated thread for ZeroMQ transmission to better utilize the image buffer * jfjoch_broker: Experimental implementation of transmission with raw TCP/IP sockets * jfjoch_writer: Fixes regarding properly closing files in long data collections * jfjoch_process: Scale & merge has been significantly improved, but it is not yet integrated into mainstream code Reviewed-on: #34	2026-03-02 15:57:12 +01:00
leonarski_f	1ab257af6c	v1.0.0-rc.125 (#32) ... This is an UNSTABLE release. This version adds scalign and merging. These are experimental at the moment, and should not be used for production analysis. If things go wrong with analysis, it is better to revert to 1.0.0-rc.124. * jfjoch_broker: Improve logic on switching on/off spot finding * jfjoch_broker: Increase maximum spot count for FFBIDX to 65536 * jfjoch_broker: Increase default maximum unit cell for FFT to 500 A (could have performance impact, TBD) * jfjoch_process: Add scalign and merging functionality - program is experimental at the moment and should not be used for production analysis * jfjoch_viewer: Display partiality and reciprocal Lorentz-polarization correction for each reflection * jfjoch_writer: Save more information about each reflection Reviewed-on: #32 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-18 16:17:21 +01:00
leonarski_f	07fe4dd3bb	v1.0.0-rc.124 (#31) ... This is an UNSTABLE release. This version significantly rewrites code to predict reflection position and integrate them, especially in case of rotation crystallography. If things go wrong with analysis, it is better to revert to 1.0.0-rc.123. * jfjoch_broker: Improve refection position prediction and Bragg integration code. * jfjoch_broker: Align with XDS way of calculating Lorentz correction and general notation. * jfjoch_writer: Fix saving mosaicity properly in HDF5 file. * jfjoch_viewer: Introduce high-dynamic range mode for images * jfjoch_viewer: Ctrl+mouse wheel has exponential change in foreground (+/-15%) * jfjoch_viewer: Zoom-in numbers have better readability Reviewed-on: #31 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch> Co-committed-by: Filip Leonarski <filip.leonarski@psi.ch>	2026-02-01 13:29:33 +01:00