200 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	bf866a0d4c	CUDA: the engines' setup copies belong on the engine's stream ... Making the worker streams non-blocking removed the implicit ordering that the constructors were still relying on. Each engine uploads its static inputs - the pixel mask, the pixel-to-bin map, the corrections, the ROI map - with a blocking NULL-stream cudaMemcpy, and then reads them from kernels on its own stream. A pageable host-to-device cudaMemcpy returns once the source has been staged, with the DMA still in flight, and a non-blocking stream no longer waits for the NULL stream. The failure mode is a silently unapplied mask or a stale mapping, not a crash, so it would not have announced itself. Put them on the stream the engine already owns, and synchronise once at the end of the constructor - that is required for the preprocessor, whose source is a local vector, and leaves the others settled rather than in flight for the cost of one one-time sync. The GPU spot-finder test uploaded its image the same way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:51:22 +02:00
leonarski_fandClaude Opus 5	a2adc4e021	Stills scaling: an image whose scale collapsed is dropped, not merged unscaled ... Leaving it at G = 1 looked like the conservative choice and is the more damaging of the two errors. The per-image scale enters as rlp/(partiality*G) and multiplies intensity and sigma alike, so substituting 1 for a scale that was really 1/200 of the run median puts the intensities in 200x too low with sigmas 200x too low too - 1/G^2 times the weight they deserve. The merge cannot defend itself against that, because the number that is wrong is the number the weight is built from. And if the collapsed value was instead a failed fit, G = 1 merges the image mis-scaled by an unknown factor. Per-crystal scales on serial stills genuinely span orders of magnitude, unlike frames of one rotation sweep, so both readings are live. An image whose scale is not believable has no usable scale. Write NaN into its image_scale_corr, which every merge path already skips on, so it drops out of the merged intensities, the error model and the statistics consistently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:51:08 +02:00
leonarski_fandClaude Opus 5	da74197dea	Stills partiality: an unmeasurable CC is not a reason to adopt the refined tilt ... The "keep what the crystal came in with" gate required std::isfinite(cc) before it would reject, so a refined model whose CC could not be measured at all was adopted. ImageReferenceCC returns NaN when fewer than 20 reflections clear the partiality cut - which is exactly what a refinement that collapsed the partialities produces, since the cut is on the partialities it just rewrote. The gate therefore failed open on precisely the crystals it exists to catch, and wrote the NaN into image_scale_cc, on which --min-image-cc then drops the image from the merge, the error model and the statistics. Treat a CC that cannot be measured as worse than one that can, so the crystal is put back exactly as it arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:51:08 +02:00
leonarski_fandClaude Opus 5	196c72a7fe	Bragg prediction: the rotation GPU launch was one plane short in each direction ... The kernel guards against 2*max_hkl+1 and maps thread i to h = i - max_hkl, but the host launched a grid sized 2*max_hkl. The h = k = l = +max_hkl planes were therefore never launched while -max_hkl was, so the GPU predicted an asymmetric subset of what the CPU loop (inclusive on both ends) does. The same bug was fixed on the stills twin when the whole hkl range moved to the GPU; the rotation predictor kept the old expression. It only bites where the cell actually reaches |h| = 100 inside d_min - a ~150 A axis at 1.5 A - so most data never noticed. Over the 33-crystal rotation battery 29 crystals are bit-identical and 4 gain observations, all of them large-cell or high-resolution: +8519, +4693, +901 and +758 observations, with the high-shell CC1/2 up 15.0->15.1%, 52.0->52.2%, 76.3->76.6% and 51.6->52.1%. Nothing is lost anywhere, and R-meas and ISa move by at most 0.01. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-31 11:50:10 +02:00
leonarski_fandClaude Opus 5	2be8680422	CUDA: let worker streams run concurrently ... Every per-thread stream was created with cudaStreamDefault, and the 20 MB raw image upload went to the legacy NULL stream. A NULL-stream operation implicitly synchronises with every blocking stream in the process, so with one engine per worker thread no two workers' GPU work could ever overlap - the whole GPU pipeline ran serially however many threads were asked for. Create the streams non-blocking and put the upload on the engine's own stream. Measured on 2000 serial stills, interleaved, medians of three: 24.6 -> 19.2 s at -N 32 (-22%), 32.7 -> 21.0 s at -N 16 (-36%), CPU utilisation 436-570% -> 723-859%. Output bit-identical - same observations, uniques, completeness, R-meas, CC1/2, error model and cell. The stream is synchronised at the end of the same function, so the ordering the code relies on is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 23:05:11 +02:00
leonarski_fandClaude Opus 5	07607d3d47	Keep the online reflection cap where the transport can carry it ... Raising the per-image reflection limit to 65536 for offline reprocessing also raised the image-buffer headroom derived from it, and that headroom divides a FIXED total buffer - so every slot grew from compressed+4 MB to compressed+16.7 MB and the receiver's slot count, i.e. how much of a burst it can absorb, fell by about three. Online never needed the raised limit: measured on three serial stills datasets the worst frame predicts 1380 reflections, 14% of even the old cap. So split them, the same way the geometry refinement's stopping rule is split: online keeps the transport-sized 10000, offline gets the full 65536, and the buffer headroom derives from the online one. Both still come from BraggPrediction so the cap, the prediction and the headroom cannot drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 22:49:16 +02:00
leonarski_fandClaude Opus 5	3e56d96921	Stills partiality: adopt the refined tilt only when it fits better ... RefineOne re-measured the image's correlation to the reference after writing the refined partialities - because --min-image-cc drops images by it - and then ignored what it measured. A crystal the tilt model suits worse than the fixed partiality it replaces kept the refined model anyway, and the refinement is on by default. Compare against the CC the crystal arrived with and put it back untouched when the refinement does not improve it, which is the same state a crystal with too few reflections to fit ends in. Also four things noted in review and left until now: AdaptiveThresholdTest.cpp was listed twice in the test target, AdaptiveThreshold.h was the one header in image_analysis/spot_finding not in its library's source list, CLAUDE.md said update_version.sh rewrites VERSION when it only reads it, and the CHANGELOG did not mention that image_scale_b is gone from the plot_type enum - which breaks a client that asks for that plot. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 22:03:18 +02:00
leonarski_fandClaude Opus 5	b161da05c1	Geometry refinement: bound offline reprocessing by iterations, not the clock ... The per-image refinement stopped on a wall-clock budget (40 ms, and 20 ms for the rotation-only extra pass). Online that is exactly right - the budget is real and an image that overruns it costs the acquisition. Offline it means the same file refines to a different lattice depending on what else the machine was doing at the time, which is not a property reprocessing should have. Bound it by iteration count instead when the caller is offline. IndexAndRefine takes the workflow as a constructor argument: the receiver asks for the wall-clock bound, rugnux and the viewer get the reproducible one. 50 iterations is Ceres' own default; the per-image problem converges well inside it, so it bounds the pathological case rather than the normal one - measured on five battery crystals, every number is unchanged from the timed version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 20:42:04 +02:00
leonarski_fandClaude Opus 5	ade61eea60	Rotation: predict without truncating, and keep the better of the two passes ... Digging into the selection logic showed the caps were not deciding the science - the two-pass geometry post-refinement was, and the caps only fed it randomness. Caps. The prediction buffer now grows to whatever a frame predicts instead of keeping an arbitrary subset of it, and the per-image reflection limit is raised to 65536, with the image-buffer transport headroom derived from the same constant so the two cannot drift. Measured: bit-identical output on five battery crystals, because a normal cell never approached the old limits - only a large cell (~2.8e6 A^3, ~30000-44000 predictions per frame) ever did. Pass-2 guard. The refined pass is normally the better answer, which is why it is the canonical output, but it was adopted whatever it produced. On that same crystal it merged more unique reflections than its own cell can hold - completeness "117%", which is arithmetically impossible - while the header- geometry pass sat at 92.6% and CC1/2 0.98. Compare the two and, when the refined pass is not credible, go back to the header geometry and re-run so the canonical files are the ones that are kept. Both bounds are set where only a failure reaches them. Together on that crystal: 111639 unique against XDS's 118730 (was 88000-99000 and different every run), CC1/2 98.0% (was 96.9-97.7%), ISa 8.54, and two runs now agree bit for bit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 19:20:51 +02:00
leonarski_fandClaude Opus 5	46bb3bdbab	Bragg prediction: say so when a frame overflows the prediction buffer ... Found while chasing a 12% run-to-run spread in the merged reflection count of one crystal. The GPU kernels claim output slots with an atomicAdd and, on overflow, undid the increment with an atomicSub - so the counter saturated at the capacity and the host could not tell a full buffer from an overflowing one. Which reflections survived was then decided by CUDA block scheduling and changed every run. Measured on that dataset: every frame predicts 23000-44000 against a 20000 buffer, and the spread reached the merged output (161591 / 165193 / 166110 / 166479 unique across four runs of the same command). Single-threaded runs diverge too - this is entirely GPU-side. Stop clamping the counter, so the true number predicted reaches the host, and warn once per predictor when it exceeds the buffer. Which reflections are kept is unchanged: making that reproducible means deciding what to keep when a frame predicts more than the pipeline carries, and the obvious answers are worse - the capacity is not the real limit, kPredictionOutput (10000, selected by smallest excitation error) is, and on this crystal both a bigger buffer and a strided selection collapse the merge, because the rotation combine rebuilds fulls from exactly the partials that a smallest-excitation-error cut throws away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 18:22:39 +02:00
leonarski_fandClaude Opus 5	ffbf38d2ab	Rotation scaling: guard the per-frame scales whatever else is switched on ... The protection against a per-frame scale collapsing toward zero lived inside ComputeSmoothGWindow, so it only existed when smooth-G did: --smooth-g=0, a dataset whose oscillation width is unknown, and any caller that never sets a smoothing range - the viewer among them - merged with no guard at all. A collapsed G multiplies that frame's intensities by 1/G and its sigmas by the same factor, so nothing downstream can see it; the merge's n-sigma cut scales with the number that is wrong. Pull it out into ReplaceCollapsedScales, called unconditionally right after the partial scaling loop, and let the smooth-G window assume what it now guarantees instead of computing its own median and floor. The fulls guard built its median from every frame including those never fitted - those sit at the combine's corr = 1, so a run with many unfitted frames dragged the median toward 1 and the floor with it. It also reported the absolute amplification where the message says "below the run median". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 17:51:44 +02:00
leonarski_fandClaude Opus 5	a95aca382c	Remove dead code left behind by recent changes ... None of this has a reader: - ScalingSettings::scaling_regularize and its setter/getter - ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read), with the timing that only fed the latter - JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the comment claiming the retry uses it - the retry keys off initial_fit_done_ - JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include in a file that uses none - an unused gemmi::Op accumulator in the spindle-angle helper - <random> in Merge.{h,cpp}, from before the half-set split became a hash - an orphaned comment describing the Ceres B-factor residual deleted in 014e43a4c, and two trailing comments that had collided on one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:29:42 +02:00
leonarski_fandClaude Opus 5	f90969ea21	Stills partiality: do not adopt a tilt from a failed solve ... The Ceres summary was discarded, so a solve that diverged or aborted left its last iterate in psi and that tilt was written onto the partiality of every reflection of the crystal. Restore the tilt the crystal came in with and stop refining it; the scale fit alone is still a usable model, which is what the other three early returns in this function fall back to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:14:26 +02:00
leonarski_fandClaude Opus 5	3e8a994d2e	Stills scaling: leave an image unscaled when its scale collapses ... SolveScaleIRLS returns whatever it converged to and both writers accept any G > 0, so a fit that collapsed to ~1e-3 multiplies that image's intensities by a thousand. Nothing downstream notices, because the sigmas are multiplied by the same factor and the merge's n-sigma outlier test is therefore blind to it - only a total collapse self-heals, by overflowing corr to inf. The rotation path refuses a per-frame scale this far below its neighbours; the stills path had no guard. Judge each image against the median of the images that did scale, and put a collapsed one back to G = 1 - the same state as an image with too few reflections to fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:13:51 +02:00
leonarski_fandClaude Opus 5	af97ad61e3	Rotation scaling: the search-only filters must not outlive their pass ... Two faults in the same block, both of which let a search pass corrupt the production merge that follows it. The device's corr was only copied back to the host for the diagnostic dump, but the |zeta| filter runs on the host and then uploaded the whole host array - so on a CUDA build it wrote the values ingested BEFORE scaling over the scaled and smoothed corr the device had just computed. With the rotation default --search-min-zeta 0.85 that means the space-group search was deciding the symmetry from an unscaled merge. Copy corr back first, and upload once after both filters instead. Zeroing corr also has no owner: it is how an observation leaves the merge, but the only thing that ever rewrites it is the scaling loop, which skips frames it cannot fit. A frame left with too few well-measured reflections therefore kept its dropped observations at zero for the rest of the object's life - and the final production merge re-uses the same object without re-ingesting. Snapshot corr before the filters and restore it at the start of the next pass, so each pass decides for itself and the final merge keeps everything, as documented. The frame rejection (--min-image-cc) is now applied on the host for both paths; its separate device path did nothing whenever the CPU combine was in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:12:13 +02:00
leonarski_fandClaude Opus 5	04450eb618	Adaptive spot finder: sum the rings across blocks in double ... The ring sigma is the cancelling difference sum2/n - m^2, and both sums were float accumulated by atomics whose order is arbitrary. Two costs: the cancellation left only ~4 digits in the variance, and the ordering moved the resulting threshold by ~0.05 counts between runs - enough to flip a pixel sitting on the hard "value >= threshold" test, and with it a connected component's size. So the GPU engine did not reproduce the CPU one and did not reproduce itself. Only the accumulators that span blocks are widened. The per-block staging stays float, because a block contributes a few dozen similar-magnitude pixels to a ring and there is nothing to lose there - that also keeps the shared-memory footprint of the hot loop, and hence its occupancy, exactly as it was: measured on a 4.5 MP frame, 0.960 vs 0.966 ms/frame (40.9x over the CPU path, unchanged). finalize_rings now does the cancellation in double and rounds to float last, which is what AdaptiveSpotFinderCPU::AccumulateRings does. The device properties are also read from the current device rather than device 0; callers round-robin engines across GPUs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 11:05:54 +02:00
leonarski_fandClaude Opus 5	0076b76566	Bragg prediction: launch the whole hkl range on the GPU ... The host sized the grid from 2*max_hkl while the kernel guards against 2*max_hkl+1, so whenever the rounded-up grid landed exactly on 2*max_hkl threads (max_hkl a multiple of 4, with the 8x8x8 block) the h = +max_hkl plane was never launched. The CPU loop runs -max_hkl..+max_hkl inclusive, so the GPU predicted a strict subset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:56:18 +02:00
leonarski_fandClaude Opus 5	009555bc49	Spot finding: a zero high-resolution limit means no limit here too ... Every other reader of spot_finding.high_resolution_limit spells "unset" as value_or(0) and compares, so 0 and nullopt are interchangeable - except in SpotAnalyze, which passed the 0 straight to ResolutionShells and threw "Resolution must be above zero" on every image. Reachable over the REST API, where 0 is the natural way to say "no limit" and the settings check lets it through; the rugnux CLI already maps 0 to unset before this point. While here, check that a limit that IS set is finite regardless of its sign - NaN fails the > 0 test and was skipping validation entirely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:55:19 +02:00
leonarski_fandClaude Opus 5	64c8117a96	Space-group search: absences always come from the all-observation merge ... The two-arm search is meant to use the Lorentz-filtered merge for the point group only - systematic absences live in the weak reflections a |zeta| cut removes, and reading them off the filtered arm is what cost four crystals their screw axes. That is what the code comment and CPU_DATA_ANALYSIS both say, but the filtered-arm-wins branch kept its whole result, screws and centering included. Let a search be pinned to a point group decided elsewhere (fixed_point_group) and re-run Stage B on the all-observation merge when the filtered arm rescues the point group. The point group is passed as its symmorphic representative, not by name: gemmi calls both P321 and P312 "32". Reporting that representative also lets the ambiguity check see two arms that disagree about which 2-folds are real - by name they looked identical - and the advice it prints now names a space group -S can actually be given. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:54:02 +02:00
leonarski_fandClaude Opus 5	1355d6b2aa	Space-group search: a screw threshold needs a row with controls behind it ... The row-relative cut scales the "too strong to be absent" threshold by the axial row's own median E^2, floored at the plain value - so it can only raise the bar, and a row whose control class holds a single strong reflection sets it from that one reflection. That direction invents screws: a genuine 4_2 whose 00l happen to be observed only at l=4n reads its l=4n+2 reflections as absent and ranks 4_1/4_3 above the truth. Require three controls before the row may set the scale; below that the row keeps the plain cut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:52:32 +02:00
leonarski_fandClaude Opus 5	72cd91ccc3	Space-group search: judge a promotion against the worst of its tied parents ... A candidate can have several confirmed subgroups of the same order - 422 has both 4 and 222 - and on a twinned crystal the rival is not a harmless alternative: a P4 crystal twinned by 2[100] has 222 confirmed too, and 222 CONTAINS the twin laws, so its own merge b is already ballooned. The H test already answers to every tied parent; the systematic-b veto and rescue took whichever one the enumeration happened to list first (222 before 4, by space- group number), which disabled the veto on exactly the case it exists for. Take the smallest parent b, which is the conservative direction for both tests. The refusal message also quoted the raw parent b rather than the floored value the veto actually compared against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:52:05 +02:00
leonarski_fandClaude Opus 5	609650b061	Rotation merging: keep the systematic sigma floor when ISa is unmeasurable ... The cap that refuses to report an impossible ISa was zeroing the asymptotic b itself, and that same value is the floor passed to SigmaWithSystematicFloor - where zero means "no floor". So on the degenerate low-multiplicity fit the guard is written for, instead of capping merged I/sigma at 100 it removed the cap entirely. Report the asymptote as unmeasured, keep the fitted value for the floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:50:30 +02:00
leonarski_fandClaude Opus 5	386f10ad07	Merging: mirror the negative-intensity Poisson guard on the GPU ... 6be94f2be stopped subtracting a negative intensity's Poisson term from the background variance, but only in the host Combine(). The CUDA combine is the path that actually runs: Run() selects it whenever a device is present and no observation dump was asked for, so the correction never took effect on a normal run, and a --dump-observations run merged differently from a normal one - the two are meant to be identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-30 10:49:37 +02:00
leonarski_fandClaude Opus 5	6be94f2be0	Merging: do not subtract a negative intensity's Poisson term ... The expected-variance weights decompose an observation's sigma^2 into a background part and a Poisson signal part, then rebuild the signal part at the reflection's merged mean. The decomposition subtracted corr*I with I taken as-is, so a negative I ADDED to the background part: an observation at I = -1.5 with sigma^2 = 1 came out with a base variance of 2.7 rather than 1. That inflates the variance of precisely the down-fluctuated observations the correction exists for. Below about one photon they are then under-weighted and the merged mean is biased high - the same direction of error, in the same regime, that weighting by the observation's own sigma produces. Subtract max(0, I) instead: a negative intensity has no Poisson signal to remove. Both users of the decomposition are fixed - the stills merge, where expected-variance weighting is now the default, and the rotation combine it was mirrored from, which had it first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:36:40 +02:00
leonarski_fandClaude Opus 5	157698d31e	Space-group search: report the order of the point group it chose ... The two-arm search compared its arms by the order of the space group each had picked, but Stage B leaves best_space_group unset whenever no candidate is eligible - no candidate had enough observed absences to trust. That is not rare on the Lorentz-filtered arm, and for a systematic reason: the filter removes the badly-measured observations, which is where the weak systematically-absent reflections are. An arm that confirmed 422 but stopped short of naming a space group therefore scored order 0 and lost to an arm supporting P2, and the demotion was logged as "taking the higher symmetry" - the comparison and the message both wrong, in the one direction the design says cannot happen. Carry the point-group order in the result, set from the order Stage A actually adopted, and compare on that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 09:33:29 +02:00
leonarski_fandClaude Opus 5	d28db19ab1	Space-group search: judge a screw axis against its own axial row ... A reflection the group predicts absent counted as a violation when I/sigma > 3 AND E^2 = I/<I>(shell) > 0.3. Neither half survives contact with real data: * merged sigma is floored at b|I|, so merged I/sigma saturates at ISa for nearly every reflection - the I/sigma half is an on/off switch keyed on ISa vs 3, not a per-reflection test. On one crystal the absent class read <I/s> 4.10 against 3.73 for the present class while being genuinely extinct; * <I>(shell) decays with resolution while a systematically-absent reflection keeps a small NON-decaying residual (background / profile leakage), so absent reflections drift over an absolute E^2 cut at high resolution. That cost a tetragonal 42_12 crystal its 4_1: 18 of its 47 absent 00l crossed the cut, all beyond 3.7 A, at absolute intensities identical to the low-resolution ones correctly judged absent, while their l=4n row-mates sat 20-60x higher at the same resolution. A screw extinguishes only the reflections that lie ON its axis, so the fair yardstick is the rest of that same row. The threshold is now 0.3 * max(1, median E^2 of the reflection's own row), the row being the gcd-reduced reciprocal-space direction and the control class the same-row reflections the group predicts present. Floored at 1, so it only ever relaxes: a screw can be recovered by it, never lost. Per row, not pooled. A 4_1 along c and a 2_1 along a are separate conditions with separate controls; pooling let the weak a/b rows (median E^2 ~0.5) set the threshold for a strong c row (8.4) and the rescue never fired. The candidate table now reports the screw evidence (median E^2 of the absent class and of its rows) - the <I/s> columns are the centering evidence and say nothing about screws, for the sigma-floor reason above. Rotation battery, 33 crystals: 31 decisions bit-identical, the 42_12 crystal recovers its 4_1 (0 violations, row E^2 8.4 vs absent 0.12), and one crystal with a long axis and heavy 00l overlap moves to a 4_1 group at exactly 10.0% violations - marginal, and its sister crystal of the same form sits at 13.3% and does not move. Real screws now span 0-9.3% violations, so max_absent_violation_fraction cannot be tightened below 0.10 without risking a genuine one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 08:27:34 +02:00
leonarski_fandClaude Opus 5	25458265d3	Space-group search: ask twice - all observations, and only the well-measured ones ... --search-min-zeta rescues a point group that the full merge cannot confirm, but used on its own it is a trade: on the crystal it was built for it recovers the correct 422, and on four others it costs the space group outright, because discarding 40-80% of the observations starves operator correlations that were perfectly healthy. Both ways of applying it - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade the SAME crystals, so the cut itself is the problem, not where it is applied. Filip's observation makes it one-way: every disagreement between the two is a LOST operator, never an invented one. Discarding observations can starve a correlation; it cannot manufacture symmetry that is not there. So run the search on both merges and keep whichever found MORE symmetry, and the failure mode disappears - each arm rescues the other exactly where it fails. crystal all observations Lorentz-filtered adopted thaumatin (weak) 222 422 422 tetragonal lysozyme 422 222 422 cubic insulin x3 23 2 / 222 23 The filtered merge is used ONLY to rescue the point group. The screw and centering determination always comes from the merge with all the observations, because systematic absences are decided by the WEAK reflections and the filter throws most of them away. Preferring the filtered arm on a tie is not a conservative choice, it is a wrong one: it cost four crystals their screw axes (P2(1) read as P2, P4(1)2(1)2 as P42(1)2) with the point group and every intensity statistic identical - a regression invisible to CC1/2, R_meas and ISa. Where the two find the same ORDER but different symmetry, nothing can prefer one, so the run says so: it names both space groups, states that the data do not decide, reports which one processing continued in, and gives the flag to force the other. Two candidates of the same order imply different molecular replacement searches, and trying both is cheap next to reprocessing - much cheaper than a confident wrong answer. Rotation battery, 33 crystals, both spot finders: fixed-threshold finder 30/33 - ZERO crystals differ from the single search adaptive finder 30/33 - the same three mismatches, gap CLOSED The adaptive finder now matches the fixed-threshold one exactly, which it has not done before: its last remaining loss was the thaumatin set whose 4-fold sits 88.9 deg from the spindle, and it now reads P42(1)2 (all-observation merge -> 222, Lorentz-filtered -> 422, higher taken). A merohedral twin stays refused in BOTH arms at all three frame ranges where it over-promotes, and at one of them the second opinion is strictly better than shipping behaviour - the full merge collapses to P1 where the filtered one finds the correct H3. Cost is the extra scale-combine-merge on already-ingested partials, with no re-integration: 47.2 s against 47.8 s on the same crystal back to back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 21:11:05 +02:00
leonarski_fandClaude Opus 5	f2b92e3f4d	rugnux: --search-min-zeta drops badly-measured observations from the symmetry search ... zeta is the sine of the angle between a reflection's rocking path and the spindle. Near 0 the reflection crosses the Ewald sphere almost tangentially, spends many frames in diffracting position and is measured worst. The de-novo space-group search asks how EQUAL an operator's paired intensities are, so its answer is dominated by whichever reflections are measured worst - and when the spindle lies in a lattice plane, an operator that permutes the two in-plane axes samples a different mixture of measurement qualities than one that only flips signs. That is not a fair comparison, and it can make a real symmetry operator look like a twin law. Measured on a thaumatin set mounted that way (its 4-fold is 88.9 deg from the spindle), the added operators' disagreement is 1.74x the parent's over pairs where both reflections have zeta < 0.85 and 1.003x - i.e. the symmetry is exact - over pairs where both are above it. The search consequently refuses the 422 promotion and merges the crystal in P222, while the same data forced to the right group give CC1/2 99.2% at multiplicity 10.7, matching XDS. With the option the de-novo pass ignores those observations (the final merge keeps everything - there completeness is the point): zeta cut observations ignored H ratio adopted 0 (off) - 1.47 P222 0.5 1620648 1.44 P222 0.7 3006013 1.34 P21212 0.85 4536724 promoted P4212 (correct point group) OFF BY DEFAULT, and it must stay off, because the same cut costs four other crystals their space group (P41212 -> P212121, I23 -> P2, I23 -> I222 twice): at 0.85 it discards 40-80% of all observations, which on a crystal whose geometry is not the problem simply starves the search. Two independent implementations - filtering the pairs that enter the statistic, and filtering the observations that enter the merge - trade exactly the same crystals, so this is a property of the cut and not of where it is applied. Verified bit-identical to the previous binary when off. The companion diagnostic is already there: the run now reports how close a symmetry axis lies to the spindle, which is the geometry that makes this option worth reaching for. Implementation note for anyone tempted by the cheaper route: excluding these observations from the ASU grouping alone does NOT work. The 3D combine selects partials on corr, not on their group, so their intensity still reaches the fulls and the merged intensities are unchanged - measured, the statistic did not move by 0.03 while 67% of observations were nominally excluded. Zeroing corr is what removes an observation from the combine, the merge and the error model alike. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 20:03:43 +02:00
leonarski_fandClaude Opus 5	4895dc1018	Rotation: let --min-image-cc drop frames that disagree with the merged reference ... The flag was accepted on rotation data and did nothing - it is read only by the stills merge (Merge.cpp), and the CLI warned about that rather than fixing it. Meanwhile RotationScaleMerge already COMPUTES a per-frame correlation against the merged reference and writes it to the per-image table; nothing acted on it. Wire the two together. A rejected frame has its partials' corr set to 0, which is how a frame already leaves the pipeline - every consumer requires corr > 0, so the combine, the merge and the error model all drop it together. The GPU path reuses the SmoothCorr kernel with a ratio of 0, so one implementation covers both. Off by default (0), and verified bit-identical to the previous binary when off. What it catches, on the two rotation datasets that have a population to catch: a two-lattice crystal - two lattices in two physical AREAS of the sample, so the sweep passes from one to the other and whole blocks of frames measure a different crystal from the one being merged (frames 500-700 index perfectly well at a per-frame CC of 0.22 against 0.47-0.56 either side, in 11 contiguous runs). R_meas 28.6 -> 24.6%, CC1/2 93.6 -> 95.1, high-shell CC 23.4 -> 38.3. a second dataset with 9.5% of frames below CC 0.30: R_meas 24.3 -> 23.4%, CC1/2 92.6 -> 93.4. The criterion is "this frame disagrees with the merged reference", NOT "this frame is off-crystal". It happens to catch both, because a frame that measures nothing and a frame that measures a DIFFERENT crystal fail the same test, and it does not need to know which. For the two-area case that is a workaround, not a treatment: it recovers one crystal by discarding the other, where processing the two as separate sweeps would keep both. The frame-block structure is clean enough that such a split could be detected automatically. WHY THERE IS NO DEFAULT. The per-frame CC is not comparable between datasets - it is as much a measure of data quality as of frame validity. Measured medians across the battery run from 0.30 to 0.81, so one absolute bound removes 13 frames from one dataset and 584 of 1800 from another: battery at --min-image-cc 30, 33 crystals: no point group changed (30/33), four crystals clearly better (one +5.4 CC1/2 points, the two-lattice case above, and ISa gains of 1.3-4.6 on three others) - and one healthy crystal lost a third of its frames and with them its high-resolution shell (CC1/2_hi 26.2 -> 2.0). This is the same trap as an absolute bound on any per-operator or per-frame agreement statistic, and the same one the per-frame scale guard avoids by measuring against the run's own median. A principled version would cut on the SHAPE of the per-frame CC distribution - a dataset with a bad subpopulation is bimodal, a uniformly weak one is not - rather than on an absolute value. Until that exists this stays opt-in, and the per-image CC it keys on is already in the _image.dat table for anyone choosing a value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:45:49 +02:00
leonarski_fandClaude Opus 5	11c7cab2e5	Space-group search: take the operator disagreement as a median, not a mean ... A merohedral twin mixes EVERY reflection with its twin mate, so it shifts the whole distribution of |I1-I2|/(I1+I2). A minority of badly measured reflections shifts only the tail. The mean cannot tell those apart; the median is blind to the second and just as sensitive to the first. Measured on real crystals, moving the statistic from the mean to the median leaves genuine promotions where they are and pushes every twin up: genuine tetragonal 1.016 -> 1.013 genuine lysozyme 1.051 -> 1.067 genuine tetragonal 1.238 -> 1.231 twin (-e 1050) 1.272 -> 1.447 twin (-e 450) 1.280 -> 1.622 twin (full) 1.441 -> 1.522 twin (-e 600) 1.427 -> 2.010 The margin around the 1.25 bound widens from 2.7% (genuine 1.238 against twin 1.272 - uncomfortably tight for a decision that cannot be undone downstream) to 17.5% (1.231 against 1.447). The bound itself does not move. Rotation battery, 33 crystals in both detection modes: no point group changed in either (30/33 and 29/33, as before), and only one crystal's numbers move at all - the one already documented as nondeterministic between repeat runs of the same binary. The synthetic twin-fraction x multiplicity grid passes unchanged. So this buys margin, not outcomes. Found while testing a different hypothesis, which the same measurement refuted: a tetragonal crystal whose 422 promotion is wrongly refused reads 1.484 by the mean and 1.472 by the median, i.e. its disagreement is distribution-wide and is NOT a badly-integrated minority. That crystal's cause is elsewhere and is not addressed here - see the note below. Its indexing-ambiguity operator (-k,-h,-l) lies INSIDE 422 but OUTSIDE 222, so the subgroup merge the search is given mixes lattices indexed in the two alternative hands. That corrupts exactly the 4-fold relationships and leaves the 2-fold ones intact - measured, the 222 step reads 0.917 and the 422 step 1.484 - and the corruption is indistinguishable from a twin law. Forcing the tetragonal group merges the two hands as equivalent and the same data give CC1/2 99.2% at multiplicity 10.7, matching XDS. The failure is worse the BETTER the frames index (99.9% vs 63.3% for the run that gets it right), because indexing more frames picks up more of both hands. So no statistic computed on a subgroup merge can arbitrate a promotion whose added operators include an indexing-ambiguity operator. Fixing that means resolving the ambiguity before the search, or detecting the coincidence and deciding another way; the operators needed to detect it are already computed (the run warns about them). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 11:07:06 +02:00
leonarski_fandClaude Opus 5	ae126c3d5b	Per-image refinement: weight each spot by how strong it is for its resolution ... `RefineGeometryIfNeeded` hands XtalOptimizer the WHOLE spot list, not the indexed subset, and the first pass admits anything within 0.3 fractional-Miller units of an integer - which is 11.3% of RANDOMLY placed spots, since the admitted volume is (4/3)*pi*t^3. Every one of them then enters an unweighted L2 fit with an arbitrary rounded index. On images with many detections the refined orientation ends up 2.3-2.8 degrees from the goniometer-consistent one and explains 14 of its own 250 spots where the undragged orientation explains 68; mosaicity and profile radius inherit the error and integration follows. Weight every spot by its intensity divided by the median intensity of its own equal-count resolution shell, applied as w^2 on the squared residual with w^2 = r/(1+r). The shell normalisation is the point: refinement needs the high-resolution spots because they carry the cell and distance, and those are LEGITIMATELY weaker, so a raw intensity weight would suppress exactly the spots the fit depends on. Measured, the weight is resolution-neutral - median exactly 0.707 in every shell, and corr(w, 1/d^2) = -0.20 / -0.11 against -0.32 / -0.34 for the same function of un-normalised intensity. This is a PRIOR: it is computed from the spot alone and never looks at the current residual, so unlike a robust loss it cannot mistake a genuine spot for an outlier while the starting geometry is still far off and leave the fit unable to move. That failure is not hypothetical - a CauchyLoss on this same residual, at the scale the multi-frame GeometryRefiner uses, collapsed one crystal's indexing rate from 99.89% to 19.83% and was rejected. It does not work by telling good spots from bad, and it does not need to. No per-spot property separates spots that index from spots that do not: measured AUC is 0.53 for peak pixel, 0.53 for total intensity, 0.51 for pixel count, 0.45 for peakedness, and a logistic regression on all twelve available features with pairwise interactions reaches only 0.64. What the weight does is halve the EFFECTIVE COUNT of every spot (mean w^2 = 0.517), and the damage scales with the absolute count of unexplained spots in the objective - 80.6 per frame here against 36.8 for the finder that was never damaged. That is also why an empirical `--max-spots 66` cap works while leaving the list no purer than before: it reaches the same operating point by discarding spots. This reaches it without discarding any, and without a tuned constant. Rotation battery, 33 crystals, both spot finders: finder A 29/33 -> 30/33 point groups (one crystal P222 -> P4212 = XDS, its high-shell CC1/2 86.0 -> 98.4) finder B 28/33 -> 29/33 point groups (one crystal I222 -> I23, its high-shell CC1/2 14.8 -> 38.0) No crystal lost its point group in either mode and no run failed. On the meta-stable multi-lattice dataset the CC1/2 spread over four frame ranges falls 19.7 -> 13.1 for finder B, and the indexing rate rises in 8 of 8 configurations. The crystal that the rejected robust loss destroyed keeps its 99.89% indexing rate exactly. The cost, stated plainly: ISa falls by 0.2-1.7 on about five crystals (and rises on two). Point-group correctness is worth more than that - merging in the wrong symmetry cannot be undone from the output, whereas ISa is a quality metric of data that remain correct - but it is a real trade and not a free win. Off by default. The indexers pass a spot list they have already selected, so their calls are unchanged; only the per-image refinement, which gets the raw list, turns it on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-28 09:53:13 +02:00
leonarski_fandClaude Opus 5	ec7a826136	Rotation scaling: guard the fulls refit against a collapsed per-frame scale too ... 704098712 guarded the per-frame scale on the partials, but it put the check inside ComputeSmoothGWindow - which only the partials path calls. Step 4 refits the scale from scratch on the COMBINED FULLS (Unity model, so corr is exactly 1/G), with no smoothing and no floor, and that refit was still free to collapse toward zero. It is the same failure and it is worse here, because there is no window average to dilute it: the collapsed frame's own fulls are multiplied directly. Measured on a dataset where the previous commit had already fixed the partials stage, the fulls refit put 1/G = 559x and 175x on two frames carrying 517 observations, and the merged CC1/2 read 26.2% where the intensities ENTERING that stage were fine - better, in fact, than the comparison run's in all ten resolution shells (R_meas 34.5% vs 39.5%, CC1/2 89.1% vs 83.0%). Reject a collapsed scale here as well, on the same measured criterion, and let those fulls merge unscaled - the state the combine left them in, and the same fallback the fit already uses for a frame with too few reflections. The check reads the host fulls after both the CPU loop and the GPU ScaleFulls, so one implementation covers both paths; the corrected corr is pushed back to the device exactly as the correction surfaces already do. THIS IS NOT A DETECTION-MODE PROBLEM. Over 8 configurations (both spot finders x 4 frame ranges) the separation is exact: every run with a collapsed scale had CC1/2 <= 58.5%, every run without had CC1/2 >= 74.1%, and nothing else predicted it. On one frame range it is the DEFAULT finder that collapses (CC1/2 58.5%) while the other is clean at 93.8%. The instability was never specific to the finder; it was latent in the scaling stage and either finder could trip it. Same 8 configurations, with this commit: finder A full 93.7 -> 93.7 (untouched) finder A -s 1 58.5 -> 91.3 (recovered) finder A -e 899 94.0 -> 94.0 (untouched) finder A -e 898 87.2 -> 87.2 (untouched) finder B full 26.2 -> 91.1 (recovered) finder B -s 1 93.8 -> 93.8 (untouched) finder B -e 899 8.1 -> 91.7 (recovered) finder B -e 898 74.1 -> 74.1 (untouched) Every collapse recovers; every healthy run is unchanged. The CC1/2 spread over the four frame ranges falls from 35.5 to 6.8 points for one finder and from 85.7 to 19.7 for the other - this dataset was not sampling a deep instability when its CC1/2 swung between 17 and 94 across frame ranges, it was sampling whether this bug happened to fire. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 23:08:50 +02:00
leonarski_fandClaude Opus 5	7040987125	Rotation scaling: do not trust a per-frame scale that has collapsed toward zero ... The per-frame scale enters every intensity as 1/G, and SolveScaleIRLS floors G at zero and nothing else. A frame whose fit is not determined by its data can return G ~ 0.002 against a run median of 0.865, and every observation it carries is then multiplied by ~500 - sigma by the identical factor, which is why no sigma-based outlier test can see it and why this looked for a long time like a partiality problem. (The 1/partiality path is in fact guarded: min_captured_fraction floors it at 0.7 by default on rotation.) The window smoothing that should have absorbed such a frame instead made it permanent. It averages log G over a window, so a scale collapsing toward zero does not merely corrupt its own frame - its logarithm drags the whole window down. Worse, where a run has a stretch of frames too sparse to fit at all, the only FITTED frames in a window can be the collapsed ones, and the geometric mean then averages the fault with itself. Measured on a multi-lattice dataset: frames 816 and 818 fitted G = 0.0023 and 0.0014 with every neighbour from 800 to 839 unfitted, so smoothing set G = 0.0018 across the whole neighbourhood - a 546x amplification. About 500 observations of 152000 (0.66%) then carried 99% of sum(I^2), and the merged CC1/2 read 17.2% where the same data with the classic finder read 93.7%. Treat a fitted scale far below the run's median as what it is - an undetermined scale, exactly like the too-few-reflections case the code already handles - rather than as a successful fit. Such frames no longer contribute to the smoothing mean, and a frame whose own scale is not credible takes the neighbourhood's, or the run's typical scale when the neighbourhood holds nothing credible either. The bound is a RATIO to the run's own median because the rotation per-frame G is not gauge-fixed: G and the group means have an exact global multiplicative degeneracy, and the fitted median drifts over 0.745-1.358 across the battery. An absolute floor would reject everything in a run that drifted low. MIN_CREDIBLE_SCALE_RATIO = 0.02 was chosen from measurement over 12 crystals in the default configuration, where the smallest legitimate min(G)/median(G) is 0.070; the failing case sat at 0.0017. It is 3.5x below anything real and 12x above the failure. Effect on the intensity tail of the failing case: max I 10224 -> 438, and the top 1000 observations' share of sum(I^2) 0.990 -> 0.421 (the classic-finder reference is 0.632, so the tail is now cleaner than the run this was compared against). Rotation battery, 33 crystals in the default configuration: ZERO crystals differ - no space group, CC1/2, high-shell CC or ISa change anywhere. The guard fires only on the pathology. It does NOT rescue that dataset: with the amplification gone its CC1/2 is 26.2% and R_meas 49.2% against the classic finder's 93.7% and 27.8%. Adaptive detection degrades those intensities for a second, independent reason that is still open. This commit removes a latent hazard for any run with a sparse stretch of frames; it is not the fix for that dataset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 22:43:58 +02:00
leonarski_fandClaude Opus 5	0cd8cb7ba3	Space-group search: name the veto that actually refused a promotion ... The refusal message fell through to the chi^2 branch whenever the systematic-b balloon veto was the binding test, so it reported a chi^2 ratio that did not justify the refusal at all - on one battery crystal it printed "merge chi^2 is 1.25x the subgroup's (bound 1.85)", i.e. a number comfortably inside its own bound, as the reason for processing in the lower symmetry. A diagnostic that names the wrong cause is worse than none: it sends the reader after the wrong statistic. Report the b test when it is what fired, with both b values and the bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 21:38:07 +02:00
leonarski_fandClaude Opus 5	3171b071e6	Space-group search: judge a promotion against its parent, not against the error model ... The point-group decision moved with the AMOUNT of data at fixed physics: a partially twinned trigonal crystal was promoted into the twin's holohedry whenever the search happened to see a larger first-pass merge, and kept its true subgroup when it saw a smaller one. Simulation over 6 noise draws with only the merge multiplicity varying: the twin is promoted 0/6 at multiplicity 2 and 6/6 at 18, while the genuine control is promoted 6/6 throughout. The cause is that every existing gate is a ratio to the merge error model - b_parent grows toward the true systematic scatter as sigma shrinks with 1/sqrt(N), while b_cand is already saturated by the twin's disagreement, so the ratio slides down through a fixed veto. The parent statistic moves with data amount and the candidate statistic does not. Gate promotions on the operator disagreement H = <|I1-I2|/(I1+I2)> instead, as the ratio of the operators a promotion ADDS to the parent group's own operators on the same reflections. There is no sigma in it, so it cannot drift with the error model, and the parent normalisation cancels data quality. Measured over 27 runs, 5 promotion types and 450-1800 images: genuine symmetry 0.862-1.219, merohedral twins 1.270-2.084. On the synthetic grid it is flat across a 9x change in multiplicity - genuine pinned at 1.00, twins 3-12x the bound - which is precisely the property the old gates lacked. chi^2 and the systematic-b stay as secondary vetoes; they protect against non-crystallographic pseudo-symmetry, which is where correlation-based scoring is weak. Pick the parent carefully: 422 has two maximal subgroups of order 4, and on a tetragonal crystal twinned by 2[100] the rival (222) is CC-confirmed too and CONTAINS the twin laws, so normalising against it hides the twin among the promotion's own real operators (ratio 8.19 against the true parent, 0.78 against the rival). Where several parents tie, judge on the most damning. Also: - Report a refused promotion instead of silently processing lower. Merging a twin in the twin's holohedry averages non-equivalent reflections into each other and cannot be undone from the output; keeping the subgroup costs only redundancy. The refusal names the group and the number that caused it. - Stop the twinning report from arguing in a circle. It ran after adoption and conditioned on the adopted group, so a promotion into a holohedral Laue class made it print "no merohedral twin law exists" - the test was conditioned on the decision it should audit. Twinning is now also measured on the subgroup merge before adoption, and the post-adoption text says when its own conclusion is not authoritative. - Compare PRIMITIVE cell volumes in the first-pass scheme tie-break. A centred setting's cell is an exact integer multiple of its primitive one (a rhombohedral lattice in hexagonal axes is exactly 3x), so the integer-supercell test fired on a pure setting difference and demoted a good scheme to a threefold-smaller merge - which is what let the twin see the small merge to begin with. Rotation battery, 33 crystals: point-group agreement 30/33 -> 29/33, one crystal moved. That crystal (P422 -> P222) is the one with the known unresolved integration defect where reflections near the rotation-axis plane are wildly mis-integrated; its symmetry mates genuinely disagree, and its lower-symmetry merge is measurably better (ISa 2.72 -> 3.63, high-shell CC 75.4 -> 86.0). The threshold was not moved to accommodate it: 1.25 sits inside the measured gap and widening it would admit real twins. Separately the tie-break improved one crystal's CC1/2 from 77.7 to 84.0. Tests: a synthetic twin-fraction x multiplicity grid, which is what the search had never had - the existing tests are noise-free and exercise only Stage B absences. A NOTE ON WHAT WAS TRIED AND REJECTED, so it is not rebuilt: the obvious "physics-anchored" statistic is the disattenuated cross-validated correlation rho = corr(I_half0(h), I_half1(Rh)) / corr(I_half0, I_half1), which is 1 for real symmetry at any data quality and 2a(1-a)/((1-a)^2+a^2) for a twin. It passes the synthetic grid perfectly and FAILS ON REAL DATA IN BOTH DIRECTIONS - five false refusals of genuine symmetry on the battery, and it waves through a twin (rho 0.998) that H refuses. The reason is that cc_half correlates the two halves of the SAME reflection and so measures only random error, while cc_cross compares DIFFERENT reflections carrying different systematic error; dividing by cc_half removes the noise and leaves a systematic floor that varies by crystal AND by operator. Genuine rho measures 0.9987 on strong data and 0.73 on weak. A synthetic generator validates a statistic's arithmetic, never its premise, and this premise - that the only departure from exact symmetry is noise - is false for every real crystal. Any per-operator agreement statistic needs a same-crystal reference; an absolute threshold on one cannot be made to work by tuning. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 21:05:16 +02:00
leonarski_fandClaude Opus 5	c9b52857e0	rugnux: warn when --min-image-cc is ignored; drop a dead robust estimator ... --min-image-cc is consumed only by the stills merge (MergeOnTheFly); RotationScaleMerge never reads it. On rotation data it was accepted and then silently did nothing, so a run that looked filtered was not. It now says so. FitProfileRadius_MAD had zero callers - a robust twin sitting uncalled next to the non-robust estimator that is actually used is a trap, so it goes. Neither changes any result: verified on a rotation dataset (indexing rate, cell, space group and merge statistics identical, warning emitted). Context for anyone tempted to wire that estimator in: I tested exactly that today and it is NOT justified. The population it would clip is truncated by construction - a spot is only marked `indexed` when its fractional-Miller norm is inside the indexing tolerance - and is measurably shorter-tailed than Gaussian (kurtosis 2.85). Across four serial-stills datasets a MAD-clipped variant only narrowed the prediction window (-17% integrated reflections everywhere), which was neutral on strong data and destroyed real signal on weak data (one set lost completeness 96.0 -> 93.9%), with R-free 0.3753 -> 0.3767. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 10:46:52 +02:00
leonarski_fandClaude Opus 5	16bf3408f0	Address code-review findings; make detection limits detector-driven ... One changeset, developed together in response to a review of this branch, so the files carry several of the changes at once. Full test suite passes (733 cases). Spot finding - Split ImageSpotFinder into Detect() (flag strong pixels - the expensive per-pixel pass) and ExtractSpots() (CCL + min/max-pix + resolution mask), with Run() = both. The per-image min-pix escalation now detects ONCE and repeats only the cheap extraction, instead of re-running the whole finder four times per frame as it did on the default path. It also keeps the winning attempt's spot list rather than re-extracting it, so the frame that is integrated is exactly the frame that was scored - which a GPU re-extract could not guarantee (float atomic ordering). - spot_finding_time_s no longer swallows indexing time, and indexing_time_s now sums every escalation call instead of reporting only the last. Detection limits follow the detector - The azimuthal-integration upper q and the spot-finding high-resolution limit are now std::optional, in the C++ structs AND in the OpenAPI schema, and resolve to the detector's own maximum (DiffractionExperiment::GetDetectorMaxQ_ recipA). Adaptive detection reads a pixel's ring from the azimuthal bins, so a pixel outside that q range could never be strong - the integration range silently bounded what detection could see, regardless of the requested resolution limit. Regenerated the C++ and TypeScript clients; the viewer and the web frontend each gained a "to detector edge" switch. Detection defaults are now per workflow (measured, not assumed) - Stills: adaptive detection, min-pix chosen per image, no resolution clipping. - Rotation: fixed-threshold finder, min-pix 2, 1.5 A limit. On a 33-crystal rotation battery, adaptive detection helped four hard crystals but deterministically broke three (a lost space group, a halved indexing rate, a collapsed merge), and the detector-edge limit cost indexing on a strong rotation set (100.0 -> 96.8%). Each is still overridable by its flag, and --no-adaptive-spots is new. Indexer seed escalation - Stop escalating once a seed's lattice explains >= 90% of the seed spots. Previously any frame with >= 80 spots always paid three indexer calls, online broker included. Merge-consistency filter - --min-image-cc gated on a per-image CC computed BEFORE the stills partiality post-refinement and never refreshed; the refiner now recomputes it, so the reported CC describes the data that are actually merged. - Replaced the per-call cc_mask argument with one MergeOnTheFly flag, so the merge, the error model and MergeStats can no longer disagree about which images are in (the --scale path merged unfiltered while its statistics were filtered). Per-image B-factor refinement (-B) removed - Measured on four serial-stills datasets: it is a no-op where the per-image fit is well conditioned and actively harmful where it is not (CC1/2 -8.1, R_meas +23.2 on the weakest large-cell set, whose fits hit their [-50, 200] bounds on 14-25% of images). It had also been silently DISCARDED since the partiality post-refinement landed - reported but not applied. Rather than fix and keep a knob with no demonstrated benefit, the flag and the whole image_scale_b_factor chain are gone: setting, scaling fit, message field, CBOR, HDF5 write and read-back, per-image plot, OpenAPI enum, viewer column and checkbox, docs. ScaleOnTheFly no longer needs Ceres at all - the fit is a linear IRLS. (The Wilson per-image b_factor is a different quantity and stays.) Stills partiality width now fits both of its components - sigma^2 = gamma0^2 + (gamma_e*d*)^2 instead of a purely angular gamma_e*d* with gamma0 pinned to 0. Fitted per crystal by least squares of dist_ewald^2 on d*^2. The angular-only width is fitted over a d*^2-dense population, so it was pinned by the high-resolution edge and collapsed at low d*: median partiality 0.008 beyond 13 A for reflections that were plainly recorded, 55% of them under the merge's partiality floor, and the survivors divided by those values - which inflated the merged low-resolution intensity scale 3.6x (~ +9 A^2 of apparent B). Measured on 5000 stills: the ramp flattens to 0.89x, no observation is dropped any more (701750 -> 716811), shell-mean CC1/2 and R-free improve slightly. Note CC1/2, R_meas, completeness and a B-refining R-free are all blind to that ramp, which is why it survived earlier validation; the cost is high-resolution R_meas (98.5 -> 101.9 shell-averaged). Removed dead code from add-then-remove churn - Prediction-time "still partiality" (unreachable: no setter), the phantom IndexingSettings::min_indexed_spot_fraction knob (getter, no setter - now the constant it always was), StillsPartialityRefine's caller-less Settings constructor and its reference to a long-gone env var, ProcessImage's unread bool return, an unused include, and a dead viewer overlay hook. Also - Viewer: the magnifier compared a QImage with itself, so its scene rect was set once ever and it could not pan into a larger dataset; the hover tail timer could fire after leaveEvent and resurrect the resolution readout outside the image. - update_version.sh regenerated the frontend lock file BEFORE bumping the version (every release shipped an off-by-one lock), and did git rm/git add on a path that has not existed since the client moved to src/client - with no set -e, both failed silently. - fpga/pcie_driver/postinstall.sh tested "[ ! occurrences > 0 ]", which is a redirect, not a test, so dkms add never ran. - Unit tests for the adaptive-threshold host functions, which had none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-27 09:07:00 +02:00
leonarski_fandClaude Opus 4.8	a8c1006c49	Choose min-pix-per-spot adaptively per image for serial-stills indexing ... For stills indexing the minimum-pixels-per-spot filter is now chosen per image instead of being fixed: the frame is indexed at min-pix 3/2/1 and the setting that maximises indexed-spot count weighted by indexed fraction (n_indexed^2 / n_total) is kept, then integrated once at that min-pix. The fraction factor keeps a smaller min-pix's extra spots only when the lattice actually explains them, so strong frames retain their real weak spots (extending resolution) while noise-flooded frames stay strict. The mode is selected by the presence of --min-pix-per-spot, now optional (SpotFindingSettings::min_pix_per_spot is std::optional<int64_t>): omit it for the adaptive per-image path, give a value to force a fixed min-pix. It applies only to the stills indexing path -- rotation indexing builds one global lattice and keeps a fixed min-pix, and the online receiver and the FPGA host path always carry a concrete value, so neither changes. IndexAndRefine::ProcessImage now returns whether the frame indexed, to drive the per-image selection. Exposed in the jfjoch_viewer spot-finding settings (adaptive-threshold and adaptive-min-pix checkboxes, each greying out the control it overrides); the broker uses neither. Validated on the full rotation regression battery (no regression) and the whole serial-stills target battery at full image count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-26 18:31:35 +02:00
leonarski_fandClaude Opus 4.8	9fdeed282a	Add fused GPU adaptive spot finder (azint + spot finding in one pass) ... AdaptiveSpotFinderGPU does the per-resolution-ring reduction once on the GPU and drives both products from it: the azimuthal-integration profile (corrected space) and the self-calibrating adaptive spot-detection threshold (raw counts). This replaces the separate GPU azint pass and the host-side adaptive spot finder that runs on the GPU path today. On a ~4.5 MP detector it does both jobs in ~1 ms/frame versus ~40 ms for the CPU adaptive finder (~42x), with an identical spot list and azimuthal profile. The per-ring threshold math (Poisson tail + read-floored Gaussian, operating point from the false-pixels-per-frame knob) is factored into AdaptiveThreshold.h so the CPU and GPU finders share one source of truth and cannot drift. Wired opt-in via a MXAnalysisWithoutFPGA constructor flag, default on for the rugnux offline path and the interactive viewer, off for the online receiver (so the broker path is unchanged). When on, Analyze() skips the separate azint pass and lifts the profile from the fused engine. The viewer gains an "Adaptive threshold" checkbox that greys out the signal/noise and photon-count sliders (the adaptive finder uses neither). Dedicated tests exercise both products (spot-finding parity vs the CPU finder, azimuthal profile vs a standalone GPU azint) plus a speed benchmark. Validated end-to-end on lysozyme serial stills: fused == CPU-adaptive index rate and merge stats. Docs: new section 3.2 in docs/CPU_DATA_ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 20:10:45 +02:00
leonarski_fandClaude Opus 4.8	014e43a4c9	Remove non-helping stills merge/scaling knobs ... Trims three opt-in stills parameters that did not improve data quality on the external-reference (PDB R-free) battery and only added code: - --partiality-uncertainty: the (1-p)/p merge-sigma term was null on all four serial-stills datasets of the battery vs their reference structures (and neutral-to-harmful at higher coefficients); removed the flag, setting and CorrectedSigma term. - --stills-modulation: the detector-plane flat-field surface was net-negative on flooded data; removed the flag, setting and MergeOnTheFly::RefineModulation (the rotation modulation in RotationScaleMerge is unaffected). - --min-indexed-fraction: every value other than the 0.20 default collapsed CC1/2; removed the override flag/setter, keeping the fixed 0.20 acceptance floor. Default behaviour is unchanged (all three were off / at their default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:40:48 +02:00
leonarski_fandClaude Opus 4.8	20bbcb1cd3	Remove --soft-weight and --local-snr spot-finder options ... Both were opt-in adaptive-spot refinements that did not help. Soft per-spot weighting was index-rate neutral across the battery (re-ranking only bites when spots exceed the max-spot cap, which weak serial data does not reach). The local-SNR gate was neutral on index rate and degraded merged CC1/2 on flooded XFEL data. Drops the flags, ApplyWeights/FilterByLocalSNR, the per-spot weight field, and the by-weight FilterSpotsByCount branch (now strongest-first only). --adaptive-spots itself is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 17:31:35 +02:00
leonarski_fandClaude Opus 4.8	f72b4484e2	Remove dead per-crystal deltaCChalf rejection ... Drops the --reject-delta-cchalf flag and MergeOnTheFly::DeltaCChalfReject. The CLI value was parsed but never consumed (the method had no call site), so the flag was already a no-op. Wiring it up and testing against an external reference structure showed it is confirmation bias: on a spurious-crystal flood it raised internal CC1/2 while CCref (correlation to the true structure) fell, and it never improved R_meas. The merge weights are already correct; per-crystal merge-side rejection has no genuine lever here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 15:05:06 +02:00
leonarski_fandClaude Opus 4.8	ecf79af018	Remove threshold-free persistence spot-detection variant ... Drops --persistence-spots and AdaptiveSpotFinderCPU::RunPersistence (the 0-D topological-persistence detector added in 5a33b0743). It was a research variant that never beat the hard-threshold adaptive detector on a CC1/2 basis and is a GPU dead-end (global candidate sort + union-find), so it is not a production path. The hard-threshold --adaptive-spots detector is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-25 12:41:44 +02:00
leonarski_fandClaude Opus 4.8	ca7cbe206a	Add opt-in local-SNR spot gate and acceptance-fraction knob (serial stills) ... Two opt-in tools for weak serial-stills tuning; both default-off, so the default pipeline is bit-identical (verified: a serial-stills reference run reproduces HEAD's 7.85% indexing rate exactly). --local-snr <sigma> (AdaptiveSpotFinderCPU::FilterByLocalSNR): after the loose per-ring adaptive threshold builds connected-component spots, drop any spot that does not stand this many sigmas above its OWN LOCAL background (robust median/MAD of a square annulus), not just the azimuthal ring mean. On structured-background (XFEL) frames the ring mean underestimates the local diffuse level in some sectors, so the ring threshold floods; a real Bragg peak still stands many local sigmas proud. Validated on XFEL stills to separate real peaks from flood at the pixel level (real median local-SNR ~70 vs flood ~2.6; SNR>=5 keeps ~99.8% of real peaks, ~14% of flood). GPU-portable (a per-spot local reduction). NOTE: on the current serial-stills battery it is index-rate/CC1/2 neutral -- the flood that survives as CC clusters overlaps weak-real spots, and only lattice-fit separates those -- but it is the correct tool for genuinely floody data (ice/jet/loosened detector) and the right substrate for the online FPGA path. --min-indexed-fraction <f>: exposes the previously hardcoded 0.20 minimum indexed-spot fraction (AnalyzeIndexing) as a per-run setting. Lowering it admits weaker/sparser crystals; on flooded XFEL data the extra lattices are spurious (pair with --min-image-cc to gate them), on clean synchrotron data there are no marginal frames so it is a no-op -- useful as a gating-experiment primitive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-24 11:12:22 +02:00
leonarski_fandClaude Opus 4.8	17eed80ff9	Seed still indexing with the strongest spots; refine with all ... On flooded or noisy still frames (weakly-diffracting detectors, XFEL background, ice) the full spot list derails the known-cell indexer: its many spurious peaks compete with the true reflections for the search, so genuinely diffracting frames fail to index. Seed the indexer with a few spot-count subsets (30 / 80 / all) and keep the lattice that explains the largest FRACTION of its own seed -- a lean, clean seed that a good lattice indexes almost fully beats a flooded seed it fits only in small part. This auto-selects a lean seed on noisy frames and the full seed where the extra spots are real signal, with no per-dataset setting. Geometry refinement and integration still use the full spot list (the orientation refiner filters spots by lattice match, so the flood is ignored while high-resolution spots are kept), so resolution is preserved. Costs at most ~3 indexer calls per frame, only on frames that do not index on the first, lean seed. Lifts the indexed-crystal yield on mildly-flooded synchrotron serial data with no regression elsewhere. Stills only; the rotation indexing path is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-24 00:07:24 +02:00
leonarski_fandClaude Opus 4.8	7c5bedfd74	Add soft per-spot quality weighting for adaptive spot detection ... Add --soft-weight (implies --adaptive-spots): give every detected spot a continuous quality weight in (0,1] and keep the highest-weight spots rather than the brightest, so a deliberately loose detector self-cleans -- bright ice / salt / jet blobs and single-pixel noise no longer evict faint clean Bragg spots from the max-spots cut. The weight is a product of dimensionless gates (AdaptiveSpotFinderCPU::ApplyWeights, computed against the per-ring background the adaptive finder already builds): a logistic ramp in the spot's SNR and a soft size band (rises from one pixel, plateaus, falls for oversized ice/salt/streak blobs). It carries on DiffractionSpot -> SpotToSave and is consumed by FilterSpotsByCount, which ranks by {non-ice, weight, intensity} when requested and by intensity otherwise, so the classic and FPGA paths are unchanged. Honest result: on the serial-stills battery this is index-rate-NEUTRAL. The weighted ranking only changes the outcome when the spot count exceeds the max-spots cap and the weight disagrees with intensity in a way that affects indexing; the adaptive detectors already produce clean spot lists and the weak sets sit under the cap, so re-ranking is a wash there (and a wash, not a regression, on the one set that floods). Its intended benefit -- robustness to ice/jet-contaminated frames and to a loosened detector -- is not exercised by this battery; kept opt-in as the substrate for that. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 19:53:55 +02:00
leonarski_fandClaude Opus 4.8	6de03bc443	Add threshold-free persistence variant of adaptive spot detection ... Add --persistence-spots, a second parameter-free detector alongside --adaptive-spots. Instead of a hard per-ring threshold it builds the noise-normalised image z = (I - ring_mean) / sqrt(ring_sigma^2 + read^2) (same per-ring background as the hard variant) and scores every intensity maximum by its 0-D topological persistence: sweeping the height from high to low, each maximum is born and, when its basin meets a taller one at a saddle, dies with persistence = birth - saddle, in sigma. A lone noise spike merges into the background almost immediately (persistence ~1 sigma); a real peak stands many sigma proud. Emitting maxima whose persistence clears the same z(E) significance bar needs no photon threshold and no min-pix, and it deblends touching peaks (each keeps its own maximum). Implemented with the same union-find idiom as the connected-component labeller. On serial stills this auto-adapts with no per-dataset tuning like --adaptive-spots, finding fewer but cleaner (deblended) spots; the hard-threshold variant remains more sensitive on the very weakest data. Both share the per-ring background and read-noise floor. comp_of is allocated lazily so the default and hard-adaptive paths pay nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 17:38:18 +02:00
leonarski_fandClaude Opus 4.8	9a8c946555	Add self-calibrating adaptive spot detection for offline stills ... The offline CPU spot finder marks a pixel strong when it clears a fixed photon count AND a local-window SNR. The fixed photon floor forces per-dataset tuning: its sweet spot tracks the background level (weak sets want a low threshold, strong or high-background sets a high one) and the usable window is narrow, so users hand-tune --spot-threshold/--spot-sigma per dataset. Add an opt-in --adaptive-spots mode (AdaptiveSpotFinderCPU) that replaces the fixed floor with a per-resolution-ring threshold derived from each image's own noise. Per ring it computes a peak-excluded background mean and sigma (one plain pass + two sigma-clip passes over the assembled photon image, binned by the azimuthal-integration ring index) and sets thr = max( PoissonTail(mean, p), mean + z * sqrt(sigma^2 + read^2) ) with p = false_pixels_per_frame / n_pixels the single portable knob (default 100) and z = Phi^-1(1 - p). The Poisson arm is the correct significance where the background is countable (it carries the sqrt(mean) shot noise, so a bright low-resolution ring gets a high threshold); the read-noise-floored Gaussian arm keeps the threshold physical where the background vanishes (empty high-resolution rings), without which those rings flood. read is a detector-level constant, not a per-dataset knob. Both arms are needed: Poisson alone floods near-zero background, Gaussian alone drops the shot-noise term and under-thresholds bright rings. One --adaptive-spots setting then adapts across a wide range of serial datasets with no per-dataset threshold, matching or beating hand-tuned thresholds and the peakfinder8/xgandalf reference on both weak large-cell and strong serial data, with equal merged R-free. The finder runs on the CPU (offline/viewer path) and reads the host image, which the GPU pipeline already keeps in sync, so it works in either build. The default (non-adaptive) path and the online/FPGA path are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 17:23:19 +02:00
leonarski_fandClaude Opus 4.8	1a2b0181a5	Add physical partiality post-refinement for stills (default on) ... Replace the frozen scalar-sigma stills partiality with a physical, refined model. Per crystal, refine an orientation tilt (dpsi_x, dpsi_y) against the running merge and recompute each reflection's partiality analytically from the refined geometry (angular Ewald-proximity model, sigma(d*) = gamma_e*d*), with the per-crystal scale G profiled out by the existing robust IRLS - no re-integration. A soft Gaussian prior on dpsi tames weak-data overfit while staying inert on strong data. The merge <-> refine loop iterates a few times. This is now the stills default via ScalingSettings::stills_partiality_refine (on). A single opt-out flag `--simple-stills` reverts to treating every reflection as a full (p=1, single pass). Retires the experimental `--still-partiality` flag. The viewer gains a "Partiality post-refinement (stills)" checkbox in Scaling settings. Validated (integrate-once / --scale): CC1/2 and R_meas both improve on three monochromatic serial-stills datasets (+2.8 / -10, +5.6 / -3.4, +2.1 / -4); neutral on a pink-beam DMM set (already-full reflections); R-free/R-work down vs a fixed model; competitive with CrystFEL partialator on matched frames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 14:45:43 +02:00
leonarski_fandClaude Opus 4.8	cbc6a85157	Debias stills merge with expected-variance weighting ... The serial-stills merge (MergeOnTheFly::CorrectedSigma) weighted each observation by 1/sigma^2 using the observation's OWN sigma. Below ~1 photon the Poisson signal part of that sigma correlates with the observation's up/down fluctuation, so the inverse-variance mean is biased low: an up-fluctuated observation acquires a larger sigma and is over-downweighted. The rotation combine (RotationScaleMerge:: process_rawrun) already avoids this by rebuilding the signal variance at the pooled estimate; the stills path did not. Decompose each observation's variance into a background/read part (kept per-observation) and a Poisson signal part, and rebuild the signal part at the reflection's expected <I>. Bit-identical when an observation sits at its reflection mean; only weak-shell weights move. Now default on, so the stills path matches the rotation path; --no-expected-variance-merge restores the old observed-sigma weighting. Validated by paired refinement (phenix, 5 free-set seeds, byte-identical free flags across arms): R-free-neutral on strong lysozyme and lower R-free on weak serial-stills data checked against an independent deposited model (6/6 seeds). The CC1/2 dip on strong data reflects precision, not accuracy. Applies to both offline rugnux and the online broker stills merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 22:50:25 +02:00