3 Commits

Include renames

Author	SHA1	Message	Date
leonarski_fandClaude Opus 5	61a7c91b90	Ice: detect it on two channels, and only handle it when it is there ... The per-image ice score was read off the PLAIN azimuthal profile. That profile is a per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it exactly as ice would. Measured over 37 rotation crystals, that did not merely add noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all (4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its 99th percentile and 2.70 at its maximum, so that metric cannot support any absolute threshold whatsoever. The adaptive spot finder already computes the right input for its own threshold: a sigma-clipped per-resolution-ring background, in the same bins. A powder ring is azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at 2.08-2.37, against a decoy null that never exceeds 1.29. That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE spots and leaves the radial profile flat. So a second channel counts found spots on the rings against the same q width of ice-free flanks beside them. The two barely overlap - the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6, while a clean crystal reads 1.04 on both. Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 % of the unique reflections at typical resolutions whether or not the crystal has ice, so flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now all skipped when neither channel sees any. The gate is applied in the full pipeline and in --scale, which reads the stored per-image values back out of the _process.h5. Also fixes the merge-time mask's control: the shoulder now excludes reflections that are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on the neighbours and the test compared ice against ice. Measured, that is the only thing this changes: it removes firings on those three rings and leaves every other firing's CC pair identical to three decimals. And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the same data got a narrower band online than the measured ~0.06 ring FWHM justifies. Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses. Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3 and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa 3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The one crystal that loses reflections improves on both R_meas and CC1/2. Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5 write/read and the receiver plots are. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-06 16:17:23 +02:00
leonarski_fandClaude Opus 5	b22e1b6822	rugnux: raise the ice-ring mask margin to the measured null ... The mask drops a hexagonal-ice ring when its merged half-set CC1/2 falls a fixed 0.05 below its resolution shoulders. That margin is not a significance level: at the populations these rings actually have, 0.05 spans 1.1 to 7.3 sigma across firings, and a nominal Fisher-z error understates the real scatter of these heavy-tailed intensities by ~2.7x, so the null has to be measured rather than derived. Measured it with decoy bands - the identical ring/shoulder statistic evaluated at q positions carrying no ice ring - over the 37-crystal rotation battery: the gap's empirical null is p95 +0.032, p99 +0.095. So 0.05 sits near the 96th percentile, about 4% of ice-free bands clear it, and roughly half the 22 observed firings are indistinguishable from bands with no ice in them. The firing gaps are continuous, not bimodal, with 12 of 22 in [0.05, 0.10). Raise it to 0.10, the 99th percentile of that null. Firings 22 -> 10, crystals 12 -> 5, decoy false-positive rate 3.4% -> 0.8%. An independent check against XDS - which integrates through ice rings and so measures exactly what we delete - agrees: of the firings with a usable comparison, 9 true / 9 false becomes 7 true / 2 false. Battery: space groups 34 OK / 3 DIFF, the same three crystals as baseline, and no other discrete decision changes on 37/37. The heavily iced crystal keeps all five of its rings and its CC1/2 of 96.6; eight others recover 3.9-11.9% more unique reflections and up to 10.4 completeness points. Cost is CC1/2 -0.84 on one crystal, -0.35 on another, and agreement with XDS on the common reflections worse by a median 0.0004. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-05 19:00:10 +02:00
leonarski_fandClaude Opus 5	ea667cb306	rugnux: handle ice rings in --scale as the full pipeline does ... --scale did none of the ice handling the run that wrote the _process.h5 had done, so re-scaling a stored dataset silently produced a different - and flatteringly more complete - answer than the pipeline it was meant to reproduce. Three separate gaps: * --detect-ice-rings was accepted and ignored. The --scale block returns before the line that applies it. * Reflections were never flagged as sitting on an ice ring, so the per-image scale fit included them. The flag is not stored per reflection, so it has to be recomputed from the resolution. * RotationScaleMerge was constructed with the ice half-width hardcoded to zero. That is what turns a resolution into a ring index, so every ice test inside the merge was a no-op whatever was passed to it. The CC1/2 ring test that decides which rings to drop moves into FindDecorrelatedIceRings, shared with the full pipeline so both reach the same verdict on the same data, and --scale now re-merges with the mask the way the pipeline does. The stills branch re-runs only the merge: the scaling has already been applied to the reflections and repeating it would compound it. Measured on a rotation dataset with three decorrelated rings, --scale went from 8765 unique / 36.3% completeness / R-meas 18.5% / <I/sig> 1.1 to 7638 / 31.6% / 18.0% / 1.3, against the full pipeline's 7692 / 31.8% / 17.9% / 1.3 - the reported completeness had been inflated by reflections the pipeline drops. The full pipeline is bit-identical across the refactor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 19:49:55 +02:00