v1.0.0.rc-161 #71
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: mx/Jungfraujoch#71
Reference in New Issue
Block a user
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
mxandscalerun writes a<prefix>_report.txtresults report modelled on XDS'sCORRECT.LP. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice.jfjoch, andimage_analysiscompiling under MSVC again.Breaking change to the rugnux command line:
--azint-onlyand--scaleare removed, replaced by--mode azintand--mode scale; the full pipeline is--mode mxand remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one.-t/--strideis refused on rotation data: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with-s/-einstead.--mode azintand--force-stillstill take a stride.Breaking changes to OpenAPI - regenerate the client (
jfjoch-client1.0.0-rc.161,frontend/src/client) or read the affected fields as optional:image_scale_bis removed from theplot_typeenum, so a client requesting that plot now gets an error rather than a curve.azim_int_settings.high_q_recipA,spot_finding_settings.high_resolution_limitandspot_finding_settings.low_resolution_limitare no longerrequired. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing.Breaking changes to the stored formats - a consumer reading these fields must treat them as optional:
/entry/MX/imageScaleBFactoris absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more._reflns.jfjoch_diffrn_ISanow carries the whole-range1/sqrt(a*b)that XDS's ISa denotes, and the error-modelaandbare reported in XDS's convention; the strong-reflection asymptote moves to_reflns.jfjoch_diffrn_ISa_asymptotic. A file written by an earlier version carries the asymptote under the plainISaname.The offline CPU spot finder marks a pixel strong when it clears a fixed photon count AND a local-window SNR. The fixed photon floor forces per-dataset tuning: its sweet spot tracks the background level (weak sets want a low threshold, strong or high-background sets a high one) and the usable window is narrow, so users hand-tune --spot-threshold/--spot-sigma per dataset. Add an opt-in --adaptive-spots mode (AdaptiveSpotFinderCPU) that replaces the fixed floor with a per-resolution-ring threshold derived from each image's own noise. Per ring it computes a peak-excluded background mean and sigma (one plain pass + two sigma-clip passes over the assembled photon image, binned by the azimuthal-integration ring index) and sets thr = max( PoissonTail(mean, p), mean + z * sqrt(sigma^2 + read^2) ) with p = false_pixels_per_frame / n_pixels the single portable knob (default 100) and z = Phi^-1(1 - p). The Poisson arm is the correct significance where the background is countable (it carries the sqrt(mean) shot noise, so a bright low-resolution ring gets a high threshold); the read-noise-floored Gaussian arm keeps the threshold physical where the background vanishes (empty high-resolution rings), without which those rings flood. read is a detector-level constant, not a per-dataset knob. Both arms are needed: Poisson alone floods near-zero background, Gaussian alone drops the shot-noise term and under-thresholds bright rings. One --adaptive-spots setting then adapts across a wide range of serial datasets with no per-dataset threshold, matching or beating hand-tuned thresholds and the peakfinder8/xgandalf reference on both weak large-cell and strong serial data, with equal merged R-free. The finder runs on the CPU (offline/viewer path) and reads the host image, which the GPU pipeline already keeps in sync, so it works in either build. The default (non-adaptive) path and the online/FPGA path are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Add --soft-weight (implies --adaptive-spots): give every detected spot a continuous quality weight in (0,1] and keep the highest-weight spots rather than the brightest, so a deliberately loose detector self-cleans -- bright ice / salt / jet blobs and single-pixel noise no longer evict faint clean Bragg spots from the max-spots cut. The weight is a product of dimensionless gates (AdaptiveSpotFinderCPU::ApplyWeights, computed against the per-ring background the adaptive finder already builds): a logistic ramp in the spot's SNR and a soft size band (rises from one pixel, plateaus, falls for oversized ice/salt/streak blobs). It carries on DiffractionSpot -> SpotToSave and is consumed by FilterSpotsByCount, which ranks by {non-ice, weight, intensity} when requested and by intensity otherwise, so the classic and FPGA paths are unchanged. Honest result: on the serial-stills battery this is index-rate-NEUTRAL. The weighted ranking only changes the outcome when the spot count exceeds the max-spots cap and the weight disagrees with intensity in a way that affects indexing; the adaptive detectors already produce clean spot lists and the weak sets sit under the cap, so re-ranking is a wash there (and a wash, not a regression, on the one set that floods). Its intended benefit -- robustness to ice/jet-contaminated frames and to a loosened detector -- is not exercised by this battery; kept opt-in as the substrate for that. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>image_fp is the base class's one pixel representation, and it earns that for three of the four image widgets: the azimuthal image is already float, the grid scan holds computed 1/sigma^2 floats, and the calibration viewer accepts eight source types from uint8 to float64. The diffraction image is the odd one out -- its source is a large int32 buffer -- and it is the one paying: a full int32 -> float pass plus a second resident copy of the image, on every frame. Split the mapping from the source. PixelColorMap holds the precomputed LUT constants and does value -> colour; a virtual ColorRow() picks the pixels out of whatever buffer the subclass has. Both paths now go through the same Apply(), so only the gap/bad/saturated dispatch differs, and it lines up exactly with the encoding LoadImageInternal used: GAP_PXL_VALUE -> NAN -> gap ERROR_PXL_VALUE -> -INF -> bad SATURATED_PXL_VALUE -> +INF -> saturated The base class still needs real pixel values for ROI statistics and per-pixel labels, so image_fp is filled on demand instead of per frame -- and only when something reads it: a non-empty scratch ROI, or labels above 30x zoom. Neither happens while simply looking at frames, and nothing else routinely sets roiBox (the named ROIs are computed in the reading worker, not here). 18.1 Mpx: 10.5 -> 5.9 ms per frame and 72 MB less resident. 4.5 Mpx: 1.8 -> 1.0 ms and 18 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>FullViewportUpdate redraws the whole viewport on any change. The attached comment ("keep overlays in pixel units independent of zoom") does not describe what the setting does, and nothing here needs it: SmartViewportUpdate repaints the changed rectangles and falls back to a full repaint by itself once there are too many to be worth tracking. This is the view used by the calibration window and the magnifier, and the magnifier is driven from every hover, so on a remote session it repainted its whole viewport per pointer motion. Note: not exercised visually -- both windows are opened from menus, which the headless harness does not drive. The change is a repaint-mode switch with no effect on what is drawn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>`RefineGeometryIfNeeded` hands XtalOptimizer the WHOLE spot list, not the indexed subset, and the first pass admits anything within 0.3 fractional-Miller units of an integer - which is 11.3% of RANDOMLY placed spots, since the admitted volume is (4/3)*pi*t^3. Every one of them then enters an unweighted L2 fit with an arbitrary rounded index. On images with many detections the refined orientation ends up 2.3-2.8 degrees from the goniometer-consistent one and explains 14 of its own 250 spots where the undragged orientation explains 68; mosaicity and profile radius inherit the error and integration follows. Weight every spot by its intensity divided by the median intensity of its own equal-count resolution shell, applied as w^2 on the squared residual with w^2 = r/(1+r). The shell normalisation is the point: refinement needs the high-resolution spots because they carry the cell and distance, and those are LEGITIMATELY weaker, so a raw intensity weight would suppress exactly the spots the fit depends on. Measured, the weight is resolution-neutral - median exactly 0.707 in every shell, and corr(w, 1/d^2) = -0.20 / -0.11 against -0.32 / -0.34 for the same function of un-normalised intensity. This is a PRIOR: it is computed from the spot alone and never looks at the current residual, so unlike a robust loss it cannot mistake a genuine spot for an outlier while the starting geometry is still far off and leave the fit unable to move. That failure is not hypothetical - a CauchyLoss on this same residual, at the scale the multi-frame GeometryRefiner uses, collapsed one crystal's indexing rate from 99.89% to 19.83% and was rejected. It does not work by telling good spots from bad, and it does not need to. No per-spot property separates spots that index from spots that do not: measured AUC is 0.53 for peak pixel, 0.53 for total intensity, 0.51 for pixel count, 0.45 for peakedness, and a logistic regression on all twelve available features with pairwise interactions reaches only 0.64. What the weight does is halve the EFFECTIVE COUNT of every spot (mean w^2 = 0.517), and the damage scales with the absolute count of unexplained spots in the objective - 80.6 per frame here against 36.8 for the finder that was never damaged. That is also why an empirical `--max-spots 66` cap works while leaving the list no purer than before: it reaches the same operating point by discarding spots. This reaches it without discarding any, and without a tuned constant. Rotation battery, 33 crystals, both spot finders: finder A 29/33 -> 30/33 point groups (one crystal P222 -> P4212 = XDS, its high-shell CC1/2 86.0 -> 98.4) finder B 28/33 -> 29/33 point groups (one crystal I222 -> I23, its high-shell CC1/2 14.8 -> 38.0) No crystal lost its point group in either mode and no run failed. On the meta-stable multi-lattice dataset the CC1/2 spread over four frame ranges falls 19.7 -> 13.1 for finder B, and the indexing rate rises in 8 of 8 configurations. The crystal that the rejected robust loss destroyed keeps its 99.89% indexing rate exactly. The cost, stated plainly: ISa falls by 0.2-1.7 on about five crystals (and rises on two). Point-group correctness is worth more than that - merging in the wrong symmetry cannot be undone from the output, whereas ISa is a quality metric of data that remain correct - but it is a real trade and not a free win. Off by default. The indexers pass a spot list they have already selected, so their calls are unchanged; only the per-image refinement, which gets the raw list, turns it on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>It was held back because a 33-crystal rotation battery showed it breaking three crystals deterministically - a lost space group, a halved indexing rate and a collapsed merge. None of those causes turned out to be in detection. The extra spots adaptive finds are real. Measured per spot against a finder-neutral local background: 64% recur at the same position on the adjacent frame (chance rate 0.5%) with 2-frame rocking curves, and 0.00% would fail a conventional local SNR >= 4 test, median local SNR 34. What they include is genuine peaks belonging to no lattice the indexer found, and the damage they did scaled with their absolute COUNT (80.6 per frame against 36.8 for the fixed finder), not with their quality - which is why nothing aimed at judging individual spots ever worked. The three failures fell to fixes elsewhere: merge collapsed - a per-frame scale free to collapse toward zero amplified two junk frames by 546x (704098712,ec7a82613). Not a detection problem at all: the fixed-threshold finder trips the same bug on a different frame range. space group lost - the per-image geometry refinement was dragged 2.3-2.8 deg off by the weak-spot tail in an unweighted fit; weighting each spot by how strong it is FOR ITS RESOLUTION fixed it (ae126c3d5), and gained a point group for the fixed finder too. indexing halved - gone with the same two; that crystal is now better under adaptive (CC1/2 92.6 -> 95.9, high-shell 44.9 -> 56.4). Battery, 33 crystals, adaptive vs the fixed finder: exact space group matching XDS 26/33 vs 25/33 point group matching XDS 29/33 vs 30/33 ISa better on 6 crystals recovers a screw axis the other misses (P321 -> P3121) The one point group it loses is a tetragonal crystal where adaptive collects 2.4x the observations at better R_meas (21.4% vs 28.3%) and better ISa (4.58 vs 2.91), and forced to the right group gives CC1/2 99.2% at multiplicity 10.7 - matching XDS. Only the automatic symmetry call fails there, and six candidate causes have been measured and refuted (mixed indexing hands, off-crystal frames, a badly integrated minority, radiation damage, pseudo-tetragonality, uncorrected anisotropy). It is left as the subgroup, which is the recoverable direction: -S gives XDS-quality data from the same run, whereas the failures this unblocks were not recoverable. --no-adaptive-spots reverts to the fixed-threshold finder. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>A reflection the group predicts absent counted as a violation when I/sigma > 3 AND E^2 = I/<I>(shell) > 0.3. Neither half survives contact with real data: * merged sigma is floored at b|I|, so merged I/sigma saturates at ISa for nearly every reflection - the I/sigma half is an on/off switch keyed on ISa vs 3, not a per-reflection test. On one crystal the absent class read <I/s> 4.10 against 3.73 for the present class while being genuinely extinct; * <I>(shell) decays with resolution while a systematically-absent reflection keeps a small NON-decaying residual (background / profile leakage), so absent reflections drift over an absolute E^2 cut at high resolution. That cost a tetragonal 42_12 crystal its 4_1: 18 of its 47 absent 00l crossed the cut, all beyond 3.7 A, at absolute intensities identical to the low-resolution ones correctly judged absent, while their l=4n row-mates sat 20-60x higher at the same resolution. A screw extinguishes only the reflections that lie ON its axis, so the fair yardstick is the rest of that same row. The threshold is now 0.3 * max(1, median E^2 of the reflection's own row), the row being the gcd-reduced reciprocal-space direction and the control class the same-row reflections the group predicts present. Floored at 1, so it only ever relaxes: a screw can be recovered by it, never lost. Per row, not pooled. A 4_1 along c and a 2_1 along a are separate conditions with separate controls; pooling let the weak a/b rows (median E^2 ~0.5) set the threshold for a strong c row (8.4) and the rescue never fired. The candidate table now reports the screw evidence (median E^2 of the absent class and of its rows) - the <I/s> columns are the centering evidence and say nothing about screws, for the sigma-floor reason above. Rotation battery, 33 crystals: 31 decisions bit-identical, the 42_12 crystal recovers its 4_1 (0 violations, row E^2 8.4 vs absent 0.12), and one crystal with a long axis and heavy 00l overlap moves to a 4_1 group at exactly 10.0% violations - marginal, and its sister crystal of the same form sits at 13.3% and does not move. Real screws now span 0-9.3% violations, so max_absent_violation_fraction cannot be tightened below 0.10 without risking a genuine one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The search runs in pass 1 of the rotation two-pass; pass 2 only reuses the group it decided, so the result the CLI renders carried no search at all and the whole report - operator correlations, the space-group candidate table, the refused higher symmetry - was silently dropped on every rotation dataset. Most costly of all, the "or <group> (indistinguishable from these data)" line never appeared, so an enantiomorphic pair the intensities genuinely cannot separate was reported as a single answer. Carry pass 1's search into the returned result. Also de-duplicate the alternatives when the centred-lattice test swaps the metric-matching candidate into the answer: the group it displaced was left out and the chosen one listed twice ("C2 or P21 or C2"). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Ten places where the document and the implementation had drifted apart. Each was checked against the source before rewriting: * 7.5 rotation post-refinement: it is TWO separate cross-validated steps (cell+axis from the angles, then distance+beam from the positions with the cell fixed), not one joint fit against the merged fulls; the held-out split is an hkl hash, not a frame split; the bounds are +-5% on distance and +-15 px on the beam, not "under ~1%"; and only the distance and beam centre reach pass 2, which re-indexes de novo. * 9.2 the trimmed-mean background: it is computed in the shared background pass, so it DOES apply to --integrator boxsum. Only the broadband sigma-clip is excluded. The section previously said both, contradicting itself two paragraphs apart. * 9.3 per-reflection profile rebuild, sub-pixel centring and radial elongation are gaussian-only; the empirical profile keeps the fixed per-shell grid and is accumulated on rounded predicted positions, not centroids. * 10.5 the asymptotic ISa and the b_ISa sigma floor are rotation-only; stills report 1/b and floor with the whole-range b. * 8.4 centering absences are applied only when the user fixes the space group - de novo, prediction runs in P so the search can confirm the centering from the intensities. * 10.6 per-batch relative-B cross-validates on ASU-group parity, not the frame parity the other surfaces use. * 13 the resolution cutoff sits one reported-shell width PAST the CC1/2 = 0.30 crossing, so data below 0.30 are kept. * 7.1 the orientation-only prior penalises all three components of the angle-axis vector. * 10.1 rlp is the RECIPROCAL Lorentz factor, L = 1/rlp. * 14.3 the model scaling is fitted over work and free reflections alike, so R-free is free of refinement, not of the scaling fit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Rotation kept a 1.5 A high-resolution limit for spot finding on the strength of one indexing-rate measurement (100.0 -> 96.8% on a strong set). Measured properly, over the whole 33-crystal rotation battery, it does not earn its place: * no space-group decision changes - the same 30/33 agree with XDS, and the three that differ are the known pre-existing cases, unchanged; * 29 of 33 crystals are identical to the digit - same indexing rate, R_meas, CC1/2, ISa. The limit was doing nothing on the large majority; * where it does bite, the limit is the worse setting. The one crystal that loses appreciable indexing rate without it (99.50 -> 94.22%) comes back with lower R_meas (29.4 -> 28.1), higher high-resolution CC1/2 (27.9 -> 29.1) and higher ISa (5.77 -> 6.17). Another loses 0.4% of frames and gains 2.8 points of CC1/2_hi. Fewer frames indexed, better data from them; * runtime is unchanged (16m46s vs 17m32s over the battery). So the indexing-rate cost is real but does not carry through to the merged data, which is what the limit was protecting. Unset now means "as far as the detector reaches" for rotation as well as stills; --spot-high-resolution still sets a limit for weak, high-background data where the extra high-resolution spots are genuinely noise. This also removes the flag that distinguished "the user asked for no limit" from "the user said nothing" - with no rotation default left, both mean the same thing. While rewriting the comment block, corrects its neighbouring claim that rotation keeps the fixed-threshold finder; adaptive detection has been the default for both workflows since6f4917dce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The document had accumulated development narrative: what was tried and rejected, which datasets a change rescued or cost, measured percentages from test batteries. That belongs in commit messages and reports, not in a reference description of the pipeline - it dates quickly, and a reader looking up what an algorithm does has to sort it out from how it came to be. Removed throughout, keeping the algorithmic content and the design reasoning that explains a choice on principle: * 3.2 the whole paragraph justifying the rotation spot-finding limit from battery measurements, and the CPU-vs-GPU per-frame timings; * 3.3 "a significance/z-score was considered but is uninformative"; * 7.4 / 7.5 the comparisons to a robust loss and to joint refinement as approaches that had failed; * 9.2 the R_meas / CC1/2 outcomes attributed to the trimmed-mean background; * 9.3 "per-detector-region and crystal-anisotropy profiles were evaluated and add nothing"; * 10.2 the stills tilt "succeeds where a freely-fitted width collapses"; * 10.5 the CC_anom argument, trimmed to why the statistic behaves as it does; * 10.6 the survey of per-frame correlation medians across datasets; * 13 the space-group bullet, restructured into the three gates it actually applies, dropping the dataset anecdotes; * 14.2 the free-form per-shell rescale, stated as a design choice rather than an experiment. Section 13's space-group text was one 20-line paragraph; it is now a numbered list of the three tests, which is what the code does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>None of this has a reader: - ScalingSettings::scaling_regularize and its setter/getter - ScaleOnTheFlyResult::succesful (never set) and ::time_s (set, never read), with the timing that only fed the latter - JFJochImage::last_fit_viewport_ (written twice, read nowhere) and the comment claiming the retry uses it - the retry keys off initial_fit_done_ - JFJochDiffractionImage::ice_ring_width_Q_recipA, and a QtConcurrent include in a file that uses none - an unused gemmi::Op accumulator in the spindle-angle helper - <random> in Merge.{h,cpp}, from before the half-set split became a hash - an orphaned comment describing the Ceres B-factor residual deleted in014e43a4c, and two trailing comments that had collided on one line Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>--scale did none of the ice handling the run that wrote the _process.h5 had done, so re-scaling a stored dataset silently produced a different - and flatteringly more complete - answer than the pipeline it was meant to reproduce. Three separate gaps: * --detect-ice-rings was accepted and ignored. The --scale block returns before the line that applies it. * Reflections were never flagged as sitting on an ice ring, so the per-image scale fit included them. The flag is not stored per reflection, so it has to be recomputed from the resolution. * RotationScaleMerge was constructed with the ice half-width hardcoded to zero. That is what turns a resolution into a ring index, so every ice test inside the merge was a no-op whatever was passed to it. The CC1/2 ring test that decides which rings to drop moves into FindDecorrelatedIceRings, shared with the full pipeline so both reach the same verdict on the same data, and --scale now re-merges with the mask the way the pipeline does. The stills branch re-runs only the merge: the scaling has already been applied to the reflections and repeating it would compound it. Measured on a rotation dataset with three decorrelated rings, --scale went from 8765 unique / 36.3% completeness / R-meas 18.5% / <I/sig> 1.1 to 7638 / 31.6% / 18.0% / 1.3, against the full pipeline's 7692 / 31.8% / 17.9% / 1.3 - the reported completeness had been inflated by reflections the pipeline drops. The full pipeline is bit-identical across the refactor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>CPACK_DEBIAN_MAIN_COMPONENT does not exist. Only CPackRPM.cmake has a MAIN_COMPONENT variable, which is why the RPM came out as "jfjoch" while the .deb of the same component came out as "jfjoch-jfjoch" - the DEB generator names every component package "<CPACK_PACKAGE_NAME>-<component>" unless CPACK_DEBIAN_<COMPONENT>_PACKAGE_NAME overrides it, and the line we set was read by nobody. It also named a component ("broker") that does not exist, so it could not have matched anything either way. Set the name the generator actually reads, and declare Replaces/Conflicts on the old one: the new package owns the same files, so dpkg would otherwise refuse to unpack it over an installation that already has jfjoch-jfjoch. The DKMS component's postinst and prerm are the driver's postinstall.sh and preuninstall.sh, configure_file'd into place. They were mode 644, and CPACK_DEBIAN_PACKAGE_CONTROL_STRICT_PERMISSION is off, so they were packaged as they are - and dpkg cannot execute a maintainer script it cannot execute. The RPM path is unaffected: it reads the same files as text into the spec. The viewer's Freedesktop menu entry and its icon are installed on Linux only now. On Windows the Start Menu shortcut comes from CPACK_PACKAGE_EXECUTABLES and on macOS from the .app bundle, so on those two the installer was carrying a share/applications entry and a share/pixmaps icon that nothing reads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>A detector tilt does NOT appear as a cos(2 phi) modulation of the ring radius, as the previous comment claimed. To first order a misalignment beta gives r(phi) = R + (R^2 / F) (beta_x cos phi + beta_y sin phi) which is a cos(phi) term - the same harmonic a wrong beam centre produces. What separates them is the radius dependence: the centre's amplitude is the same on every ring, the tilt's grows as R^2. So they are told apart across rings, not within one, and on a single ring they are exactly degenerate. Measured on a powder standard the true cos(2 phi) term is of order R^3 beta^2 / F^2 - hundredths of a pixel, at the noise floor - so it carries nothing usable. Also add the tilted round trip, which was missing. It doubles as a check that RingOptimizer's open-coded rotation agrees with DiffractionGeometry's: the fitter applies Rx(-rot2) Ry(+rot1) by hand rather than going through the geometry's Rz(-rot3) Rx(-rot2) Ry(+rot1), and those had never been held against each other. They agree - 0.020 / -0.015 rad recovered as 0.0197 / -0.0148. Dropping rot3 is right rather than an omission, since rings cannot constrain in-plane roll. The tilted case yields fewer ring points than the centred one, which is expected and worth knowing: the extractor searches a window centred on where each ring is EXPECTED, so a large enough geometry error carries part of a ring out of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>A weighted mean is only unbiased while the weights are independent of the values being averaged. The IUCr's own nomenclature report (Schwarzenbach et al., Acta Cryst A45 (1989) 63-75) puts it directly: weights in averaging "should not be based on the counting statistics of the individual observations whose estimated variances are biased and result in larger weights for accidentally low intensities". Two places in the rotation pipeline were doing exactly that, and between them they drove whole resolution shells of merged intensity negative. 1. The profile fit computed its non-signal variance as var_bkg = max(0, 1/den - max(0, I) + bkg-estimate term) The point of a separate var_bkg is that it does NOT move with the reflection's own fluctuation, and 1/den - I is the quantity that does not: 1/den is the fit variance taken at the fitted intensity and grows with it roughly one for one. Clamping the subtrahend at zero left a down-fluctuated reflection's own deflated variance standing as its background variance. Measured over 6.9 M partials of one weak rotation dataset, var_bkg/bkg came out at 3.7-5.4 for observations with I < 0 against 11.4-13.7 for I > 0 - the down-fluctuated half of every reflection carried a variance ~2.7x too small and was weighted up by the same factor, first in the 3D combine and then again in the merge. Removing the clamp makes var_bkg flat in I (~13 x bkg across the whole range). 2. The merge then weighted each combined full by 1/sigma_full^2, and sigma_full is by construction a function of the full's own answer: the combine's variance carries a corr*max(0, F) signal term, so every full with F <= 0 got the smallest variance the model allows while the strongest quartile got 2.26x more. The merge now rebuilds that variance at the reflection's mean instead, from a linear model var(I) = var_bkg + var_per_I * I that the combine measures and stores on the full. This mirrors MergeOnTheFly::CorrectedSigma, whose comment already claimed to mirror the rotation combine. Verified against an estimator that cannot see the fluctuation - summing the partials and dividing by the summed partiality, the classical construction every other program uses (Greenhough & Suddath, J. Appl. Cryst. 19 (1986) 400-409, via Leslie, Acta Cryst D55 (1999) 1696-1702: profile fitting biases the individual partials but not their sum). Reproducing the merge on dumped observations, the shipped weighting sat ~1.9 sigma below that reference in the noise shells; the two changes recover most of it, and every intensity-independent weighting scheme agrees with the reference once (1) is in. Four-crystal probe, XDS resolution limits, branch fingerprint identical on all four (so none of these is a two-pass branch flip): weak cubic case last shell <I/sig> -1.6 -> +0.2 (XDS +0.10), last shell R_meas 478% -> 250% (XDS 246%), overall <I/sig> 6.1 -> 7.5 (XDS 7.18), R_meas 18.3% -> 18.1%, CC1/2_hi 38.2% -> 43.7% tetragonal case outer shells <I/sig> -0.4/-0.8/-0.9/-1.0 -> +1.8/+1.2/ +0.9/+0.4, R_meas 184%/595%/7614%/nan -> 95%/119%/135%/232% (the nan was the shell mean crossing zero), R_meas 33.3% -> 32.9%, CC1/2_hi 38.3% -> 56.5% trigonal case R_meas 13.0% -> 12.5%, CC1/2_hi 14.4% -> 16.5% strong control unchanged to every printed digit but ISa Cost: ISa falls (17.2 -> 14.0 and 16.7 -> 14.9 on the two mid-strength cases, 28.3 -> 27.8 on the control). Strong reflections are untouched by (1) - their partials are all positive, so var_bkg is bit-identical - but the joint a/b fit redistributes: honest weak sigmas lower a, and b rises to keep the strong bins fitted. The median reduced chi^2 improves (1.25 -> 1.14, 1.35 -> 1.28) so the new split describes the scatter better, but ISa is the one headline metric that moves the wrong way and it should be watched over the full battery. The integrator change is shared, so the stills merge sees it too; there it feeds GetExpectedVarianceMerge, which had been handed the same contaminated var_bkg. That path is untested here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Rotation processing no longer refuses to integrate a frame that fails to index on its own, which is right - no other program does that - but it means a genuinely bad stretch of a sweep is now integrated instead of quietly dropped. Some sweeps have a real problem behind that stretch: the crystal partly or wholly out of the illuminated volume, off the rotation axis, or dying of dose. That is actionable at the beamline ("recollect", "re-centre"), and until now nothing said it. MeasureSweepQuality reports it as contiguous RANGES, never per-frame flags, and reports only - no observation is excluded on the strength of it. A single weak frame is noise; forty consecutive ones are a fact about the experiment, and the frames still carry signal worth merging. The discriminator is that the incident flux is already out of the per-image scale before that scale is fitted (DivideOutIncidentFlux runs from Ingest), so a drop in G that the beam does not explain is on the sample side by elimination. Measured on one crystal with a dead arc: the flux proxy spans 1.4x across the run where the fitted scale spans 246x. A range needs BOTH per-frame channels down: the scale, and the CC to merge. The CC channel is what keeps a merely attenuated stretch out - absorption and flux scale a frame's intensities without changing how well they correlate with the merged reference. Without it the clean high-multiplicity control, whose per-image scale swings 4x on a 180 degree period, would be reported as a bad crystal. It is not: it produces no ranges at all, and neither does the other control. Five codes, each the field's own words and each a phrase a report can print: no diffraction - essentially nothing was recorded from the indexed lattice over the range out of beam - frames were lost: the range gets a scale far less often than the run does weak diffraction - the frames all still index, with much less intensity; cause not determined loss of centring - one cycle of modulation per revolution (autoPROC's words for the phenomenon) radiation damage - the range runs to the end of a sweep whose quality was already decaying Only the last two claim a cause, and each rests on its own evidence. Damage is progressive, so it must have been setting in before the range and must not recover. Loss of centring rests on the one signature that breaks a documented degeneracy: Evans (Acta Cryst. D62, 72-82) notes that illuminated volume and absorption are indistinguishable, but a crystal's own shape absorbs on a 180 degree period, so a dominant 360 degree fundamental over a full turn cannot be the crystal's shape. That test runs on the total scale, flux included, unlike everything else here - the flux proxy is a background, a crystal leaving the beam takes its own scattering with it, and the beam cannot be periodic in an angle it does not know. Where the evidence does not reach, weak diffraction says so rather than guessing. Frame numbers are processed-image ordinals, inclusive at both ends, the numbering of _image.dat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>The monitor fitted each batch's relative-B on SINGLE observations - ln(I_ref/I_obs) regressed on s^2, weighted by (I_obs/sigma)^2, with the logarithm requiring I_obs > 0. The observation therefore sits in the response and in its own weight, and the positivity requirement keeps only the upward half of the noise, so the estimate is biased downwards wherever I/sigma approaches 1 and is unbounded in the limit. Simulated: on a batch with no relative-B at all and <I/sigma> = 0.3 it reads -28 A^2; on a batch whose true relative-B is +30 A^2 it reads -29. The bias grows with dose, so it inverts the answer on exactly the data the number exists for. That is not a corner case. Re-measured on stored integrated intensities, a 360 deg sweep obstructed over a 60 deg wedge - whose honest curve is flat for 130 deg, dips over the wedge and comes back - printed a per-batch curve saturated at -31.4 A^2 for twelve consecutive batches (the +-50 A^2 clamp less the low-dose anchor, "no data here" reported as a measurement) under a headline of -19 A^2 of radiation damage. A deliberately dosed dataset printed -39 A^2 where the honest measurement is about +34: the one crystal with real damage got the sign wrong. Eight of thirty-eight datasets reported |dB| > 5 A^2 and their curves oscillate by tens of A^2. So pool the observations into ten equal-occupancy resolution shells per batch before taking the logarithm, and fit slope AND intercept over the shell means, weighting each shell by its own pooled (I/sigma)^2. A shell mean is well determined where a single observation is not, it admits negative intensities, and it carries the I/sigma that says whether the batch can be measured at all. The same simulations then reproduce the truth to under 1 A^2 at every signal level. The intercept keeps a batch that is merely dimmer than the run - an attenuated beam, a mis-fitted frame scale - out of the damage number: a batch mis-scaled by 2x read +12 A^2 of "damage" without it and +0.05 with it. A batch whose shells are too weak to fit is now absent from the curve, printed as "-", instead of pinned to the clamp. And the clamp itself is now an argument of the solve rather than one shared constant: it guards against divergence, and the correction keeps the bound it was tuned with, but with the estimator fixed a heavily dosed crystal's honest relative-B runs past it - the monitor pinned thirteen consecutive batches at +49 A^2, which is the same defect in the other direction. The monitor is given room a real relative-B cannot reach and drops any batch that lands on it anyway. The shells are laid inside the range the run actually diffracted to, not across the whole merged range: a resolution limit taken from another program or left generous spends most of an equal-occupancy grid on noise and leaves a batch with too few shells to fit at all - on the battery that silenced three crystals outright and cost two of them nineteen batches of thirty-six. Where the merged range already sits inside the signal the grid is unchanged and so is every number. The first->last headline is reported only where a straight line explains at least half of the curve's variance, or where the curve is flat to within a couple of A^2 and the answer is simply "no damage"; otherwise there is no headline and the report says the loss was not dose and points at the sweep-quality section. The three shapes separate cleanly - progressive damage R^2 0.97, the obstructed sweep 0.24, the clean control flat at +0.35 A^2. And the label now follows the sign: damage fades the high-resolution intensity, so only a positive change is dose, where before any |dB| > 5 was called damage. Report-only throughout - the monitor never touches corr, and the per-batch curve's only consumer beyond the report is a sweep-quality field no reader reads; classification runs on the per-frame scale and CC, and is unmoved. The decay correction's global slope and the opt-in per-batch relative-B share this estimator and are left alone here: they fold into the scale, so Full 38-crystal rotation battery, twice (the second confirming the shell placement), against a clean baseline at the same base: space groups unchanged at 35/38 merge metrics move on three crystals only - the same three whose two-pass lattice search takes a different branch on nearly every arm run this session, one of which moves its own R_meas by 1.5 points on thread count alone A report-only change ought to be bit-identical and this is not quite, which is worth saying plainly: the three crystals that move are the known unstable ones and no space group moves, but "identical except where nothing is ever identical" is a weaker statement than "identical", and the residue has not been chased to ground. The three validation cases behave as they must: 60 deg beam-obstructed wedge, no decay -23.02, labelled damage, twelve batches printing the clamp -> NOT_A_TREND, curve within 3 A^2, the two unmeasurable batches absent, and a pointer to the sweep-quality section genuine progressive damage -39.46, sign inverted -> +94.50, monotone, corroborated by a per-image CC that falls 0.608 -> 0.159 and never recovers clean control +0.18 -> +0.76, flat within 1 A^2 Across the battery the report now names four crystals as radiation-damaged instead of ten; the other three are the two lowest-energy datasets and the pink-beam one, each showing a monotone rise of about ten square Angstroms. Sweep-quality classification is untouched, and the coupling that was assumed to exist does not: rad_damage_b_batch reaches it through one field that is written and never read. Ranges and reasons are identical on 35 of 38, the three that differ by one to seven frames are the same unstable crystals, and the census of stretches called radiation damage is one before and one after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>