d74ff7fc291ce82ac7ba9c00af973c30f5ffb663
1386
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  leonarski_fandClaude Opus 5
|
d74ff7fc29 |
Viewer: the settings panel says how the sample moved, and how polarized the beam was
Two experiment properties the panel could not state, both of which the analysis it drives has an opinion about anyway. The polarization factor is one number that corrects both communities' output - the azimuthal profile through AzimuthalIntegrationMapping and the integrated intensities through BraggIntegrationEngine - so it goes in the Geometry section, which the MX and AzInt pages already share. Files carry no polarization factor at all, so before this the interactive analysis integrated with none while "Analyze dataset" applied 0.99 from the rugnux defaults: the panel showed nothing and the two front ends disagreed. The viewer's starting experiment now takes the same rugnux defaults, so the panel shows what the analysis actually uses, and a processing job takes the panel's value over the default - as it already did for the scaling fields. The Goniometer section states which of the three things a dataset is - a still, a rotation, or a grid scan - which is exactly the choice a file makes at /entry/sample/transformations/omega vs /entry/sample/grid_scan vs neither. That choice IS the rotation/stills switch, so "Process as stills" is gone from the Indexing section rather than sitting beside it as a second control. It is not tied to what the file says: a still file can be given an axis or a grid, and a rotation file can be processed as stills. The inactive modes grey out but keep their values, so switching away and back does not lose an axis; a file that names none offers omega / -1 0 0 / 0.1 deg and a 10-point, 20 um raster. Only the fast axis of a grid gets a count field, because that is all there is: GridScanSettings derives the slow one from the image count, and the file stores n_fast alone. The grid the settings make is spelled out under them instead. The steps are signed - the sign is the direction the scan runs in - so only zero is rejected, and a half-typed entry falls back to the default rather than throwing out of a widget signal, which would abort the viewer. JFJochReader::UpdateGeomMetadata carries a fixed whitelist of the fields a panel edit may change, and it grew by three. Without them the new controls reset themselves on the next dataset refresh, since a non-whitelisted field comes back as the file's. The function has two call sites, both in the viewer's reading worker, and it is not virtual: objdump on the rugnux binary shows the code linked in and never called, so offline processing is untouched. RugnuxCommandLine emits --force-still from the axis the experiment carries, which the panel now clears when the mode is not Rotation - so the worker remembers the axis the file was opened with, and the copied command line is built with that put back. rugnux has no flag for the axis itself, so an axis edited or invented in the panel still cannot be expressed on a command line; the in-process run gets the experiment object and is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
e4ce573075 |
Bragg integration: keep a reflection that lost a wing, not one that lost its peak
MINPK asks how MUCH of the expected profile is readable. It does not ask WHERE, and the two are not the same question. The renormalisation argument the rescue rests on - a fit over a subset of a normalised profile is unbiased - needs the pixels to go missing for reasons unrelated to the reflection. A gap, a mask or the edge of the sensor is such a reason: the loss is set by the detector, and the fit renormalises over what is left. A pixel invalidated BY THE FLUX IT SAW is not: it goes missing because the reflection was bright, and it is the peak. Measured on the combined fulls, against the mean of the complete observations of the same reflection, in the innermost resolution shell of the high-multiplicity control and of a weaker crystal: a rescued reflection whose unreadable pixel sits within a pixel of the predicted centre reads |I - <I>|/I of 0.50 and 0.53, against 0.073 and 0.212 for a complete observation - 6.8x and 2.5x - and carries several times the mean intensity of its shell. On the control that is 0.21% of the shell's observations supplying 1.77% of the R_meas numerator; on the weaker crystal 0.52% supplying 6.82%. Rescues that lost only rim pixels are unremarkable by the same measure, 1.19x and 0.88x. Dropping the peak-losers alone takes the shell's R_meas from 7.440% back to 7.315% (unrescued: 7.307%) and from 22.03% to 21.06% (unrescued: 21.28%) - which is the whole of the low-resolution R_meas the rescue cost, and on the second crystal rather more. Raw frames say what they are. The pattern is a dead-centre invalid pixel with 5878, 9875 and 27583 counts around it: the detector's per-frame invalid marker on the brightest reflections. MINPK cannot catch them because it cuts on profile MASS, and the peak of a broad spot is a few percent of the mass. So a second condition, in the loop that already measures the readable fraction: no unreadable pixel may carry more than 0.9 of the profile's own peak value. A fraction of the peak rather than a radius in pixels because the peak is as wide as the spot - for a Gaussian the cut is at sqrt(-2 ln f) sigma, 0.46 sigma here, which is the peak pixel alone where sigma is 0.8 px and the crest of the ridge where the profile is a bandwidth streak. Swept against the alternatives on two crystals: a fixed radius needs 1.0-1.5 px to do the same work and costs 3-9x more observations for it, and 0.5 px does not reach the peak of a sub-pixel-offset prediction at all; tightening the fraction to 0.5 or 0.2 buys nothing beyond 0.9 and costs 7x more. Six crystals, three detectors, against the rescue as it stands: the rule keeps 99.86-99.96% of the recovered observations and returns R_meas to its unrescued value or below (4.6 -> 4.5%, 6.7 -> 6.6%, 25.1 -> 25.0%), R_meas in the innermost shell likewise (2.7 -> 2.6%, 5.9 -> 5.3% against 5.4% unrescued, 16.5 -> 16.4%), <I/sigma> up or level everywhere, and every unique reflection the rescue won is kept. Raising --overlap-minpk to 0.90 instead reaches the same place on two of them and short of it on the third, while discarding 0.8% of the recovered observations rather than 0.05%. An elongated pink-beam profile on a 9M detector and an EIGER2 16M dataset are both untouched at 99.9%, so the crest protection does not over-reject a streak. One crystal is not improved: a dataset whose error model rugnux declines to fit for want of strong reflections, whose <I/sigma> is <= 0 in eight of its ten shells and whose R_meas is undefined in as many. There the rule costs about 3% of <I/sigma> in the one shell that has signal, reproducibly, on top of the 9% the rescue itself costs there - while its overall R_meas moves 1.5 points on nothing but the thread count. The parity test gains four sections. Unreadable pixels were only ever punched into empty sky, so neither the rescue nor this rule had any CPU/GPU coverage at all; they now go into the signal disks - the peak of every fifth reflection, ~1.1 sigma out of every seventh, the disk edge of every eleventh - for both profile modes, a box sum and an elongated stencil, with a check that the clipping actually costs reflections so the coverage cannot go quietly vacuous. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery against the rescue without this rule, both on the same base: ISa better 19 / worse 4, +0.73 CC1/2 better 3 / worse 1, +1.3 R_meas_lo better 4 / worse 3, -0.3 space groups unchanged for 17 770 observations, 0.09 % of the run total and under 2 % of what the rescue had won. The two crystals whose peak-loss population was measured beforehand land on their predicted values: a tetragonal reference goes R_meas_lo 2.7 -> 2.6 % and ISa 27.11 -> 27.42, a cubic insulin 5.9 -> 5.3 % and 20.34 -> 20.65. One crystal pays: a cubic case with 2381 unique reflections goes R_meas 8.8 -> 9.6 % and ISa 4.08 -> 3.49. It is the crystal in the battery with the fewest uniques, so its rescued population is small and its shell statistics are coarse, but the loss is real and not noise in the R_meas. The R_meas sum over the battery reads +1.3, of which +3.2 is one crystal whose R_meas moves 1.5 points on thread count alone; without it the sum is negative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
cafc312681 |
docs: SHADOW_FINDER.md described a wiring plan that had already been carried out
The document opened with "It is not yet wired into any workflow" and carried a "Wiring plan (deferred - implement later)" whose first step was to add the source to image_analysis/CMakeLists.txt. That step, and the rugnux consumer, are both done: the finder runs by default over 60 equally-spaced frames and writes PixelMask bit 9. This is the same stale claim just corrected in CLAUDE.md, and this file is where it came from. The broker consumer sketched further down is genuinely still open, so that part is left as a plan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
3224ca9d8f |
docs: beam_stop is built and runs, not an unbuilt prototype
CLAUDE.md said `beam_stop/` was in no CMakeLists. It is in image_analysis/CMakeLists.txt:29-30, included at rugnux/Rugnux.cpp:27 and constructed at :126, where the shadow detection runs by default over 60 frames. An agent sent to look for a pre-indexing beam-centre estimator read the line, correctly took the machinery to be absent, and had to measure the code to find out otherwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_f
|
12ae6c5228 |
Bragg integration: fit a reflection over the pixels it has, not only over all of them
A predicted reflection was discarded outright if ANY pixel of its signal disk was unreadable - masked, untrusted, in a detector gap, or overloaded. On a battery crystal that is 11.1% of all predictions, thrown away for a defect in one pixel of fifty, and the pixels concerned sit at fixed places on the detector, so the loss is systematic in reciprocal space rather than random. Neither XDS nor dials does that. Both estimate the missing part from the profile instead and keep the reflection while enough of it was seen: XDS's MINPK (default 75%, "the missing intensity is estimated from the learned profiles"), dials' integration.profile.valid_foreground_threshold (default 0.75). MOSFLM is the one program that rejects by default, and even it relaxes to 50% with PROFILE EDGE. We already had the argument and the machinery: a profile fit is the amplitude of a NORMALISED profile, so leaving pixels out renormalises the estimator by construction - it costs information, which sum P^2/v duly loses and sigma duly gains, and biases nothing. That is exactly why --overlap exclude drops a neighbour's pixels from the fit rather than the reflection. Unreadable pixels are the same case with a different reason, so they take the same treatment, cut on the same threshold, in the same place: the readable fraction of the expected profile, measured against the profile mass that lands on the detector at all so a reflection is judged on the pixels that exist. A box sum has no profile to renormalise with and keeps the all-or-nothing rule. Two consequences handled. The summation seed and its variance now count the pixels actually read, and the runaway guard scales the fit back to that same disk before comparing - both exactly as before wherever nothing is missing. (Its fallback then hands back that partial sum unrescaled, which would read low; the guard fires on 8 of 96 688 recovered reflections, and on none at all on a weak crystal, so it is not worth a branch.) And the profile, its resolution shells and their widths are learned from COMPLETE reflections only, as is the box-sum centroid post-refinement reads as an observed position: a disk with a hole gives a centroid pulled away from the hole, and the hole does not move between frames. That sigma gains what the missing pixels carried is the claim the whole change rests on, and it is measurable. Force the conventional CENTRED cell of a body-centred crystal in P1: the predictor then enumerates every lattice point, and the reflections the centring makes systematically absent have a true intensity of exactly zero, so their scatter about zero must equal their reported sigma. Over 7.1 M such observations, matched by resolution shell, the trimmed std(I)/rms(sigma) of the recovered reflections is 0.99 / 1.20 / 2.33 / 1.04 / 1.69 against 0.98 / 1.22 / 2.29 / 1.03 / 1.56 for the reflections that were complete - the same calibration to a few percent. The lever there is small, because the typical recovered reflection is missing only 5% of its disk. Lowering the threshold to 0.50 admits a band missing 25-50%, which is a real lever: there sigma comes out 8-43% larger than a complete reflection's in the same shell, and the scatter about zero tracks it, 0.97 / 1.09 / 1.92 / 0.99 / 1.37, at or below the complete population. Sigma grows, and by the amount it should. The threshold stays at XDS's and dials' 0.75, on that evidence and on quality. Below it the estimator starts to run out: on those same zero-intensity reflections the recovered ones read +0.8 counts high at 0.75 and +1.9 counts high in the 0.50-0.75 band, against a sigma of 12-17, and at 0.25 the fit degenerates outright, single reflections carrying sigma in the thousands. Above it there is nothing to buy: 0.90 leaves a fifth of the recoverable observations behind and measures no better for them. On the high-multiplicity control, R_rim over as-shipped / 0.90 / 0.75 / 0.50 runs 4.49% / 4.51% / 4.56% / 4.78% while <I/sigma> runs 33.47 / 34.02 / 33.89 / 33.43 - 0.50 is where the recovered observations stop paying for themselves. Probe against the previous commit, six crystals. The high-multiplicity control gains 4.2% more observations, 924 803 -> 963 946, which lands it on XDS's 961 379 from the same images, for <I/sigma> 33.47 -> 33.89, R_rim 4.49% -> 4.56% at 4.3% more multiplicity, CC1/2 unchanged at 0.9998 and ISa 27.80 -> 27.12. Five weaker crystals gain 3.3-4.8% of their observations and up to 1.0 point of completeness, for <I/sigma> +0.4 to +3.6%, R_rim between -8.1% and +5.8% relative, CC1/2 +6.6 / +0.3 / +0.2 / -0.0 / -1.2 points, and ISa between +0.3% and -3.4%. Some of that ISa is the point rather than the price: a reflection integrated over fewer pixels carries less information, and the absence test above says the sigma that reports so is honest. The GPU and CPU engines agree as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Full 38-crystal rotation battery, against the same binary without it: observations better 38 / worse 0, +937 100 unique refl better 30 / worse 0, +9 229 overall <I/sig> better 33 / worse 1, +7.00 CC1/2 better 5 / worse 1, +6.2 space groups unchanged at 35/38 Every crystal gains observations and not one loses a unique reflection. The two costs are small and both are understood. Low-resolution R_meas is worse on eight crystals, by +0.8 pp at most and +3.2 pp summed - a reflection whose own peak pixel is unreadable loses the part of the profile that carries most of the amplitude, and that population sits at low resolution; the following commit handles it. And ISa falls on 32 crystals, by 10.9 summed, which is what admitting 937 000 further observations does to the strong-reflection asymptote: R_meas excluding the one crystal whose thread-count noise is 1.5 pp is flat. |
||
|
leonarski_fandClaude Opus 5
|
e0e429ae2f |
Rotation: pass 2 divides partiality by the width pass 2 measured
The second pass of the two-pass rotation loop widens its prediction window to the frame-order-smoothed mosaicity the first pass fitted, taking the max with this frame's own estimate so no reflection is dropped. That widened value was then reported onward as the frame's mosaicity, so it also became the divisor RotationScaleMerge recomputes every partiality from - a number chosen for prediction safety, applied to the intensities. The two uses are not symmetric. Prediction only decides membership: a reflection just inside a generous window arrives with a partiality near zero and is weighted as such, and widening the window by 28% was measured to move the merged statistics by under half a percent. The divisor multiplies every partial, and forcing it 34% wide on a rotation crystal cost ISa 10.7 -> 8.0 and R_meas 12.7 -> 14.5%. Pass 2 has also just re-measured the rocking width against the post-refined cell, which is the better of the two numbers - the literature's own remedy for a mosaicity fitted against a stale cell is exactly to re-estimate it after post-refinement (XDS documents this as a manual second INTEGRATE/CORRECT round). So keep the widened value where it was wanted, on the prediction window, and let the partiality divide by what the frame measured. The predicted reflection population is unchanged. Measured on three rotation crystals at their XDS resolution limits, identical partial counts in every arm: a weak monoclinic ISa 6.3 -> 6.7, R_meas 19.1 -> 18.7%, <I/sigma> 3.3 -> 3.5; tetragonal lysozyme ISa 27.1 -> 27.4, R_meas unchanged at 4.5%; a second monoclinic ISa 13.9 -> 13.8, R_meas unchanged at 12.3%. The full 38-crystal rotation battery then says the defect was almost never active: 36 of 38 crystals are untouched, and the reported per-image mosaicity moves on exactly one of them (0.3380 -> 0.3385 deg). The three-crystal probe above does not reproduce against the current baseline - lysozyme reads 27.81 -> 27.80 - because the max() rarely bites: pass 1's per-frame width is already smooth in frame order, its median frame-to-frame step being 0.0005 deg against a run spread of 0.028, so it seldom exceeds what pass 2 measures for itself. The two crystals that do move are marginal ones whose two-pass lattice search takes a different branch (validation frames 25/60 -> 26/60 and 48/60 -> 49/60); on that evidence their merge numbers measure the branch, not this change. No space group moves. So this lands as a correctness fix with no measurable effect on today's data, not as an improvement: the widened window must not become the divisor, whether or not the two happen to coincide on the crystals we have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_f
|
a0604ff244 |
Rotation: a diagnostic that opens the prediction window on its own width
Prediction and partiality read one number, the per-image sigma_M, so a sigma_M that moves takes the integrated reflection population with it and there is no way to ask which of the two uses carries a downstream difference. --prediction-mosaicity fixes the width the prediction window opens to while the partiality keeps using the measured sigma_M, which separates them. Measured with it on a rotation crystal whose lattice search returns two different cells a few tens of microns of detector distance apart: widening the prediction window from the narrower branch's 0.203 deg to the wider branch's 0.272 deg adds 28% more partials and moves the merged statistics by less than half a percent (I/sigma 6.1 -> 6.2, R_meas 12.7 -> 12.6%, ISa 10.7 -> 10.9); narrowing the wide branch the other way removes 26% of its partials and recovers nothing. On a tetragonal reference crystal a 4.7x over-wide window costs 11%. The prediction window is not where a mosaicity difference turns into a merged-data difference - the captured-fraction gate and the partiality weighting downstream absorb a generous window. Diagnostic only; off by default, so nothing changes unless it is asked for. |
||
|
leonarski_fandClaude Opus 5
|
677ece7b59 |
Post-refinement: correct a goniometer that turned further than it was told
The angles a rotation dataset stores are the COMMANDED ones, so a stage whose travel is miscalibrated leaves no trace in the header - every angle is self-consistently wrong. No existing parameter can absorb it either: the cell scale, the axis direction, the detector distance and the beam centre are all orthogonal to an error in rotation MAGNITUDE. So fit it as what it is - one scalar k, the ratio of the travel to the commanded angle - on the rocking events the geometry post-refinement already builds, after step A so the cell scale and the axis direction are fixed and k is the only free quantity. Two details decide whether the number means anything. The angle enters measured from the CENTRE of the sweep: the reference orientation was fitted against the commanded angles and has already absorbed their mean error, so measured from the goniometer's zero instead a constant missetting about the spindle leaks into k with a gain of <phi>/<phi^2>, which depends only on where the sweep happens to sit - on a short sweep starting near zero a 0.14 deg missetting fakes 1.4 % of k. Referred to the sweep centre that leak is identically zero at any width. And the robust loss is scaled to the scatter the events actually have, which varies by more than a decade between datasets, so any fixed constant is either inert or throws away real data. A stage fault is rare and a 1 % angle correction applied to a healthy dataset would damage it silently, so the correction is committed only when every test passes: at least 30 deg of sweep and 5000 events, |k-1| over 0.5 %, a misorientation of at least 0.5 deg at each end of the sweep, and the same k from every fifth of the sweep left out. The last test is not optional. A second lattice that dominates ONE END of a sweep - exactly what happens where the primary stops indexing - fakes a k that passes the other two, and the hkl-hash split used elsewhere in this file cannot see it, because both of its folds sit at the same angles and anything structured in phi survives in both. When it commits, the second pass re-integrates against the corrected angles. The pre-pass mosaicity is dropped with it: that is a width in degrees fitted against angles the second pass has just stopped using, and since the override can only ever raise the second pass's own estimate, carrying it over would hold the second pass at the rocking width the uncorrected angles produced - the correction half-applied. --rotation-scale asserts a known stage calibration by hand and overrides the fit. On the 38-crystal rotation battery the gate fires on exactly one dataset, at k = 1.01318 with 0.74 of that k surviving every fifth left out. The largest of the other 37 is 1.00211, which fails the end-error test; 34 of them sit below 1.0006. On the one that fires: R_meas 39.2 -> 23.9 % (XDS 37.1) CC1/2 86.5 -> 96.0 % (XDS 94.3) CC1/2 outer 1.4 -> 53.4 % (XDS 42.5) unique refl 40990 -> 41540 (XDS 41322) observations 74975 -> 103858 (XDS 129322) mosaicity 0.181 -> 0.159 deg which takes it from losing to XDS on R_meas, CC1/2 and outer-shell CC1/2 to beating it on all three, and the mosaicity drop is the inflation the uncorrected angles were producing. Its low-resolution R_meas is the one number that moves the wrong way, 12.0 -> 13.9 %, still well inside XDS's 18.3. No space group moves anywhere, and every other crystal's merge is unchanged beyond the two-pass loop's own jitter - measured here as the spread of the post-refined distance across arms that do not touch post-refinement at all, which is larger than anything this commit produces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
ce3199748d |
Scaling: report the stretches of a sweep the crystal did not deliver
Build Packages / build:viewer-tgz:cpu (push) Successful in 19m5s
Build Packages / build:viewer-tgz:cuda (push) Successful in 22m34s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m21s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 24m15s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 28m59s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 29m12s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 29m4s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 19m35s
Build Packages / XDS test (durin plugin) (push) Successful in 11m6s
Build Packages / build:rpm (rocky9) (push) Successful in 20m57s
Build Packages / Generate python client (push) Successful in 46s
Build Packages / Build documentation (push) Successful in 1m29s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m36s
Build Packages / build:rpm (rocky8) (push) Successful in 27m38s
Build Packages / DIALS test (push) Successful in 21m0s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 11m35s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 21m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m27s
Build Packages / Unit tests (push) Successful in 1h18m5s
Build Packages / build:windows:nocuda (push) Successful in 19m18s
Build Packages / build:windows:cuda (push) Successful in 23m19s
Rotation processing no longer refuses to integrate a frame that fails to index on its own, which is
right - no other program does that - but it means a genuinely bad stretch of a sweep is now
integrated instead of quietly dropped. Some sweeps have a real problem behind that stretch: the
crystal partly or wholly out of the illuminated volume, off the rotation axis, or dying of dose.
That is actionable at the beamline ("recollect", "re-centre"), and until now nothing said it.
MeasureSweepQuality reports it as contiguous RANGES, never per-frame flags, and reports only - no
observation is excluded on the strength of it. A single weak frame is noise; forty consecutive ones
are a fact about the experiment, and the frames still carry signal worth merging.
The discriminator is that the incident flux is already out of the per-image scale before that scale
is fitted (DivideOutIncidentFlux runs from Ingest), so a drop in G that the beam does not explain is
on the sample side by elimination. Measured on one crystal with a dead arc: the flux proxy spans
1.4x across the run where the fitted scale spans 246x.
A range needs BOTH per-frame channels down: the scale, and the CC to merge. The CC channel is what
keeps a merely attenuated stretch out - absorption and flux scale a frame's intensities without
changing how well they correlate with the merged reference. Without it the clean high-multiplicity
control, whose per-image scale swings 4x on a 180 degree period, would be reported as a bad crystal.
It is not: it produces no ranges at all, and neither does the other control.
Five codes, each the field's own words and each a phrase a report can print:
no diffraction - essentially nothing was recorded from the indexed lattice over the range
out of beam - frames were lost: the range gets a scale far less often than the run does
weak diffraction - the frames all still index, with much less intensity; cause not determined
loss of centring - one cycle of modulation per revolution (autoPROC's words for the phenomenon)
radiation damage - the range runs to the end of a sweep whose quality was already decaying
Only the last two claim a cause, and each rests on its own evidence. Damage is progressive, so it
must have been setting in before the range and must not recover. Loss of centring rests on the one
signature that breaks a documented degeneracy: Evans (Acta Cryst. D62, 72-82) notes that illuminated
volume and absorption are indistinguishable, but a crystal's own shape absorbs on a 180 degree
period, so a dominant 360 degree fundamental over a full turn cannot be the crystal's shape. That
test runs on the total scale, flux included, unlike everything else here - the flux proxy is a
background, a crystal leaving the beam takes its own scattering with it, and the beam cannot be
periodic in an angle it does not know. Where the evidence does not reach, weak diffraction says so
rather than guessing.
Frame numbers are processed-image ordinals, inclusive at both ends, the numbering of _image.dat.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 5
|
e5c0066129 |
rugnux: write a results report next to the reflections
Everything a run determines went to stdout and nowhere else. The space group and the evidence behind it, the error model, the post-refine commit-or-reject decisions and their held-out residuals, the two-pass adopt-or-roll-back, the resolution cut, the merging statistics - all of it scrolled past interleaved with progress lines and was gone. A user who was not watching had no record, and nothing could read it. `rugnux` had no log file at all; the `rugnux.log` in the regression harness is that harness capturing stdout. Write `<prefix>_report.txt` alongside the .cif/.mtz/.hkl, always, with no option to ask for it. It holds what the run DETERMINED; timing, rates, per-image progress and engine chatter stay on stdout, where they belong. Every line rugnux logs was classified result-or-process against the regression corpus to decide what crosses over. The format follows XDS's CORRECT.LP, which has been read by people and parsed by other programs for twenty years: `KEY= value` assignment lines a script greps one at a time, fixed-width tables with stable headers and a total row, `WARNING:` sentences in plain English, section banners. REPORT_VERSION says when that interface last changed. It is assembled from results the pipeline already computed, so an unconditional file costs nothing, and a failure to write it is logged and swallowed - a run that produced good reflections must not be lost to a side file. One thing CORRECT.LP does not have to solve: a rotation run integrates twice and writes both passes, so every report says which pass it describes and why that pass was adopted. `--no-merge` gets a report too, saying MERGE= NOT_PERFORMED rather than leaving a reader to infer it from absent sections. An empty output prefix still writes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
b9078d9a59 |
Scaling: drop a frame whose scale collapses, do not merge it unscaled
Two guards catch a per-frame scale far below the run median. Both then invented a value for it - one substituted the run median, the other set corr = 1 and merged the frame "unscaled". For a frame whose scale really is 1/17402 of its neighbours', asserting 1 is worse than asserting nothing, and it is the assertion that does the damage: those observations enter the merge at full weight carrying an intensity scale that is wrong by four orders of magnitude. It surfaced when rotation started integrating every frame the sweep's lattice explains, but it is not caused by that change - six crystals in the battery already tripped these guards before it. What the extra frames did was find a crystal where the collapsed population is large enough to dominate: R_meas 19.1 -> 90.1%, ISa 25.60 -> 4.58, from 17% more observations. The frames are not sparse and the fit is not running away. A per-frame dump shows 3394 observations on the median collapsed frame against 3469 on live ones - the scale is over-determined 3400:1 for one parameter - and 113 frames fit exactly zero. They form one contiguous arc of about 68 degrees once the sweep's wrap is accounted for, over which the per-frame correlation to the merge is 0.035 against 0.85 elsewhere, while the flux measured from the background varies by only 1.55x. So the fitted zero is a well-determined measurement that the frame holds no diffraction from this lattice, not a failure to measure. The frames are empty, not under-determined. That is also why the smooth or shrunk alternatives do not apply, and both were built and measured rather than argued away: giving a collapsed frame the geometric mean of its credible neighbours is worse than the baseline (R_meas 115.3%), because it merges noise at the weight of a good frame, and a dead region 112 and 232 frames wide has no local neighbourhood to borrow from in any case. Dropping them: R_meas 90.1 -> 38.3%, low-resolution R_meas 26.9 -> 10.2% (past XDS's 14.3), ISa 4.58 -> 22.00, CC1/2 99.0 -> 99.9, with 5.4% more observations retained than before frames were integrated at all. Over the full battery, against the same binary without either change, ISa moves from -22.5 to -3.4 summed, CC1/2 +24.6, and 216247 more observations. The crystal that motivated the integration change is untouched by this one, bit for bit. The detection and the MIN_CREDIBLE_SCALE_RATIO threshold are unchanged. Note that threshold is now marginal: its own comment records 0.070 as the smallest legitimate ratio seen, and one crystal here has a legitimate live frame at 0.026, so it cannot be raised to catch the partly-dead transition frames at the edges of an arc without risking real data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
9885d1fc28 |
Rotation: integrate every frame the sweep's lattice explains
A rotation dataset has ONE lattice. Once the first pass has found it and the goniometer gives each frame its orientation, every frame of the sweep is a frame of that crystal - yet integration was gated on each frame re-indexing on its own, a test that carries an absolute floor of 9 indexed spots. A weakly diffracting crystal shows a handful of spots per image while the geometry still puts ~1500 reflections on the detector, so the floor threw away whole frames that had nothing wrong with them. Measured on a 360-degree battery crystal: 1484 of its 1800 frames failed that gate, all of them on the spot-count floor alone and none on the consistency test - the median failing frame had 4 spots and the lattice indexed all 4. Integration therefore ran on 17.7% of the sweep and the merge came out 35.7% complete at multiplicity 1.1, against XDS's 97.7% at 2.81 from the same images. XDS's own INTEGRATE.LP shows why the floor is the wrong test there: 964 of its frames have fewer than 9 strong spots and it predicts ~1483 reflections near the Ewald sphere on every one of them, because INTEGRATE works from the global orientation and has no per-frame indexing gate at all. Neither does dials.integrate. Split the one verdict into the two questions it was answering. "Does this frame index?" - what the indexing rate reports and what the first pass scores candidate lattices on - keeps the floor, because a handful of spots sit on almost any lattice by chance. "Is this frame worth integrating?" keeps only the consistency part, and only where the lattice does not come from this frame. A frame whose spots largely MISS the lattice is still refused: on another battery crystal that is 35% of the sweep, and integrating those collapsed the space group to P1 - the floor had been shielding the merge from frames the model does not describe, which is a different defect and not one to paper over here. Two consequences had to be handled. A frame that is too sparse to index is also too sparse to fit its own rocking width, and the placeholder it used to predict with was being reported onward as if measured, into the frame-order average that recomputes every partiality; report nothing instead, and fill the gaps in that average with the run's median rather than a fixed default. Probe (XDS in brackets): the crystal above goes 9 700 -> 81 956 observations, 8 618 -> 23 960 unique [23 576], 35.7% -> 99.4% complete [97.7%], R_meas 21.2% -> 68.6% [76.7%], CC1/2 96.0% -> 86.4% [81.1%], low-shell R_meas 7.2% -> 14.3% [20.6%], ISa unmeasurable -> 13.8 [10.4] - better than XDS on every statistic, where before it was merging a third of the data. A second crystal gains 41% more observations with R_meas 12.6% -> 8.5% and ISa 3.3 -> 3.7. The high-multiplicity control is unchanged to 2 observations in 924 782, and four further crystals move within recompilation noise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
d8029524e7 |
Scaling: never let an observation's own fluctuation set its weight
A weighted mean is only unbiased while the weights are independent of the values
being averaged. The IUCr's own nomenclature report (Schwarzenbach et al., Acta
Cryst A45 (1989) 63-75) puts it directly: weights in averaging "should not be
based on the counting statistics of the individual observations whose estimated
variances are biased and result in larger weights for accidentally low
intensities". Two places in the rotation pipeline were doing exactly that, and
between them they drove whole resolution shells of merged intensity negative.
1. The profile fit computed its non-signal variance as
var_bkg = max(0, 1/den - max(0, I) + bkg-estimate term)
The point of a separate var_bkg is that it does NOT move with the
reflection's own fluctuation, and 1/den - I is the quantity that does not:
1/den is the fit variance taken at the fitted intensity and grows with it
roughly one for one. Clamping the subtrahend at zero left a down-fluctuated
reflection's own deflated variance standing as its background variance.
Measured over 6.9 M partials of one weak rotation dataset, var_bkg/bkg came
out at 3.7-5.4 for observations with I < 0 against 11.4-13.7 for I > 0 - the
down-fluctuated half of every reflection carried a variance ~2.7x too small
and was weighted up by the same factor, first in the 3D combine and then
again in the merge. Removing the clamp makes var_bkg flat in I (~13 x bkg
across the whole range).
2. The merge then weighted each combined full by 1/sigma_full^2, and sigma_full
is by construction a function of the full's own answer: the combine's
variance carries a corr*max(0, F) signal term, so every full with F <= 0 got
the smallest variance the model allows while the strongest quartile got
2.26x more. The merge now rebuilds that variance at the reflection's mean
instead, from a linear model var(I) = var_bkg + var_per_I * I that the
combine measures and stores on the full. This mirrors
MergeOnTheFly::CorrectedSigma, whose comment already claimed to mirror the
rotation combine.
Verified against an estimator that cannot see the fluctuation - summing the
partials and dividing by the summed partiality, the classical construction every
other program uses (Greenhough & Suddath, J. Appl. Cryst. 19 (1986) 400-409, via
Leslie, Acta Cryst D55 (1999) 1696-1702: profile fitting biases the individual
partials but not their sum). Reproducing the merge on dumped observations, the
shipped weighting sat ~1.9 sigma below that reference in the noise shells; the
two changes recover most of it, and every intensity-independent weighting
scheme agrees with the reference once (1) is in.
Four-crystal probe, XDS resolution limits, branch fingerprint identical on all
four (so none of these is a two-pass branch flip):
weak cubic case last shell <I/sig> -1.6 -> +0.2 (XDS +0.10), last shell
R_meas 478% -> 250% (XDS 246%), overall <I/sig> 6.1 -> 7.5
(XDS 7.18), R_meas 18.3% -> 18.1%, CC1/2_hi 38.2% -> 43.7%
tetragonal case outer shells <I/sig> -0.4/-0.8/-0.9/-1.0 -> +1.8/+1.2/
+0.9/+0.4, R_meas 184%/595%/7614%/nan -> 95%/119%/135%/232%
(the nan was the shell mean crossing zero), R_meas 33.3% ->
32.9%, CC1/2_hi 38.3% -> 56.5%
trigonal case R_meas 13.0% -> 12.5%, CC1/2_hi 14.4% -> 16.5%
strong control unchanged to every printed digit but ISa
Cost: ISa falls (17.2 -> 14.0 and 16.7 -> 14.9 on the two mid-strength cases,
28.3 -> 27.8 on the control). Strong reflections are untouched by (1) - their
partials are all positive, so var_bkg is bit-identical - but the joint a/b fit
redistributes: honest weak sigmas lower a, and b rises to keep the strong bins
fitted. The median reduced chi^2 improves (1.25 -> 1.14, 1.35 -> 1.28) so the
new split describes the scatter better, but ISa is the one headline metric that
moves the wrong way and it should be watched over the full battery.
The integrator change is shared, so the stills merge sees it too; there it feeds
GetExpectedVarianceMerge, which had been handed the same contaminated var_bkg.
That path is untested here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
leonarski_fandClaude Opus 5
|
1eb9d5a2fa |
Notices: list traccc in the canonical third-party manifest
The SparseCCL connected-component labelling adapted from traccc is MPL-2.0 source in this tree, and MPL-2.0 requires the notice to be conveyed with it. The licence text has always shipped (licenses/traccc.txt, installed with the rest of licenses/ into share/doc/jfjoch) and docs/ACKNOWLEDGEMENT.md has always credited it - but the root THIRD_PARTY_NOTICES.md, which is the manifest a recipient actually reads, never named it. The generated docs/ copy DID name it, and more completely than a fresh row would have: both files it touches, and a note explaining that only one of them adapts the source while the other follows the design. So the fix is to carry that entry back into the canonical file rather than write a new one - the generated copy had been edited directly at some point, against its own banner, and the edit never reached the file it is generated from. Regenerating now reproduces docs/ byte for byte, which is the check that the two are finally in step. update_version.sh gains the link rewrite that entry needs: the acknowledgement is docs/ACKNOWLEDGEMENT.md from the root and ACKNOWLEDGEMENT.md from inside docs/, so without the rule the next regeneration would have written a path that resolves nowhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
43627e22dc |
docs: a rule for licences and academic credit, and apply it
Several methods adopted recently came from other crystallographic packages - the screw-absence test from POINTLESS, MINPK and the profile-fit reweighting from XDS/Otwinowski, the CC1/2 cutoff and merge outlier rejection from DIALS, the per-frame indexing gate from CrystFEL - and nothing in the repository said where such a debt is recorded. The licence side was already worked out (licences beside the vendored code, verbatim texts in licenses/ collected by COLLECT.sh, a row in THIRD_PARTY_NOTICES.md, all installed under share/doc/jfjoch); the credit side was ad hoc. Write the rule into CLAUDE.md. It states the distinction that matters: vendoring or linking someone's CODE creates a LICENCE obligation, discharged in licenses/ and THIRD_PARTY_NOTICES.md; reimplementing an algorithm from a PAPER creates none of that but creates an obligation of academic CREDIT, discharged in docs/ACKNOWLEDGEMENT.md and in a comment at the algorithm. Neither substitutes for the other, and taking both source and paper incurs both. It also fixes the citation form (authors, title, year, journal, volume, pages, verified DOI), and says in-source credit goes at the algorithm, not the file header, in the one-line style the code already uses. Then bring the repository into compliance for the works concerned: docs/ACKNOWLEDGEMENT.md gains a section acknowledging XDS, DIALS, POINTLESS/CCP4, MOSFLM, CrystFEL, GEMMI, the Kabsch/Otwinowski profile fit, the Diederichs & Karplus statistics and the IUCr nomenclature reports, each with a DOI checked against Crossref; docs/CPU_DATA_ANALYSIS.md's reference list gains the ones it was missing; and four algorithms gain a line naming their source where no adjacent comment carried one. No licence change. licenses/ and THIRD_PARTY_NOTICES.md are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
6d39a4e1ab |
Space-group search: decide a screw from the evidence, not from a count of absences
A screw's predicted-absent class was required to hold min_absent_observed = 8 reflections before the screw could be claimed. That count is the wrong measure of evidence, and it is wrong in both directions. A screw extinguishes one row of reciprocal space, and that row is often the one a rotation sweep records least: it lies near the spindle, where the blind cusp maps onto itself and symmetry cannot fill it in. Counting it measures the geometry of the sweep. A monoclinic crystal whose 2-fold sits 7.6 deg from the spindle contributed six 0k0-odd reflections, every one of them measured between -0.013 and 4e-5 of the shell mean with zero violations, against a 0k0 row averaging 1.44x the shell mean - and was refused its 2_1 for being six rather than eight. XDS's own integration of the same images finds seventeen of those reflections and every one of them is likewise dead. The count is equally wrong the other way: a uniformly weak axial row produces no violations at all, so with enough reflections on it a screw is claimed from no evidence whatsoever. The second new test section demonstrates exactly that on the old gate. Judge the class by how unlikely it would be if the screw did not exist. Under "no screw" the absent class and the rest of its row are both Wilson-distributed with the same mean, so with each absent intensity taken in units of its row's control mean, sum_u/(sum_u + n_control) follows Beta(n_absent, n_control) exactly; the reported evidence is -log of that lower tail. The row's own strength cancels, which is the property the count lacks, and the scale is set by the number of reflections, so few-but-decisive and many-but-marginal are told apart. It is sigma-free by design: the merged sigma carries the error model's intensity-proportional term and so shrinks with I, reading much the same on an absent reflection as on a present one. This follows POINTLESS (Evans, Acta Cryst D67, 282-292 (2011), Appendix A3), which likewise scores an absence against the rest of its own axial row rather than against a global mean or a fixed cut, and likewise lets confidence fall away with the number of axial reflections instead of refusing outright below a count. POINTLESS calibrates its null width from control transforms of non-axial reflections; the Beta tail here is an analytic null in its place. XDS is not a reference for this: it "deliberately avoids any test for the presence of screw axes as these tests would depend strongly on the completeness of the data" (Kabsch, Acta Cryst D66, 133-144 (2010), section 6), so a screw axis in a CORRECT.LP was supplied to it, not determined by it. Measured over five probe crystals, genuine screw conditions read 34-800 nats and false ones - the 4_1/4_3 conditions of a cubic crystal that has no screw, whose predicted-absent class is STRONGER than its control row - read -7 to -8.5. The bound is set at 20, in the gap, at p <= 2e-9: three well-measured dead axial reflections clear it and two do not. min_absent_observed keeps its job for CENTERING, where a count is a fair measure - that class is a third to a half of every reflection in the data set and the bound is never binding on a centering that exists. The candidate table now prints the screw-absent count and this evidence in place of the two E^2 medians that were its raw ingredients, so a refusal can be read off the log. Measured on the five probes: the monoclinic crystal above returns to P2_1 with every merge statistic unchanged (R_meas 58.8 -> 58.7%, CC1/2 49.1 -> 49.3%, ISa 6.61 -> 6.59 - P2 and P2_1 share a point group, so only the symbol and the absent reflections differ). The other four are untouched, space group included, and the two-pass branch fingerprint (indexed frames, distance, mosaicity) is identical on all five. The full battery has not been run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
7bbf072ad2 |
Scaling: do not report an ISa that was never measured
The error model's systematic term b is identified only by the spread of I^2/sigma^2 across the intensity bins the fit uses, and those bins hold equal COUNTS. So when fewer reflections are strong than one bin holds - a sixteenth of the pool - the top bin's median sits at an intensity where b cannot be measured at all, and the fit hands it the bins' own noise-selection slope instead: sorting noise by its group mean squared makes dev2 rise with I2 even when the true b is zero, and with no strong bin to out-vote it that slope becomes b. The result is not a small error. On the battery's weakest crystal, 2.2% of whose fulls reach I/sigma 2, the fit returns b = 5.6 - sigma -> 2*I at the strong end - and since corrected_sigma applies b at the GROUP MEAN, sigma^2 = a*sigma^2 + (b*mean)^2 is a per-group constant that caps merged |I/sigma| at sqrt(n)/b. The cap lands at 2.3, so 98.8% of merged reflections come out below 3 and the reported ISa is 0.50, on data whose CC1/2 is 99.3% at multiplicity 18.7. XDS fits 6.13 from the same images. Feeding XDS's own scaled observations through this estimator returns 0.84, so it is the estimator and not the data; synthetic data built with b = 0 and 1.8% strong reproduces a = 0.51 and ISa 0.50 to two digits, and recovers the truth as soon as the strong fraction passes one bin. So refuse to report what was not measured: when the strongest bin's own (I/sigma)^2 is below 4, fit a alone, hold b at zero and warn that ISa is unmeasured. The threshold is not delicate - the two crystals it fires on sit at 0.22 and 0.84 while the next crystal in the battery is at 31.7 and a healthy one at 342, so anything from 4 to 25 selects the same two. Full 38-crystal rotation battery: it fires on those two crystals and no others, and space groups are unchanged at 35/38. Dropping the spurious term also fixes the merge weights it had been distorting - on the worse of the two, R_meas 19.2 -> 13.6%, low-resolution R_meas 13.8 -> 6.9% against XDS's 14.1%, CC1/2 98.7 -> 100.0%, with chi2 1.11 on the one-parameter model. Two further crystals move slightly; the guard never fires on either, and they are marginal crystals of the kind whose two-pass branch any recompilation can shift. This reports the parameter as unmeasured rather than clamping it to something plausible, because the honest statement is that the data do not reach far enough for a systematic error to be seen - not that there is none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
b40abe31cf |
Scaling: one exact-Bragg angle per rocking event
Every partial's delta_phi was solved from its OWN frame's lattice - by the predictor, and again by SmoothGeometry. Per-frame geometry is re-refined against that frame's spots alone, so what is left of its jitter entered each frame of an event independently and the frames of one rocking event stopped sitting exactly one oscillation apart on the curve. Their partialities then no longer tile it, and because a broad rocking curve spans more frames, the error grows as 1/zeta - which is how it has been showing up: a zeta-graded systematic that nothing in the integrator could reach. For the frames of one event the geometry is exact. Each frame has already turned one oscillation further, so delta_phi is linear in frame number with slope minus the increment; the sign is checked against the data rather than derived, the measured mean frame-to-frame slope being -0.19996 deg/frame at an increment of 0.20000. Fit the one free number, the offset, over the event and lay its partials back on that line. The rms departure removed is 0.25 deg - larger than the oscillation itself, because a small orientation wobble is amplified by 1/zeta. The raw-hkl runs the merge already builds give the grouping, so this costs one pass over the partials and no extra sort. Full 38-crystal rotation battery against the same binary without it, on unchanged data (observations +0.20%, unique reflections +0.03%, so none of this is selection): R_meas_lo better 22 / worse 5, summed -41.0 pp; excess against XDS -46.9 -> -87.9 R_meas better 22 / worse 5, summed -24.7 CC1/2 better 17 / worse 2, summed +36.8 ISa better 15 / worse 22, summed +6.87; shortfall against XDS 28.1 -> 21.2 space groups unchanged at 35/38 The low-resolution R_meas gains land on the crystals that have carried this gap: 19.3 -> 10.8, 20.7 -> 13.8 (now past XDS), 25.6 -> 19.7, 17.5 -> 12.4 per cent. Exactly one crystal shows any change in the two-pass branch fingerprint, so unlike most changes on this path the result is not confounded by that bistability. ISa falls on more crystals than it rises, and that is the estimator becoming honest rather than the data getting worse: every crystal whose ISa dropped materially was over-optimistic against its own R_meas_lo and moved toward consistency, and the median ratio of reported ISa to the value its own R_meas_lo implies goes 1.09 -> 1.01, against 1.19 for XDS. The one real loss is a crystal going 1.11 -> 0.95 on that ratio. High-shell CC1/2 is worse on 22 crystals, by about 1.2 points each. It is the one metric that dissents, and it is also the one that has failed as an arbiter repeatedly on this data, while overall CC1/2, R_meas, R_meas_lo and reflection count all improve on an unchanged number of observations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
6f7b136ec2 |
Bragg integration: a shared signal pixel belongs to the nearer reflection
Nothing kept a neighbour's flux out of a reflection's own signal disk. The union mask keeps neighbour cores out of the BACKGROUND ring, but the r1 disk was read whole, so on a dense pattern a crowded reflection measures part of its neighbour as its own. Ownership is decided once per image into a per-pixel (quantised distance, reflection) key written with an atomic minimum, so the nearest predicted centre wins whatever order the writes arrive in and the lowest index breaks a tie. `--overlap exclude`, now the default, drops the pixels a nearer neighbour owns from the profile fit. A profile fit is the amplitude of a normalised profile, so leaving pixels out renormalises the estimator by construction and the reflection stays unbiased rather than being discarded; the summation-fallback guard is scaled back to the disk the box-sum seed actually read, so it still compares like with like. `--overlap reject` is the XDS MINPK alternative - drop the reflection when less than `--overlap-minpk` of its expected profile is cleanly its own. A box sum has no profile to renormalise with, so `exclude` is a no-op there and only `reject` acts on it. Widening the split - keeping a pixel only where no other centre is within its distance PLUS a margin - was built and measured, and it is worse monotonically: the residual bias of the pixels that were kept grows from +0.072 to +0.209 in ln intensity at 0 to 3 px of margin. What the margin removes is the reflection's own profile, not the neighbour's tail, so the plain nearest-centre split is the rule. Measured on the full 38-crystal rotation battery against the same binary with the treatment off: ISa better 15 / worse 8, summed shortfall against XDS 39.7 -> 28.1. Three of the losses are the two-pass loop taking its other branch - their median mosaicity moves between the two known attractors - rather than the change under test; excluding those it is better 15 / worse 5 and the shortfall goes 31.3 -> 14.4. The two crowded crystals gain 38% and 52% of their ISa, one of them passing XDS. High-shell CC1/2 over the 35 crystals that neither flipped branch nor carry a collapsed error model is better 7 / worse 7. Space groups unchanged at 35/38. The owner map is built only when a treatment is asked for and costs 1.1% of the battery's wall clock - 23% on a genuinely crowded crystal, nothing where no two predictions touch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
7627c282ba |
Battery: one unreadable column must not discard a resolution shell
The comparison read each shell as float(d), float(R_meas), float(CC1/2) inside a single try, so a shell with any column unset was dropped whole. R_meas is unset exactly when the mean intensity in that shell has gone non-positive - which happens on the WORST arm - and dropping the row took its CC1/2 with it. The high-resolution CC1/2 then silently came from the next shell in, and the arm whose outer data had collapsed was reported as the better one. Measured on one crystal where two integration settings were being compared: at the same 1.74 A shell the two arms are 61.5% and 17.8%, and the table printed 61.5% against 36.9% - the second number being the other arm's 1.83 A shell. The error is not a rounding matter and it points the wrong way. Each column is now read on its own, and each statistic is taken from the outermost (or innermost) shell that actually carries it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
06a5a118bf |
Bragg integration: widen the background ring to r3 = 13
The background is estimated from the r2..r3 ring and then subtracted from every pixel of the r1 disk, so the ring mean's own error enters the intensity n_inner times over: var(I) carries n_inner^2 * bkg / n_B. That term is first-order in sigma, and it is set by how many pixels the ring holds - not by anything about the reflection. At r3 = 10 the ring holds about 200 px against the disk's 50. Widening it to 13 roughly doubles that. The signal disk is untouched, and the pixels gained lie further from the reflection rather than nearer, so nothing is traded for them. The effect is not subtle once looked for. Matched observation by observation on one crystal, halving the ring's pixel count leaves the intensity alone and inflates sigma by 4.7%, and the inflation rank-orders with the ring collapse across the battery. Over the whole rotation battery, against the same binary at r3 = 10: ISa better on 14 crystals and worse on 4, the summed shortfall against the reference 164.7 -> 155.9, the summed low-resolution R_meas excess 69.7 -> 59.1 percentage points, and one more crystal reaching the reference space group (33/37 -> 34/37, a trigonal case that was over-promoting). Largest gains where the ring was starved worst; the four losses are 0.25 to 2.16 in ISa and none of them changes a space group. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
e0bd208666 |
Prediction: measure the rocking curve against the frame's edge, not its centre
A reflection was accepted onto an image when |delta_phi| * zeta was within the mosaicity window, where delta_phi is the offset from the frame's mid-exposure angle to the exact diffracting condition. That asks whether the frame's CENTRE lies inside the rocking curve, which is a stricter question than the one that matters: whether any of the curve lies inside the frame's exposure. The two differ by half a wedge, and the partiality computed a few lines further down already integrates over that half wedge on both sides - so the acceptance test and the quantity it gates disagreed about where the frame is. The consequence is not a clipped intensity but a lost reflection. Consecutive frame centres are one wedge apart, so the nearest centre can be half a wedge away; once the window is narrower than that, the reflection fails the test on its best frame and on every other, and is never predicted at all. That happens when sigma_eff < zeta * wedge / (2 * mosaicity_multiplier) - coarse slicing on a sharp crystal at high zeta, which is where a reflection is fully recorded on one image and measured best. Subtracting the half wedge from the tested offset restores the intended question. On a crystal that reaches the regime (0.4 deg per image, fitted sigma_M 0.051 deg) low-resolution R_meas goes 6.8% -> 5.4% and ISa 13.3 -> 14.1. Elsewhere the window merely widens by half a wedge, which admits partials whose partiality is a few parts in a thousand; those are correctly measured and correctly down-weighted, and four of the six crystals tested do not move, while one loses 1.2 ISa. Both engines carry the same test and both are changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
27d0b1db74 |
docs: state the reflection-file conventions
The mmCIF carries eleven items rugnux invents, and the rule they follow - a jfjoch_ prefix inside whichever standard category the quantity belongs to - was nowhere written down, so the only way to learn what was in a merged .cif was to read WriteReflections.cpp. Tabulate them, with the values a reader needs in order to interpret each one (the untwinned and perfect-twin values for the L test and the second moment, the sign convention for the radiation-damage B). Two of them need more than a name. The compatibility note records that jfjoch_diffrn_ISa changed meaning and that a file carries no marker saying which. And the HKLF-4 .hkl has two properties that are invisible in the file and change what a comparison means: Bijvoet mates are separate records, and the intensities carry a single global rescale so the largest fits F8.2 - harmless to SHELXC and ANODE, which use ratios, but not something to compare magnitudes across. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
adf87e8675 |
Merging: export the XDS-comparable ISa under jfjoch_diffrn_ISa
The mmCIF's _reflns.jfjoch_diffrn_ISa carried the strong-reflection asymptote, a tier XDS has no equivalent of, while the name invites comparison with XDS's ISa - which is the whole-range 1/sqrt(a*b). rugnux_vs_xds.py reads that item for the battery's ISa column, so the comparison that column exists to make was between two different quantities, flattering rugnux by the difference between the tiers. Write the whole-range value there, move the asymptote to _reflns.jfjoch_diffrn_ISa_asymptotic, and add _reflns.jfjoch_error_model_a and _b in XDS's convention so the number can be re-derived from the file rather than taken on trust. On a broadband rotation dataset the battery column now reads 13.25 against XDS's 21.18 where it read 15.6 before, and the two error models can be compared term by term for the first time: a 1.538 vs 1.249 and b 3.71e-03 vs 1.78e-03, so the gap is in BOTH the counting and the systematic term (1.23x and 2.08x, and sqrt(1.23*2.08) = 1.60 = 21.18/13.25). This is a deliberate redefinition of an exported item, not an addition: a file written by an earlier version carries the asymptote under the old name and there is no version marker to tell them apart. Noted in the changelog and in docs/CPU_DATA_ANALYSIS.md. Nothing reads the item back into the pipeline - it is written and never parsed by rugnux itself - so no stored file is reinterpreted in a way that changes a result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
ac06b5c64f |
Merging: report the error model in XDS's convention
rugnux fits sigma^2 = a*sigma0^2 + (b*<I>)^2, so its `b` is a fraction of the intensity. XDS fits
sigma^2 = a*(sigma0^2 + b*I^2) and prints ISa = 1/sqrt(a*b). The two `a` are the same number, but the
two `b` are not - b_xds = b^2/a - so the pair rugnux printed could not be read against a CORRECT.LP,
which is the only reason anyone looks at it.
Convert at the report. The fit, the merge weights and both engines' variance expressions are
untouched, so this is a re-expression and not a change: on a rotation dataset the merged intensities
move strictly less between before and after than they do between two runs of the SAME binary (99.9%
identical, max |dI/I| 9.1e-4 against the run-to-run control's 7.5e-3), with the same reflection set.
The rotation path also printed the wrong ISa for the comparison it invites. What it calls ISa is the
strong-reflection asymptote, a tier XDS has no equivalent of and which can only ever be the more
optimistic of the two; XDS's ISa is the whole-range 1/sqrt(a*b), which in rugnux units is exactly
1/b. Print both, labelled. On a broadband rotation dataset that is 13.2 (whole range) and 15.6
(asymptote) against XDS's 21.18 - so the number previously compared was flattering rugnux by 2.4.
A third, unrelated `b` lives in the space-group search: fitted with the sigma^2 coefficient held at 1,
with gate constants calibrated in that convention, and a ratio bound does not survive the mapping
(1.90 would have to become 3.61) while the absolute floor has no correct value at all, there being no
`a`. It is now commented as such, since making the three consistent is the obvious wrong move.
Also corrects three comments and two doc passages that still described a merged-sigma systematic
floor deleted in
|
||
|
leonarski_fandClaude Opus 5
|
bf80935b80 |
Bragg integration: do not let the radial correction outlive its kernel table
The kernel table is sized and built only where the correction can ever run - explicitly on, or auto, which is the same condition the GPU allocates its radial buffers under. BackgroundRadial(true) on any other engine therefore asked the CPU to correct with a single CIRCULAR kernel for rings that may be elongated, while the GPU, having no buffers, did not correct at all: a wrong kernel on one engine and silence on the other, from the same call. Only the auto path calls it today, so it was unreachable, but the setter is public and the invariant it depends on is not local to it. Remember whether the table was built and refuse to raise the flag otherwise. Also treat a zero stencil cap as "uncapped" rather than "no growth". The engine always sets max_grow, so this changes nothing that runs; it makes a caller that forgets it fail loudly instead of silently disabling the feature. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
61d24db59f |
Bragg integration: elongate the background ring per reflection
The signal disk and the r2..r3 background ring were fixed pixel circles, identical for every reflection at every resolution. A reflection is not round: a finite bandwidth streaks it radially by bw_sigma*Rpx, so at high resolution the ring sits within 1.3-2.2 sigma of the reflection's own profile and measures its tails as background. --integration-stencil <k> makes the RING an ellipse, elongated along the beam->reflection direction by k times that streak, capped at 2*r3. The tangential half-widths stay r2 and r3, and the r1 signal disk stays a circle: r1 drives the all-or-nothing n_inner_valid == n_inner gate, so growing it rejects any reflection carrying one bad pixel along a long streak, and the flux a circular r1 loses is a function of resolution alone, which the per-shell scale absorbs. The geometry lives in one shared header compiled by both the host compiler and nvcc, so the seven pixel-classification sites - the CPU mask/main/clip loops and the GPU mark_mask/main/trim/clip kernels - cannot drift apart. Rather than evaluate an ellipse, each pixel's squared distance has its radial part scaled down, d2 - q*rad^2 against r2^2/r3^2 with q = 1 - (r/(r+grow))^2, so grow = 0 gives q = 0 and both tests collapse onto d2 exactly in floating point. The width is the bandwidth streak alone, not the profile's full radial variance, which also carries the sensor parallax and weak-spot capture terms. Deriving the growth from those was implemented first and measured on the rotation battery: at k=1 it took Thau_9's high-shell CC1/2 from 75.8 to 27.9 and Benas_3's from 14.1 to 6.0, against cytC_10 +1.2 and lyso_ref flat. On a monochromatic beam they are the only terms there are, and C_CAPTURE is 64% of them. Keeping only the streak also makes the option exactly inert without a bandwidth, rather than merely small. Default 0. Measured on broadband rotation data with the bandwidth set to its spectroscopic value, matched resolution limits: high-shell CC1/2 30.6 -> 46.4 at k=4, and better in EVERY shell in both CC1/2 and R_meas (top shell R_meas 194.7% -> 138.7%), with completeness, multiplicity and space group unchanged and 28 of 98833 unique reflections lost. Anomalous peak height over 18 sites +0.107 +- 0.039 sigma (p = 0.013). The full 38-crystal rotation battery is unchanged to every reported digit, base against k=3. Two consequences of an elongated ring are handled rather than inherited. The neighbour exclusion marks the inner ELLIPSE in each neighbour's own frame, or an elongated neighbour leaks its tails into this reflection's ring. And the radial-background curvature kernel becomes a small table indexed by the growth, because its azimuthal average makes one kernel serve every reflection only while their stencils are identical; the GPU's radial window, previously a fixed 32 bins, is now sized on the host from the widest ring on the detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
52ea727650 |
Reader: reconstruct the background variance a legacy file does not store
A _process.h5 written before background_variance existed was read with var_bkg = 0, on the reasoning that zero leaves the combine with the signal term alone, "which is what it had before". It does not. Before, the combine back-derived the non-signal variance from sigma itself, and on a weak reflection that is essentially the whole of sigma^2; zero deletes the dominant term and weights the reflection by roughly 1/I instead of 1/sigma^2. Recover it from the integrator's own identity, sigma^2 = I + var_bkg, when the dataset is absent. Measured by re-scaling a stills _process.h5 with the dataset deleted, against the same file with it intact: the automatic resolution cutoff was reading 1.66 A where the intact file reads 1.81, with 14731 unique reflections against 11508 - i.e. the zeroed file looked good enough to merge 0.15 A past its own limit. Reconstructed, it reads 1.78 A and 11926, within 0.03 A of the intact file. The residue is the profile-fit path, where var_bkg is not exactly sigma^2 - I and only the box-sum identity is exact; that is recoverable to a closer approximation only by storing it, which is what files written from now on do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
a7c5d7a89b |
CBOR: carry the reflection's background variance
var_bkg was added to Reflection and to the HDF5 writer but never to the CBOR reflection map, so it survived only where rugnux drives the writer in-process. Everything that reaches the writer over the wire - i.e. every acquisition the broker records - wrote /entry/reflections/*/background_variance as an array of zeros, presented as a measured quantity, and re-processing such a file fed the merge a non-signal variance of zero. Encode and decode it. The key is optional on both sides, so a stream from an older version still reads and one from this version still reads on an older client. The round-trip test only checked h, k, l, the predicted position and d - which is why a missing float was invisible. It now gives every field a distinct value and checks all of them, so the next field added to Reflection and forgotten here fails immediately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
4869dbd984 |
Space-group search: do not judge centering on a non-positive present mean
The centering test compares the absent class's mean intensity against half the present class's. With a present mean at or below zero - which happens on a merge dominated by noise - the bound is non-positive, and the comparison stops measuring whether the absences are weak and starts turning on the sign of the absent mean. Seen in an uncut merge: absent -0.16 against present -0.03, where a more negative absent class passes and one nearer zero fails, both by accident. Require a positive present mean before the mean-ratio branch can confirm a centering. The rate branch below it counts violations rather than averaging intensities, so it cannot change sign, and it already exists for exactly the weak-data case this leaves to it. No crystal in the battery changes, at matched limits or with the automatic cutoff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
01626a54e0 |
Space-group search: keep the lowest shell when no shell passes the cut
The P1 merge that feeds the space-group search is cut at the first 1/40 shell whose mean I/sigma falls below 1. If even the lowest-resolution shell fails, the cut was abandoned altogether and the search was handed the whole merge out to the detector corner. That inverts under its own feedback. The bound is absolute, while the merged I/sigma it tests is capped by the merge's own asymptote: once noise-dominated high-resolution reflections have inflated the error model's b, no shell can reach 1 - so the case where the cut is abandoned is exactly the case where the merge is worst. Measured on one crystal: b 0.239 -> 0.782, ISa 4.2 -> 1.3, no shell above the bound, 46853 reflections into the search become 3989103, the added operator's agreement falls 0.978 -> 0.429, and a C-centred monoclinic crystal merges as triclinic. Fall back to the lowest shell's own high-resolution edge instead, when that shell is populated enough to define one. It is what the healthy case does anyway - shell 0 is the lowest-resolution fortieth of reciprocal volume, 46852 reflections on the crystal above. Only reachable when rugnux chooses its own resolution limit: an explicit --scaling-high-resolution also constrains the merge this cut is derived from and lands it above the crossing, which is why the battery at matched limits cannot see any of this. Both battery arms are therefore unchanged - space group identical on all 38 crystals, every quality column identical, whether rugnux picks the limit or takes it from the reference. On a deliberately degraded integration the crystal above returns to its true symmetry (294655 unique reflections back to 102514, R_meas 41.4 -> 29.3%) and its error model recovers. Four crystals in the battery sit within a factor of two of the bound, and one sits 0.19% from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f766a342bb |
Scaling: take the detector modulation surface to a 24x24 grid
The per-detector-plane modulation surface was learned on a 16x16 grid. Fitting the same surface to rugnux's own symmetry mates and cross-validating on held-out frames shows the grid was the binding constraint, not the data: held-out R_meas improves monotonically to 24x24 and then stops - none 12.64%, 8x8 11.13%, 16x16 10.71%, 24x24 10.58%, 32x32 10.60%, 48x48 10.58%, 64x64 10.60%. At d > 4.4 A the same ladder reads 5.29 / 4.92 / 4.86 / 4.66 / 4.72 / 4.64 / 4.74%. Frame-parity, random 50/50 and 4-fold splits agree. The structure being fitted is ours, not a reference program's: the same surface fitted to the other program's observations of the SAME events moves it 7.10 -> 7.07%, against 13.46 -> 12.81% for ours, and its amplitude is 6.5% robust sd against 1.2%. Measured across seven crystals spanning multiplicity 3.7-9.4, two detector types and 75-100% completeness, 24 never clearly hurts and mildly helps six of them; 32 adds nothing beyond it. An earlier in-sample ladder suggested 48x48 was worth twice as much - that was in-sample, and it overstated the gain about threefold. Nothing else needs adjusting: the Tikhonov shrinkage already adapts to thinly-populated cells, and the cross-validation gate already refuses the surface outright where the finer grid is too fine for the data - on the weakest crystal tested its held-out gain falls 4.3% -> 3.1% -> 1.7% and the surface is skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
7523c67655 |
Space-group search: set the operator-H bound from the measured gap
max_operator_h_ratio was 1.25. Instrumented over the rotation battery, the statistic it bounds reads 0.85-1.57 on GENUINE promotions - and 2.48 on a genuine orthorhombic step in an arm left short of pairs - while the two real merohedral twins read 1.82 and 4.01. There is a wide empty gap between the two populations and 1.25 was not in it: it sat inside the genuine range. Four genuine promotions already exceeded it and survived only because the two-arm rule happened to offer cover from the other arm; a cubic case with no such cover was refused outright, by a margin of 0.4%, and merged in the orthorhombic subgroup with twice the unique reflections. That refusal is invisible to the standard battery, which passes an explicit resolution limit: the limit also constrains the merge the search's internal cutoff is derived from, and lands it just under the crossing. It appears only when the automatic cutoff runs. Set the bound to 1.70, in the gap. On the automatic-cutoff arm the cubic case returns to its true group (unique reflections 49277 -> 23330, CC1/2 in the outermost shell 25.7 -> 50.4) and no other crystal changes symmetry. The 38-crystal battery at matched limits is unchanged, space group included. All ten SearchSpaceGroup test cases pass, including the twin decision table and the H-margin case - worth checking explicitly, because a looser bound also confirms the operator agreement more often and so suppresses the systematic-absence veto more often. The header note already predicted this failure mode: the test's known limit is angular coverage, not data quality, and a refusal on a lopsided merge says more about the coverage than the symmetry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
bc1c4c6800 |
Rotation: land the rest of the bandwidth term
|
||
|
leonarski_f
|
0b5fb4fb92 | Merge branch 'fix56-work' into integration-variance-fixes | ||
|
leonarski_fandClaude Opus 5
|
1239c49731 |
Bragg integration: separate the three things a bandwidth used to switch
Setting a bandwidth flipped three unrelated switches at once: it changed the profile's radial capture term, it moved the width measurement from the signal disk to the whole fit grid, and it silently overrode the background clip and trim, so --background-clip under --bandwidth was ignored - the two runs were bit-identical. The width measurement was the damaging one. The fit grid is an azimuthally averaged stack, so its second moment is sigma_r^2 + sigma_t^2 and the radial smear of a bandwidth leaked into the tangential model - a tangential width of 3.04 px against a 1.06 px truth, inflating the effective background pixel count where the weak signal is. The result was a step rather than a slope: on genuinely monochromatic data, declaring a 0.2% bandwidth cost ISa 28.4 -> 22.2. Measure the two widths separately, accumulated in each spot's own radial/tangential frame over the signal disk, from the signed profile cells - away from the peak a learned cell is background noise centred on zero, so the signed sum is unbiased, while clamping it at zero turns that noise into a pedestal the r^2 weight reads as width. The radial term is then the measured excess or the analytic floor, whichever is larger. With the two widths separated there is nothing left for the broadband switch to select, so it is gone - which is the proof the three were independent. The background clip and trim now come from the settings in every case; the tuned 3-sigma broadband default moves to the rugnux front end, which is the only place that knows whether the user gave a value. Monochromatic data: declaring a 0.2% bandwidth now costs ISa 28.4 -> 27.9 rather than 22.2, and forcing the old 3-sigma clip in the new build reproduces the good result, so none of the step came from the clip. On large-bandwidth data CC1/2 improves in 8 of 10 shells. Across 12 monochromatic crystals the space groups are unchanged and CC1/2 moves by at most 0.2 points. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
3d3fb0e58b |
Bragg integration: stop rectifying the fitted intensity into its own variance
The profile fit weights each pixel by 1/v with v = max(bkg, floor) + max(0, I)*P, where I is the fit's own current estimate. Rectifying it means that at true zero the plug-in is E[max(0,I)] = 0.4*sigma rather than 0, and with sum(P^3)/sum(P^2)^2 = 4/3 for a Gaussian the reported sigma comes out about 0.2 counts too large - always, additively. That is nothing at sigma ~ 7 counts and 11% at sigma ~ 2, so it only shows on data measured against roughly one background count. Clamp the whole weight instead of the intensity: v = max(bkg + I*P, bkg/2). Simulation of the real integrator gives claimed/true sigma 0.92-1.01 at zero intensity across backgrounds 0.02-2.0 ct/px and 1.000-1.007 above I = 30, where the clamp never binds. Dropping the signal term entirely instead (v = max(bkg, floor)) is exact at zero and wrong everywhere else - 1.91 at I = 5, 4.29 at I = 30, 13.3 at I = 300 - and a test built on systematically absent reflections cannot see that, because it only measures zero. Removing the clamp altogether overshoots and biases the intensity, since a downward fluctuation shrinks v at the peak and over-weights it. The pixel variance floor was 1/12, documented as the rounding of a continuous energy. That does not describe a photon counter: measured on raw frames at 0.065-0.082 ct/px, var/mean is 1.042-1.045, i.e. Poisson with no digitisation term, and a digitisation term would be additive rather than a floor. What the floor really protects is the background estimate, which a small ring can read as exactly zero, so it belongs at the resolution of that estimate, ~1/n_bkg. At 1/12 it multiplied the reported variance by floor/bkg below 0.083 ct/px - a factor of two at 0.04. Set to 0.01. Measured on systematically absent reflections, whose true intensity is zero, as std(I)/rms(sigma) binned by background - not std(I/sigma), which is deflated by the correlation between the plug-in sigma and the reflection's own fluctuation. On 2.78 M absent observations at 0.16-3 ct/px the ratio goes 1.04-1.07 to 0.99-1.00. On 2.58 M at 0.005-0.6 ct/px, decomposed: the clamp carries it above 0.08 ct/px, the floor below it. Intensities move 0.4%; this changes sigma, not I. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_f
|
1283e04ada | Merge branch 'fix23-work' into integration-variance-fixes | ||
|
leonarski_fandClaude Opus 5
|
97dbbc50b4 |
Merging: fit the error model on the reflections the cutoff keeps
The (a, b) fit ran over the whole merged range and the automatic resolution cutoff was applied afterwards, so the sigma correction applied to the reflections that survive was calibrated largely on reflections that do not. Measured on one dataset: a = 0.286 fitted over 843k reflections, 22k written. A manual --scaling-high-resolution already restricts the population at ingest, so only the automatic path was affected. Fit over the full range, merge, read the cutoff from that merge, refit (a, b) on the samples the cutoff keeps, merge again. The circularity resolves by direction: the cutoff comes from CC1/2, a correlation of the two half-set means, which the sigma scale barely moves, so the cutoff can be read first and the sigmas calibrated on the population it chose. One refinement, not an iteration; one extra merge pass. Note this is invisible to the rotation battery, which passes an explicit high resolution limit matched to XDS and so never exercises the automatic cutoff. With a manual limit the fitted (a, b) are byte-identical to before. The equivalent defect in the stills / offline --scale path is untouched; it is a different engine and needs its own validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
72efb75a8c |
Merging: do not floor the merged sigma at the systematic term
The merged sigma was floored at b*|I|, so I/sigma could never exceed the reported ISa. On one dataset every merged reflection came out at I/sigma <= 12.96 with a 99th percentile of 12.77 in every resolution shell alike, while the scatter of the observations implied about 44 and XDS reported 58. The floor is wrong in principle. `b` is fitted from the scatter BETWEEN a reflection's symmetry equivalents, i.e. from the part that is not common to them, so it averages down with multiplicity exactly like the counting term. 1/sqrt(sum_w) with the b-inflated per-observation sigma already gives b*I/sqrt(n); flooring at b*|I| puts the sqrt(n) back. That is the whole effect: 12.96 * sqrt(21.6) = 60, against XDS's 58. It was introduced on a comparison of our MERGED I/sigma against XDS's UNMERGED I/sigma. XDS's own merged low-resolution I/sigma exceeds its reported ISa on 30 of the 39 reference datasets here, median ratio 1.78 and up to 4.23. Merged low-shell I/sigma now lands where XDS's does: 22.4 -> 46.2 against 46.2 on one crystal, 26.7 -> 115.7 against 96.6 on another, 12.5 -> 45.0 against 58.0 on a third. Over the 38-crystal battery the space groups, the merged reflection sets, R_meas and CC1/2 are all unchanged - every one of them is sigma-independent, which is what makes them the right control - and <I/sigma> rises on 35 crystals with none worse. The asymptotic estimator that fed the floor stays, for the reported ISa only, and is repaired in the process: it subtracts a*sigma^2 rather than the raw sigma^2 (at a < 1 the difference is the same size as the b^2 being measured, which is what made it flip between 10.9 and 62.7 on consecutive passes of the same data), it rescales each group's variance median-unbiased before subtracting an unbiased counting term, its I/sigma gate uses the same convention, and it is bounded by the whole-range b - an asymptote exists to refine 1/b upward, not to report 0.3 because "strong" was selected on a sigma scale the fit itself rejects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f4e281b2f5 |
Rotation: give prediction and partiality the energy bandwidth
The rotation predictor and RotationPartiality used the mosaicity alone. Energy bandwidth broadens a reflection's rocking curve as (dlambda/lambda)*tan(theta_B), resolution-dependent and negligible at low angle, so on a large-bandwidth beam the modelled reflecting range was too narrow exactly where the crystal still diffracts: 0.064 deg of broadening against a fitted 0.083, i.e. 26% at the detector edge. The stills predictor has carried the term since it was written; only rotation was missing it. Add it in the three places that have to agree. The predictor widens both its acceptance window and the partiality it hands to integration; the merge widens the partiality it recomputes from the smoothed mosaicity; and the per-image mosaicity fit subtracts the same term before fitting, so what it returns is the intrinsic mosaicity rather than the mosaicity plus the beam. Without that last part the bandwidth would be counted twice. The term goes in without the 1/zeta of the usual expression: the erf already divides by zeta, so adding a per-reflection width that itself carries 1/zeta would divide by it twice - up to 20x at the minimum zeta. dphi = delta*tan(theta_B), and the zeta stays where it was. The rotation identity dtheta/dphi = zeta was checked against a numerical solve of the diffraction condition at four resolutions and three orientations. Monochromatic data is untouched by construction - the term is guarded on a non-zero bandwidth and is an assignment, not arithmetic, when there is none. Verified: 246456 reflections byte-identical through the predictor, 4.7 million rocking-fraction evaluations with no bitwise difference, and identical merge tables end to end. The bandwidth is read from the file (incident_wavelength_spread) or from --bandwidth, and is absent from every dataset in the rotation battery. On the bandwidth dataset the fitted mosaicity becomes resolution-independent (0.0745 -> 0.0719 deg), the prediction window widens, frames per rocking event go 4.6 -> 5.3, per-image correlation to the merge rises 0.710 -> 0.725, and R_meas improves 0.1-0.7 pp in every shell while CC1/2 falls 0.6-0.9 pp in the outer two. Merged quality is net neutral: the combine normalises by sum(partiality), so a uniform widening largely cancels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
e352227a2d |
Scaling: divide out the incident flux before fitting the per-frame scale
The beam is not constant. On one beamline it oscillates +-9.8% with a ~5.9-frame period, confirmed four ways: the raw images, our own azimuthal-integration total, the per-frame mean background, and XDS's per-image SCALE, which correlates +0.999 with the first three. XDS removes it inside INTEGRATE, per image. The fitted per-frame G could not: --smooth-g defaults to 5 degrees, which is 25 frames at 0.2 deg/frame, so a 5.9-frame signal is smoothed away. Measured, the applied scale carried 0.70% rms against a 9.8% modulation and correlated 0.66 with XDS's SCALE. The only thing removing the oscillation was the refit on fulls, which acts after several partials spanning most of a period have already been summed, so it removes the mean and leaves the dispersion inside each event. Take the flux from the per-frame mean background, gauge it to the run median, and divide it out of rlp as the partials are ingested, so the fitted G sees only the residual and smooth-G smooths only the residual. The background mean tracks our own azimuthal background at r = +0.971 and XDS's SCALE at |r| = 0.93, with 95% of its detrended power in the 3-8 frame band. Slower background movers - ice, a drifting shadow, absorption against the goniometer angle, radiation damage - are still absorbed by G, which keeps its low frequencies through the smoothing. The applied scale now carries 9.41% rms at |r| = 0.93 against XDS. On the affected dataset R_meas 9.9 -> 9.5%, low-resolution R_meas 6.5 -> 6.0%, ISa 12.8 -> 13.8, and the anomalous peak height rises 0.423 +- 0.069 sigma over 18 sites (p < 0.001) - the only significant move in the arbiter. Over the 38-crystal battery the space groups and the merged reflection sets are unchanged and every metric has median delta zero. A monochromatic dataset carries the same modulation at 2.0% rms, confirmed by the same three proxies; the correction engages there too but no merged statistic moves at that amplitude. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
f60768d49c |
Bragg integration: drop the 2% sigma floor and carry the background variance
Two changes to the same variance chain; they are in one commit because the second exists to remove an assumption the first was breaking, and separating them leaves a tree that is correct only by luck. The reported sigma was floored at 2% of the intensity, a per-partial I/sigma cap of 50. It applied only to the box-sum seed, never to the profile fit, so the shipped default was unaffected - but the combine back-derives each partial's non-signal variance as sigma^2 - I, and a floored sigma makes that quantity mean nothing. It then read corr^2 * (0.0004 I^2 - I), which is not a background variance. Measured on --integrator boxsum: the reported sigma understated the true scatter by up to 16x at I ~ 21000 counts per partial, and pooled_I amplified a 1 ct/px background drift into an 11.5% intensity error on the strongest reflections. What the floor stood in for - that at high intensity the error is systematic rather than counting - is already carried downstream, twice: the fitted b in v = a*sigma^2 + (b*I)^2, measured from the data rather than assumed, and SigmaWithSystematicFloor on the merged sigma. The floor was that idea applied one level too early with a hardcoded b of 0.02. It arrived without a test or a setter and was unreachable from the CLI, the API and the config. The merge now takes the non-signal variance the integrator actually measured instead of inverting sigma^2 = I + N. That identity is exact for a box sum once the floor is gone and was never exact for a profile fit, whose sigma^2 = 1/den + (wsum/den)^2 * bkg_var is formed against a fitted intensity. The value is carried through BraggFitResult, Reflection and Obs, both engines, both merges, and the process-file round trip; files written before this change are read with the term absent, which is what they had. Battery, 37 crystals, paired: space groups unchanged, reflection sets unchanged, median delta zero on R_meas and CC1/2. --integrator boxsum on the reference crystal goes ISa 8.9 -> 20.2 with a 0.947 -> 1.032. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
d565b66916 |
Post-refine: drop the rocking-width diagnostic, report frames per event
The "median rocking width -> estimated mosaicity" line took an intensity-weighted second moment of the frame-centre angles with max(0, I) weights, per event, then a median over events. For a two-frame event that moment is exactly zero whenever only one frame has I > 0 - probability 2/3 for a reflection carrying no signal - so on a noise-dominated dataset the median lands in the degenerate spike and prints 0.0000. Simulated against a known width it is wrong by 0.23x to 13x, in both directions, and on a pure-noise null it returns a plausible-looking 0.06 deg. est_mosaicity_deg was read nowhere, so nothing downstream was affected; the number only misled whoever read the log. It was built to measure a signal for a mosaicity refinement that was then abandoned, and the estimator that replaced it is the per-image one that already drives prediction. Report instead the frames per rocking event, which is what the block could honestly say: near 2.0 the reflections barely rock, so the observed angle this refinement is fitted to is under-determined. It is a geometry count, so noise cannot inflate it. Also drop phi_rms_deg, which is never assigned anywhere, and Partial::zeta, which is only written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
4d3434e2a5 |
Beam stop: compare each pixel only against its own ring
The background belongs to the beam and the shadow to the stop, and the two are not concentric - fitting the stop edge per azimuth gives offsets of 13.4 px on an 85.8 px disk, 22.2 px on 67.3 px and 6.9 px on 23.7 px, 8 to 33 per cent of the stop radius on every crystal measured. The finder bridged that gap with a radial envelope, the largest ring background over an outward window, used as the reference for an individual pixel. That quantity exceeds the local background wherever the background rises outward, so sound pixels near the stop scored below the penumbra threshold and were masked. Measured against the fitted edge on a long-distance disk stop, the mask was displaced rather than mis-sized: short by up to 20 px on one side, over-reaching by up to 45 px on the other, with eight of twenty-four azimuth sectors falling short. The ring median is already the right reference wherever a ring still has unshadowed pixels to measure, which is every ring except those lying wholly inside the disk - and it needs no assumption about where the stop sits. So the envelope is gone from the per-pixel test, and the rings it existed to cover are handled directly: walking outward, a ring whose background is a fraction of the background further out is shadow in its entirety. That comparison is only ever asked whether a whole ring is inside the stop, never to judge a pixel, which is where its failure mode lives. Blockage is deliberately not a counting test - on a bright dataset the shadow interior is still well counted. Detection is now one channel instead of two, and 113 lines shorter. Measured: no azimuth sector falls short by more than 3.4 px, over-reach drops on all three fitted crystals, and mask area moves by at most 0.04 per cent of the detector on six crystals, so this corrects the shape rather than resizing. Battery: space-group agreement with XDS unchanged at 34/37, median change in R_meas and in the lowest shell 0.000 pp. The crystal that suffered worst when masking was introduced recovers to its unmasked quality - R_meas 25.1 -> 17.2 per cent, ISa 4.45 -> 10.04 - which is what removing the over-masking should do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
a29c36600f |
Beam-stop shadow detection, and a low-resolution limit for scaling
rugnux finds the beam stop and its holder in a projection of 60 images and marks them in the pixel mask as bit 9 (--detect-beam-stop[=N|off], on by default). Reflections behind the stop are attenuated but not flagged, so they integrate low with a plausible sigma and nothing downstream catches them: the signal-box gate requires 100% valid pixels and shadow pixels are valid, the background clip is high-side only, and the |zeta| cut applies only to the space-group search merge. The detection compares each pixel's background against the typical background at the same radius on two channels. An azimuthal one (the ring median) finds the holder arm, which is a minority of its ring; a radial one (the background just outside) finds the disk, which the ring median cannot see because inside a fully blocked ring the median is the shadow itself. Pixels are pooled over a 5x5 box and tested only where the background has actually been counted, so low-background data no longer masks the whole detector. Recorded reflections are carved back out - a beam stop cannot block a reflection that was measured. Bit 9 belongs to the run that found it, not to the dataset: it is cleared when a run starts, so a mask read back from a file that carries one starts clear. The user mask (bit 8) is left alone. Scaling and merging gain a low-resolution limit, default 50 A (--scaling-low-resolution <num>, 0 removes it), applied per observation before scaling so it also protects the per-frame scale fit and the space-group search. 50 A is the value XDS configurations use; rugnux_vs_xds.py now matches both of XDS's resolution limits instead of only the high one, so the lowest shell is the same shell in the two programs. The viewer draws the detected shadow in coral with a "Show beam stop" switch in the side panel, exposes the low-resolution limit in the settings dock, and offers detection in its processing jobs. Adding an image marker meant giving the reader a MIN_REAL_PXL_VALUE, because several places classify a pixel by range rather than by equality and would otherwise read the new marker as a very negative intensity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
c673521b76 |
Space-group search: do not veto on systematic-b where the H test confirms
The systematic-b veto compares a candidate merge's fitted b against its parent's, and both move with data quality. Removing genuinely bad observations improved both merges but the subgroup more than the supergroup (parent 0.1644 -> 0.1480, candidate 0.3187 -> 0.3056), so the ratio crossed its 2.00 bound at 2.065 and a correct cubic promotion was refused - while the H statistic, which has no sigma in it, did not move at all (0.898 either way). Better data demoting a crystal is the wrong behaviour. The veto now fires only where the H test has not confirmed the promotion. H is the statistic that was measured to separate a real symmetry operator from a twin law; b's genuine and twin ranges are interleaved. A twin fails both. No bound moved and no option was added. Battery: 34/37 point-group agreement with XDS before and after with no crystal changing; with the beam-stop mask 33/37 -> 34/37, the single change being a cubic crystal recovering its true I23. Both real merohedral twins stay refused on H in every arm. Gating the guards on the L-test / second moment was tried and rejected: those indicators do not flag a real twin on the P1 pre-promotion merge, only after merging in its true symmetry, so the gate promoted a twin into its holohedry. Left alone deliberately: merge_systematic_b divides its reduced chi^2 by the observation count rather than by the degrees of freedom, which inflates the ratio more for small-orbit parents. Fixing it requires re-deriving all three b bounds, which were calibrated on the biased statistic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
df9a9c2a2c |
Fix the defects found reviewing the branch before merge
Build Packages / build:viewer-tgz:cpu (push) Successful in 19m32s
Build Packages / build:windows:nocuda (push) Successful in 19m57s
Build Packages / build:viewer-tgz:cuda (push) Successful in 22m45s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 22m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 23m24s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 28m8s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 28m9s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m18s
Build Packages / XDS test (durin plugin) (push) Successful in 11m10s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m21s
Build Packages / build:windows:cuda (push) Successful in 22m5s
Build Packages / build:rpm (rocky9) (push) Successful in 20m57s
Build Packages / Generate python client (push) Successful in 34s
Build Packages / Build documentation (push) Successful in 1m29s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m41s
Build Packages / DIALS test (push) Successful in 21m19s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 21m34s
Build Packages / build:rpm (rocky8) (push) Successful in 27m4s
Build Packages / XDS test (neggia plugin) (push) Successful in 10m19s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m58s
Build Packages / Unit tests (push) Successful in 1h17m36s
Image buffer: the per-image CBOR metadata headroom had been re-derived from the online reflection cap alone, which cut it from 4 MiB to 2.55 MB while the measured worst case - reflections plus the capped spot list plus the three azimuthal arrays - is 2.9 MB, so the receiver dropped the frames with the most to say. Restore it and give it a name that both the code and its guard test read: written down twice, the two had drifted and the test kept passing against the value the code had left. Spot finding: an unset low_resolution_limit means no limit at that end, as an unset high_resolution_limit already did. An optional rather than a zero sentinel, because zero is not a natural "no limit" here - every pixel lies above it, so the plain comparison masked the whole image instead of none of it, and nothing validated the zero. The API field is no longer required; a zero is folded into the unset case at the boundary, where older clients still send it, so one spelling reaches the analysis code. The FPGA takes its fixed-point ceiling instead, since ap_ufixed<16,9> wraps above 512 A and would have masked everything. image_preprocessing: check the CUDA calls on the fused decode path - the one new GPU file with none, and the path fed by bytes we did not produce. An unchecked synchronise returned the host-written sentinel as if it were a measurement, so the decode looked successful and the fallback to the host decoder never fired. rugnux: --stride no longer writes one past the end of the per-image arrays, whose count floored where the worker loop ceils, and the written process file links the images actually processed rather than the first N - each frame's picture now sits next to its own analysis. Powder calibration: the face-centred calibrants no longer list their systematically absent rings, so the distance fit starts from a reflection that exists rather than an extinct one; the triclinic calibrant covers both signs of h and k instead of a single octant, which is only valid for a diagonal metric. The test asserted the old behaviour - one ring formula for every cubic standard - and is rewritten. CBOR: skip an unknown tagged value in the end block, as the other four blocks already do. One advance lands on the tagged item rather than past it, so an older reader fed a newer end message threw and never finalized its file. Viewer: a settings value the setter rejects no longer escapes as an uncaught throw from a worker slot, and the field offers only what the setter accepts. Space-group search: judge stage B on the same "present" cut stage A already computes. Merged sigma is floored so no reflection reads above ISa, so on a low-ISa merge the fixed cut left both stage B tests unsatisfiable - every screw axis passed unchallenged and the centering rescue switched itself off on exactly the weak data it exists for. Where the fixed cut is the smaller of the two they are equal and this is inert: over the 37-crystal rotation battery every crystal reports the identical space group and identical merge statistics, so it is a no-op there and the low-ISa case it targets remains unmeasured. rugnux: --polarization reaches --mode azint, which parsed the flag and then dropped it; that mode also applies the same polarization default as every other mode. Acknowledge the ACTS/traccc project, whose sparse connected-component labelling both spot extractors take their algorithm from, with its citation and its license. The rc.161 change list is brought back to one line per entry, and the user-visible changes that were missing from it added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
3ccb97e31b |
Adaptive spot finder: pin the per-ring host buffers
The GPU engine copies six small per-ring arrays back to the host every frame - the clipped raw sum/sum2/count that the threshold is computed from, and the plain corrected sum/sum2/count that become the azimuthal profile. They were plain std::vectors, so the copies landed in pageable memory, and a device-to-host copy into pageable memory blocks the calling thread until it has completed whatever stream it was issued on. The profile snapshot sits between the plain pass and the two sigma-clip passes, so Detect() stopped there and the device then sat idle while the host caught up and enqueued the rest. Register them, as AzIntEngineGPU already does with its own, and the copies are genuinely asynchronous. Measured on a 4.5 Mpixel frame: 0.647 -> 0.621 ms per frame. Nothing else changes - the spot list and the profile are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
leonarski_fandClaude Opus 5
|
5830f78d57 |
Revert the azimuthal-integration sigma clip
Removes azim_int_settings.sigma_clip / rugnux --azim-sigma-clip and the clipping
machinery in AzIntEngine. This is a partial revert of
|