diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 27e47e7f..4afbc131 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -95,6 +95,12 @@ structure-factor machinery, and MTZ / XDS_ASCII I/O. Vendored in `gemmi_gph/`, s licence obligation. M. Wojdyr, "GEMMI: A library for structural biology" (2022), J. Open Source Softw. 7, 4200 [doi:10.21105/joss.04200](https://doi.org/10.21105/joss.04200). +**Hexagonal-ice ring positions** — the eleven measured ring $d$ spacings the ice-ring score, the +ice-ring flagging and the ice calibrant are all built on are taken from the measurements of, not +enumerated from a cell. D. W. Moreau, H. Atakisi and R. E. Thorne, "Ice in biomolecular +cryocrystallography" (2021), Acta Cryst. D77, 540-554 +[doi:10.1107/S2059798321001170](https://doi.org/10.1107/S2059798321001170). + **Data-quality statistics** follow the established conventions rather than any one program: R_meas and R_pim, CC1/2 and CC\*, and the reporting of I/sigma(I). K. Diederichs and P. A. Karplus, "Improved R-factors for diffraction data analysis in macromolecular crystallography" (1997), Nat. Struct. Biol. diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 4bb910a8..7f74ada2 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -36,6 +36,7 @@ The methods draw on, and in places reimplement, solutions from: - G. Winter et al., "DIALS: implementation and evaluation of a new integration package", *Acta Cryst.* **D74** (2018), 85-97, and J. Beilsten-Edmands et al., *Acta Cryst.* **D76** (2020), 385-399 (CC1/2 resolution cutoff, merge outlier rejection, scaling error model). - P. Evans, "Scaling and assessment of data quality", *Acta Cryst.* **D62** (2006), 72-82, and P. R. Evans, *Acta Cryst.* **D67** (2011), 282-292 (POINTLESS: operator-by-operator point-group scoring, and the axial-zone screw-absence test). - A. G. W. Leslie & H. R. Powell, "Processing diffraction data with MOSFLM" (2007), NATO Science Series II **245**, 41-51 (post-refinement practice: what is refined per image and what over a wedge). +- D. W. Moreau, H. Atakisi & R. E. Thorne, "Ice in biomolecular cryocrystallography", *Acta Cryst.* **D77** (2021), 540-554 (measured hexagonal-ice ring positions, used by the ice-ring score, the ice flagging and the ice calibrant). - K. Diederichs & P. A. Karplus, *Nat. Struct. Biol.* **4** (1997), 269-275, and P. A. Karplus & K. Diederichs, *Science* **336** (2012), 1030-1033 (R_meas / R_pim, CC1/2 and CC\*). - IUCr Commission on Crystallographic Nomenclature, "Statistical descriptors in crystallography", *Acta Cryst.* **A45** (1989), 63-75, and *Acta Cryst.* **A51** (1995), 565-569 (uncertainty conventions). @@ -194,6 +195,33 @@ Where the sweep is shorter than half a turn the spot symmetry cannot be formed, taken instead from the centroid of the radial background profile, which needs only a few images. Where neither method can measure the centre, the value from the file is kept. +### 1.5 Finding the beam stop + +The beam stop and its holder arm shadow part of the detector. A reflection behind them is attenuated +but otherwise ordinary — it integrates low, with a plausible $\sigma$, and no outlier test catches +it — so the shadow is found and masked instead. `--detect-beam-stop[=N|off]` (on by default, $N=60$ +frames; it reads its own frames, so it is not affected by, and does not affect, the beam-centre +pre-scan of §1.4) projects those frames to a per-pixel mean and maximum, and writes the result into +the pixel mask as bit 9, from where it excludes those pixels from every later stage. + +The shadow is a place where the background is *missing*, so it is found by comparing each pixel's +background against the background at the same radius. The mean is pooled over a small box first — +a single pixel of a sparse background carries too few counts to tell a shadow from a Poisson hole, +and the stop is much wider than the box — and the comparison is against the median of the pixel's own +radius ring, taken over the pixels not already excluded and iterated a few times so the shadow stays +out of the baseline it is measured against. A pixel is shadow when the ratio falls below 0.35, and +only where the ring has accumulated enough counts for the dip to mean anything. Nothing assumes the +stop and the beam are concentric, because only the per-ring comparison is used; a ring lying wholly +inside the stop has no unshadowed pixel for its median to find, which is exactly the case where the +comparison must fail, and such a ring is shadow in its entirety. + +What survives is then shaped into a region: the low pixels connected to the beam centre, bridged +across the module gaps the holder arm crosses, grown outward through the partially shadowed penumbra, +closed, and with the interior of the disk filled. Last, any pixel that ever recorded a real +reflection — a maximum over the frames well above background, in a cluster, so that a single-frame +zinger does not count — is given back, because a beam stop cannot have blocked a reflection that was +measured. + --- ## 2. Azimuthal integration (radial profiles) @@ -318,10 +346,6 @@ Both channels are used offline as a **gate** on ice handling: unless the run rea A further optional safeguard removes isolated high-resolution “spur” spots by detecting large gaps in $1/d$ (or $q$) space and discarding spots beyond the gap. This is intended for macromolecular diffraction where edge-of-detector backgrounds can be extremely low. -A per-image **resolution estimate** is read off the same spot list, and it predicts the resolution the *merged* data will reach rather than the highest-resolution spot on this image. Each non-ice spot is weighted by $\sqrt{I}$ — the intensity is a summed photon count, so $\sqrt{I}$ is its Poisson significance — and the quantile is taken beyond which a fraction $f=0.30$ of that weight lies; the estimate is that resolution taken $2.25\times$ further in $1/d$, clamped so it can never beat the corner of the detector. The dataset value is the median over images. - -Two things make this work where reading off the highest-resolution spot does not. A quantile near the middle of the distribution measures the *shape* of the fall-off, which is the crystal's own decay $\exp(-B/2d^{2})$; the extreme end measures where detection stops, which moves with the exposure and with how many reflections the unit cell puts on a frame, and is a poor guide to the crystal. And merging averages many observations of each reflection, so intensities go on being measurable a fixed factor past the point where one image's spot finder still detects them — that factor is what the $2.25$ carries. Both constants are calibrated on rotation data against the resolution at which per-shell CC1/2 falls through 0.30. The estimate is a prediction, good to about 0.2 Å; it is not a measurement of what a run achieved, and nothing downstream is cut on it. - ### 3.4 Connected-component labeling (CCL) Strong pixels are grouped into connected components (adjacent strong pixels) using a CCL algorithm. Each component yields a candidate spot with: @@ -355,6 +379,12 @@ $$ \frac{n_\mathrm{indexed}^2}{n_\mathrm{total}} \quad\text{(indexed-spot count is kept; the frame is then integrated once at that min-pix. The fraction factor discounts the extra spots a smaller min-pix admits *unless the lattice actually explains them*, so strong frames keep their real weak spots (extending resolution) while noise-flooded frames stay strict. Because min-pix filters the connected components *after* detection, strong-pixel detection AND the connected-component labelling both run **once** per frame, and the three attempts only repeat the spot-level filter; the azimuthal profile is the one that single detection pass computed. The winning attempt's spot list is kept rather than re-extracted, so the frame that is integrated is exactly the frame that was scored. This is a **stills-only, indexing-path** option — rotation indexing builds one global lattice from all frames and keeps a fixed min-pix. In `rugnux` it is the default; giving an explicit `--min-pix-per-spot` pins a fixed value instead. +### 3.6 Predicting the resolution the merged data will reach + +A per-image **resolution estimate** is read off the finished spot list. It predicts how far the *merged* data will reach, not how far the furthest spot on this image lies. Each non-ice spot is weighted by $\sqrt{I}$ — the intensity is a summed photon count, so $\sqrt{I}$ is its Poisson significance — the $1/d^2$ is found beyond which a fraction $f=0.30$ of that weight lies, and the estimate is that resolution taken $2.25\times$ further in $1/d$, clamped so it can never beat the corner of the detector. The dataset value is the median over images. + +Both constants carry a mechanism. A quantile from the middle of the distribution measures the *shape* of the fall-off, which is the crystal's own $\exp(-B/2d^{2})$, where the extreme end of it measures where detection stops — a threshold that moves with the exposure and with how many reflections the unit cell puts on a frame. And merging averages many observations of each reflection, so intensities go on being measurable a fixed factor in $1/d$ past the point at which one image's spot finder still detects them; that factor is the $2.25$. Both are calibrated on rotation data against the resolution at which per-shell CC1/2 falls through 0.30, and the estimate is good to about 0.2 Å there. It is a prediction and not a measurement of what a run achieved: nothing downstream is cut on it, and it is reported alone (rugnux `SPOT_RESOLUTION_ESTIMATE`, and per image in the stream, the plots and HDF5). + --- ## 4. Indexing overview diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index c7aac77a..3126b11c 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -410,7 +410,7 @@ General: | Option | Description | | --- | --- | | `-o, --output-prefix ` | Output file prefix (default: `output`) | -| `-N, --threads ` | Number of worker threads (default: all hardware threads, with the per-image loop capped at 8 workers per GPU; an explicit value is used as given) | +| `-N, --threads ` | Number of worker threads (default: all hardware threads, with the per-image loop of `--mode mx` capped at 16 workers per GPU; an explicit value is used as given) | | `-s, --start-image ` | First image to process (default: 0) | | `-e, --end-image ` | Last image to process (default: all) | | `-t, --stride ` | Process every *n*-th image (default: 1) |