diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 04e8c1046..6ae90cf23 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -27,7 +27,8 @@ across them and are the ones the rest of the documentation cites. - [From images to spots (§0–§3)](CPU_DATA_ANALYSIS_IMAGE.md) — device-side decoding, geometry and reciprocal-space mapping, azimuthal integration, spot finding. - [Indexing and geometry refinement (§4–§7)](CPU_DATA_ANALYSIS_INDEXING.md) — FFT and fast-feedback - indexing, the lattice search, geometry refinement, post-refinement and powder calibration. + indexing, the lattice search, geometry refinement, post-refinement, powder calibration, and the stills + geometry with a given cell and detector modules. - [Prediction, integration, scaling and merging (§8–§12)](CPU_DATA_ANALYSIS_INTEGRATION.md) — reflection prediction, profile-fitted integration, scaling, merging, mosaicity and the auxiliary statistics. @@ -65,18 +66,18 @@ The methods draw on, and in places reimplement, solutions from: - H. F. Walker & P. Ni, "Anderson acceleration for fixed-point iterations", *SIAM J. Numer. Anal.* **49** (2011), 1715-1735, after D. G. Anderson, "Iterative procedures for nonlinear integral equations", *J. ACM* **12** (1965), 547-560 (the acceleration of the per-frame scaling loops, §10.6). - R. A. Fisher, "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population", *Biometrika* **10** (1915), 507-521 (the z-transformation on which the correction surfaces' held-out half-set CC1/2 is compared). - R. C. Agarwal, "A new least-squares refinement technique based on the fast Fourier transform algorithm", *Acta Cryst.* **A34** (1978), 791-809 (the gradient of the refinement target, §15). -- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15). -- V. Y. Lunin & T. P. Skovoroda, *Acta Cryst.* **A51** (1995), 880-887 (the likelihood parameters from the free set, §15). +- N. S. Pannu & R. J. Read, "Improved structure refinement through maximum likelihood", *Acta Cryst.* **A52** (1996), 659-668, and G. N. Murshudov, A. A. Vagin & E. J. Dodson, "Refinement of macromolecular structures by the maximum-likelihood method", *Acta Cryst.* **D53** (1997), 240-255 (the maximum-likelihood amplitude target and the B restraints, §15). +- V. Y. Lunin & T. P. Skovoroda, "R-free likelihood-based estimates of errors for phases calculated from atomic models", *Acta Cryst.* **A51** (1995), 880-887 (the likelihood parameters from the free set, §15). - A. A. Vagin et al., "REFMAC5 dictionary: organization of prior chemical knowledge and guidelines for its use", *Acta Cryst.* **D60** (2004), 2184-2195 (the restraint dictionary, §15). - P. V. Afonine et al., "Towards automated crystallographic structure refinement with phenix.refine", *Acta Cryst.* **D68** (2012), 352-367 (the weight chosen among refinements by R-free, §15). - D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15). - J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6). - G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5). - A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). -- P. J. Huber, "Robust estimation of a location parameter", *Ann. Math. Statist.* **35** (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, *IEEE Trans. Image Process.* **6** (1997), 298-311 (the robust, pseudo-Huber scale fit of the refinement's score, §15.3). +- P. J. Huber, "Robust estimation of a location parameter", *Ann. Math. Statist.* **35** (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, "Deterministic edge-preserving regularization in computed imaging", *IEEE Trans. Image Process.* **6** (1997), 298-311 (the robust, pseudo-Huber scale fit of the refinement's score, §15.3). - J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction", *J. Mol. Biol.* **243** (1994), 100-115 (the flat bulk-solvent mask, §14.2, §15.3). -- A. Blondel & M. Karplus, *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5). -- M. M. Harding, *Acta Cryst.* **D57** (2001), 401-411 and **D62** (2006), 678-682, and F. Leonarski, L. D'Ascenzo & P. Auffinger, *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5). +- A. Blondel & M. Karplus, "New formulation for derivatives of torsion angles and improper torsion angles in molecular mechanics: elimination of singularities", *J. Comput. Chem.* **17** (1996), 1132-1141 (the torsion derivatives, §15.5). +- M. M. Harding, "Geometry of metal-ligand interactions in proteins", *Acta Cryst.* **D57** (2001), 401-411, and "Small revisions to predicted distances around metal sites in proteins", *Acta Cryst.* **D62** (2006), 678-682, and F. Leonarski, L. D'Ascenzo & P. Auffinger, "Mg2+ ions: do they bind to nucleobase nitrogens?", *Nucleic Acids Res.* **45** (2017), 987-1004 (the metal-site distances, §15.5). - P. Emsley, B. Lohkamp, W. G. Scott & K. Cowtan, "Features and development of Coot", *Acta Cryst.* **D66** (2010), 486-501 (unmodelled blobs, §15.9). - M. Wojdyr, "GEMMI: A library for structural biology", *J. Open Source Softw.* **7** (2022), 4200 (model / structure-factor / map machinery used in §14; the topology, riding hydrogens and blob search of §15). - J. P. Wright, "Experiences with GPU decompression for bitshuffle + LZ4 data", HDF5 User Group meeting (2021), and [github.com/jonwright/bslz4decoders](https://github.com/jonwright/bslz4decoders) (device-side decoding of bitshuffle+LZ4 images, §0). diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index e0c1a399e..7943fe4b2 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -117,14 +117,14 @@ Everything lands in `_report.txt` section 4 (`ANISOTROPY_*` keys), in th - **Bragg integration is profile-fitted by default** (per-shell Gaussian profile, Kabsch extraction; §9.3), with plain box summation available as a fallback (`--integrator boxsum`). The profiles are built per frame from that frame's strong spots, which suits fast-feedback and serial/streaming use; a profile shared across many frames (as in full offline workflows) is not currently formed. - **Space-group symmetry** beyond centering absences is not enforced during prediction/integration unless the space group is supplied and used downstream. - **Resolution masking** is controllable, and so is every stage of ice-ring handling (§3.3, §10.10). None of it runs unless the crystal is measured to have ice, because the fixed bands are a fixed cost in unique reflections whether it does or not. -- **Rotation vs still modes** differ substantially in prediction and scaling: partiality is angle-driven in rotation data, while stills are predicted within an excitation-error window and get their partiality from the default-on per-crystal tilt post-refinement (§10.2) — or unit partiality with `--simple-stills`. +- **Rotation vs still modes** differ substantially in prediction and scaling: partiality is angle-driven in rotation data, while stills are predicted within an excitation-error window that follows the run's rocking curve (§8.2) and get their partiality from the default-on per-crystal tilt post-refinement (§10.2) — or unit partiality with `--simple-stills`. - **Amplitudes and intensities.** The merged output carries both intensities (mmCIF `intensity_meas`, MTZ `IMEAN`/`SIGIMEAN`) and French–Wilson amplitudes (mmCIF `F_meas_au`, MTZ `F`/`SIGF`; §10.8), so a downstream program can refine against either. --- ## 14. Model-based validation: R-free against a model and electron-density maps -Offline (`rugnux --model model.pdb`) the merged data can be scored against a supplied atomic model and electron-density maps computed — enough to confirm that a model fits the data and to inspect the density, not a substitute for refinement. **The structure itself is not refined here** - a placed model is refined afterwards, on the side and deciding nothing (§15, [Refinement of a placed model](CPU_DATA_ANALYSIS_REFINEMENT.md)); the model is re-fractionalized into the data unit cell and then **placed as one rigid body** (§14.8) - or, where the rigid body does not place it and for a predicted model, by molecular replacement (§14.10) - and the observed amplitudes are the French–Wilson $|F|$ from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI. +Offline (`rugnux --model model.pdb`) the merged data can be scored against a supplied atomic model and electron-density maps computed — enough to confirm that a model fits the data and to inspect the density. **The numbers of this section are those of the model as placed, not refined** - a placed model is refined afterwards by default (`--no-refine` skips it), on the side and deciding nothing (§15, [Refinement of a placed model](CPU_DATA_ANALYSIS_REFINEMENT.md)); the model is re-fractionalized into the data unit cell and then **placed as one rigid body** (§14.8) - or, where the rigid body does not place it and for a predicted model, by molecular replacement (§14.10) - and the observed amplitudes are the French–Wilson $|F|$ from §10.8, so the R-free and the maps use exactly the same amplitudes as the written reflection file. The model, structure-factor, bulk-solvent and FFT machinery is provided by GEMMI. ### 14.1 Model structure factors diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index e9d9e99d2..a670a355c 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -405,6 +405,19 @@ reflection — a maximum over the frames well above background, in a cluster, so zinger does not count — is given back, because a beam stop cannot have blocked a reflection that was measured. +Pixels the finder reports that hold the detector's no-data mark (a PILATUS module gap stored as $-1$) +are the darkest pixels there are, but they are not shadow: they are masked as dead (bit 10) instead, so +that the reported shadow share is the shadow's and the test below does not drop every reflection a module +gap clips. A mean of zero is not taken as dead, since a deep shadow on a still counts nothing. + +**A reflection whose signal disk reaches into the shadow is not integrated.** The profile fit keeps a +disk that lost pixels to the mask and fits the pixels that remain (§9.3), which is right for a gap or a +dead pixel but not for the shadow: the mask is drawn where the background is dimmed, and the Bragg +half-shadow does not end there, so the rest of such a disk is attenuated too - measured against the +symmetry mates, such observations read low by 12-60 %. The test is made where each image is integrated, +so rotation, stills, the `_process.h5` and `--mode scale` see the same reflections; with no shadow in the +mask (the broker) it does nothing. + ### 1.6 Defective pixels A pixel that reads high frame after frame, whatever the crystal does, is integrated into whichever diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index bf2836230..614c49d2d 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -381,7 +381,7 @@ The space group is determined **after** pass 2, on the geometry the run refined, Only pass 2 is written, as the canonical `_*` output. Pass 1's merge exists to give the guard something to judge pass 2 against, so it stops short of the parts of the merge that only fill in a file — the correction surfaces, the twinning and radiation-damage analyses, the R-free flags and the amplitudes — and writes no merged files of its own. -**Goniometer rotation scale.** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. The excitation residual already measures it without a new degree of freedom: it rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor $k$ by which the stage actually turned, and normalising the axis throws it away. Pass 1 fits $k$ as a single parameter on its rocking events, with the crystal and the axis direction held at their committed values and the angle measured from the centre of the sweep. That fit **under-reads** a real error: it only sees the frames the stored angles still track, and a rate error is exactly what stops them tracking the rest. So it is not acted on directly. Between the passes, at the detector geometry pass 2 runs at, the lattice is indexed under the stored angles and under the fitted $k$, and each is scored on the validation frames spread over the whole sweep, as the share of their spots it puts on the lattice beyond what it puts there at a wrong spindle angle. The fitted $k$ is adopted only where it scores higher by more than the binomial noise of the two scores (z = 3.29); the run then integrates and post-refines at it, fits $k$ again on top of it, and repeats until the next $k$ no longer scores better - the fixed point of the fit. Otherwise the stored angles stand. The adopted $k$ drives every later pass (prediction, integration, scaling and the reported oscillation) and is reported as `GONIOMETER_ROTATION_SCALE`, with `GONIOMETER_ROTATION_SCALE_SUSPECT= TRUE`. `--rotation-scale ` asserts a calibration and skips all of this. +**Goniometer rotation scale.** A stage that turns further than it was commanded to leaves no trace in the file, because the stored $\omega$ values *are* the commanded ones; the excess then presents as the crystal drifting, in this program and in others. The excitation residual already measures it without a new degree of freedom: it rotates by $-\phi\,\mathbf{u}$ with $\mathbf{u}$ an **unnormalised** 3-vector, so $|\mathbf{u}|$ is the factor $k$ by which the stage actually turned, and normalising the axis throws it away. Pass 1 fits $k$ as a single parameter on its rocking events, with the crystal and the axis direction held at their committed values and the angle measured from the centre of the sweep. That fit **under-reads** a real error: it only sees the frames the stored angles still track, and a rate error is exactly what stops them tracking the rest. So it is not acted on directly. Between the passes, at the detector geometry pass 2 runs at, the lattice is indexed under the stored angles and under the fitted $k$, and each is scored on the validation frames spread over the whole sweep, as the share of their spots it puts on the lattice beyond what it puts there at a wrong spindle angle. The fitted $k$ is adopted only where it scores higher by more than the binomial noise of the two scores (z = 3.29); the run then integrates and post-refines at it, fits $k$ again on top of it, and walks to the fixed point of the fit: once off the stored angles, a further step is refused only where the validation frames score it significantly worse, since a first pass indexes through a degree of drift late in the sweep and the frames can score two rates alike while integration at one of them loses half the signal. Where the run integrates pass 1's own lattice at the adopted $k$, that lattice (cell and orientation, the detector and axis held) is first refit at the angles the pass runs at, on the validation frames of the whole sweep, and kept where it indexes no fewer of them - a cell fitted at the stored angles has absorbed part of the rate error and would undo it. Otherwise the stored angles stand. The adopted $k$ drives every later pass (prediction, integration, scaling and the reported oscillation) and is reported as `GONIOMETER_ROTATION_SCALE`, with `GONIOMETER_ROTATION_SCALE_SUSPECT= TRUE`. `--rotation-scale ` asserts a calibration and skips all of this. ### 7.6 Detector geometry from powder rings @@ -417,7 +417,10 @@ merging both mixes two structures. The frames merged are the population the give peak of the histogram of $v/v_\mathrm{given}-1$ nearest to zero, out to the valley on either side that separates it from another population, with a bin that follows the spread of the data, so neither a population's width nor its distance to the next is a fixed number. A run with one population has no -such valley and keeps every frame. The log says how many frames were left out on either side. +such valley and keeps every frame. The log says how many frames were left out on either side. The stills +geometry refinement fits on the same population: its sampled frames are filtered by it before the bundle +and the module placement are formed, the population told from the strong frames (at least 20 indexed +spots), whose cells show the valley between two forms where a weak frame's cell spreads across it. **Detector modules.** Where the detector is described module by module ([Module geometry](RUGNUX_FORMATS.md#module-geometry)), the same refinement places each module with diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 59ede1ac6..1bde68271 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -29,6 +29,8 @@ Accepted reflections are projected to the detector by intersecting the diffracte When the beam has a finite energy bandwidth, this window is **broadened radially per reflection**: the cutoff is combined in quadrature with a bandwidth smear, $\sqrt{\Delta_\mathrm{cut}^2 + (3\,\sigma_\mathrm{bw})^2}$, where $\sigma_\mathrm{bw}\propto|p_z|$ (the reciprocal-space depth along the beam, growing as $\sim 1/d^2$). This keeps high-resolution reflections — smeared by the bandwidth into radial streaks — from being clipped. The same $\sigma_\mathrm{bw}$ is deconvolved from the measured profile radius (§11.1), so it is not double-counted. +**The window follows the run's rocking curve.** A cutoff of twice a frame's own profile radius (§11.1) is one width at every resolution, while the rocking curve widens with $d^*$: it predicts many widths at low resolution, mostly empty, and cuts the curve at high resolution. So, with the stills partiality post-refinement on (the default, §10.2), a sample of the run - a tenth of its frames, at least 2000 - is first integrated with that cutoff, scaled against its own merge, and the run's rocking curve measured on it as the merge measures it (§10.2): its width $\sigma(d^*)$ and its centre, the radial scale. Every frame is then predicted through $|\Delta_\mathrm{Ewald}| \le 2\,\sigma(d^*)$, with $\Delta_\mathrm{Ewald}$ taken on the reciprocal lattice scaled by that radial scale, and the bandwidth term enters through $\sigma(d^*)$ instead of the quadrature above. With `--simple-stills` the frame's own cutoff stays. + ### 8.3 Rotation prediction (Laue equation + partiality model) For rotation/oscillation datasets, Jungfraujoch solves for rotation angles $\phi$ where the rotated reciprocal lattice point satisfies the Ewald-sphere condition. In an XDS-like notation, define: @@ -196,7 +198,7 @@ spot-width variance rather than as an intensity scale. ### 9.5 Choosing the signal radius from the crystal's own spots (rotation) -The three radii are one triple for the whole run, but on rotation data they are no longer a fixed constant: $r_1$ is measured from how wide *this* crystal's spots actually are (`--adaptive-integration-radius`, on by default for rotation, off for stills, ignored when `--integration-radius` is given). +The three radii are one triple for the whole run, but on rotation data they are no longer a fixed constant: $r_1$ is measured from how wide *this* crystal's spots actually are (`--adaptive-integration-radius`, on by default, ignored when `--integration-radius` is given). On stills the same option measures instead how far the background ring should reach (§9.2, *When it is applied*). **Why $r_1$ matters even though it does not set the intensity.** In the default `gaussian` mode the intensity is a profile-fit amplitude over the grid $\lceil r_2\rceil$ (§9.3), so $r_1$ is not the integration domain. It *is* the aperture the profile **width** is learned over, and a second moment taken over a disk of radius $a$ saturates at $a^2/4$. At $r_1 = 4$ the learned $\sigma$ can therefore never exceed 2 px, and a crystal whose spots are broader than that is fitted with a profile the model cannot represent. @@ -337,9 +339,9 @@ The partiality applied is fixed by the data type and scaling stage, not chosen f 2. **Unity** ($P_{ij}=1$): used for the scale-on-fulls refit (§10.6), where each observation is already a complete reflection. -3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius (resolution-independent), the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature. The fit typically drives $\gamma_e\to0$, leaving the resolution-independent $\gamma_0$ as the effective width. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times. +3. **Fixed**: use the per-reflection partiality carried from prediction. Still/serial images are predicted with $P=1$, so a single-pass stills scale is effectively unity/fixed — which is exactly what `--simple-stills` keeps. By default the stills path instead **post-refines a physical partiality**: a small per-crystal orientation tilt $(\delta\psi_x,\delta\psi_y)$ about the two axes perpendicular to the beam is refined against the running merge, and every reflection's partiality is then recomputed analytically from the refined lattice through its excitation error $\Delta_\mathrm{Ewald}=\big|\,|\mathbf{q}+\mathbf{S}_0|-1/\lambda\,\big|$ and a Gaussian width $\sigma(d^*)$ — the run's rocking width, measured once on the intensity-weighted excitation errors and shared by every crystal. It is measured per group of frames of similar prediction window (four per shell), each against its own window, the shell taking the median of the groups whose curve is narrower than their window, so the frames with the widest windows do not set it everywhere; and it is measured in eight shells of equal width in $d^*$, made non-decreasing (pool-adjacent-violators, weighted by each shell's summed signal) and interpolated linearly in $d^*$ as a table. The three-term form $\sigma^2=\gamma_0^2+(\gamma_e d^*)^2+(\mathrm{bw}\,|q_z|)^2$ — the reciprocal-lattice point's own radius, the mosaic/divergence spread, and the bandwidth smear along the beam, in quadrature — is still fitted, sets the prediction window (§8.2) and stands in for the table where fewer than three shells are measurable. The curve is centred on the run's **radial scale** of the reciprocal lattice — a wavelength or cell a little off, which the spot positions cannot see since they fix only $\lambda d^*$ — measured from where the curve's centre lies against $d^{*2}$ and applied to $\mathbf{q}$. Where the frames were predicted through the measured window (§8.2), the post-refinement uses that same curve rather than measuring it again; the curve is written to `_process.h5` (`/entry/MX/stillsRockingWidth*`, `stillsRadialScale`), so `--mode scale` merges with it too. A tilt moves reflections on opposite sides of the Ewald sphere in opposite directions, so it reshapes the *spatial* pattern of partialities — a degree of freedom the per-image scale $G$ does not have, and the reason the tilt is refined rather than a scalar partiality width, which would be degenerate with $G$. Nothing is re-integrated (the integrated intensities are fixed); the tilt is hard-bounded at about 1° and held by a soft prior, so it stays inert on sparse or weak crystals. The cycle is merge → per-crystal tilt refinement (with $G$ profiled out by the same robust Cauchy IRLS used for the per-frame scales, §10.3) → recompute $P$ → re-merge, repeated a few times; the refined partiality is adopted on every crystal it was refined on. The per-crystal fits of $G$ and of the tilt measure each residual against its expected variance at the model, background + counting + $(0.3\,I_\mathrm{model})^2$: a still's strong reflections scatter about the model by tens of per cent, and in counting sigmas alone every strong reflection would be an outlier and the scale would be set by the weak ones. -Reflections below a minimum partiality can be rejected from merging to avoid unstable corrections. +Reflections below a minimum partiality are left out of the merge (`--min-partiality`, default 0.02 on rotation and 0.5 on stills): the empirical rocking curve of a still has a floor far out on its tail, from mis-oriented crystals, and dividing such an observation by a small model $p$ makes it biased, not merely noisy. ### 10.3 Smoothing of per-frame scales @@ -493,7 +495,7 @@ The “profile radius” is the intrinsic angular width of a reflection — crys $ R \approx \sqrt{\tfrac{1}{N}\sum_i \Delta_{\mathrm{Ewald},i}^2}. $ -When the beam has a finite energy bandwidth, that bandwidth smears each reflection radially by $\sigma_\mathrm{bw}\approx \mathrm{bandwidth}\cdot\lambda/2d^2$ (largest at high resolution), which also broadens the measured $\Delta_\mathrm{Ewald}$ spread. Since prediction re-applies the bandwidth term per reflection (§8.2), this contribution is deconvolved from the estimate — $R^2 = \langle\Delta_\mathrm{Ewald}^2\rangle - \langle\sigma_\mathrm{bw}^2\rangle$ — so that $R$ is the intrinsic width and bandwidth is not double-counted. Still predictions use an excitation-error cutoff proportional to $R$. +When the beam has a finite energy bandwidth, that bandwidth smears each reflection radially by $\sigma_\mathrm{bw}\approx \mathrm{bandwidth}\cdot\lambda/2d^2$ (largest at high resolution), which also broadens the measured $\Delta_\mathrm{Ewald}$ spread. Since prediction re-applies the bandwidth term per reflection (§8.2), this contribution is deconvolved from the estimate — $R^2 = \langle\Delta_\mathrm{Ewald}^2\rangle - \langle\sigma_\mathrm{bw}^2\rangle$ — so that $R$ is the intrinsic width and bandwidth is not double-counted. Still predictions use an excitation-error cutoff proportional to $R$ where the run's rocking curve is not measured (`--simple-stills`, and the sample the curve is measured on; §8.2). ### 11.2 Mosaicity from rotation data diff --git a/docs/GPU_DATA_ANALYSIS.md b/docs/GPU_DATA_ANALYSIS.md index 1a16a2914..24a57b695 100644 --- a/docs/GPU_DATA_ANALYSIS.md +++ b/docs/GPU_DATA_ANALYSIS.md @@ -96,7 +96,8 @@ nothing. ## Prediction and integration - **Still prediction** (`BraggPredictionGPU.cu`) - the reflections of a lattice within the - excitation-error cutoff, with their detector positions and per-reflection corrections. + excitation-error cutoff - or the window that follows the run's rocking curve, centred on its radial + scale - with their detector positions and per-reflection corrections. [§8.2](CPU_DATA_ANALYSIS_INTEGRATION.md#82-still-prediction-excitation-error-cutoff) - **Rotation prediction** (`BraggPredictionRotGPU.cu`) - the reflections crossing the Ewald sphere within an image's rotation range, with their rotation angle and partiality; they are ordered, and