diff --git a/docs/CPU_DATA_ANALYSIS_IMAGE.md b/docs/CPU_DATA_ANALYSIS_IMAGE.md index 8ceb64824..267c9e392 100644 --- a/docs/CPU_DATA_ANALYSIS_IMAGE.md +++ b/docs/CPU_DATA_ANALYSIS_IMAGE.md @@ -288,10 +288,20 @@ the measured one (the measured one counting where its σ is within $\max(1\ \mat header that is far out can still be corrected. Where the refined pass is judged worse, the run goes back to the pass-1 geometry. -**Stills.** There is no second first pass and no post-refinement on stills; the background -measurement is reported only. The centre moves only through the stills geometry refinement, -`--refine-geometry`, which bundle-adjusts the beam centre, -distance and cell over strongly indexed frames and is on by default where a reference cell is given. +**Stills.** There is no second first pass and no post-refinement on stills; the centre is decided +before the images are processed, as competing hypotheses tested on the data. Spots are found once on a +sample of frames spread over the run, and three centres are tried: the file's, the background +measurement above, and the point-symmetry capture (a second estimator, because the background one is +biased where a shadow - a sample-delivery tape, say - covers part of the scattering). Each re-indexes the +same spots and is scored by how many land on their frame's lattice; the file's centre is kept unless +another puts at least 1/0.9 as many there. Where a reference cell is given, the detector distance is a +hypothesis too: a grid over 60-160 % of the file's distance, then a finer one about the best, from every +centre candidate, followed by a local grid of centres at the distance decided. A distance given with +`--detector-distance` is taken as given and not searched. Without a cell nothing is searched in +distance: on its own a wrong distance indexes as a wrong cell, not as fewer indexed spots. The stills +geometry refinement, `--refine-geometry`, then bundle-adjusts the centre, the cells and - where the +frames rejected the file's distance, or the file states it only nominally - the distance over strongly +indexed frames (§7.7); it is on by default where a reference cell is given. **What ends up in the output.** The report's `BEAM_CENTRE` (the PONI) and `DIRECT_BEAM` are the geometry the result was integrated at, and so is `JFJOCH_DATASET_SETTINGS`; `POSTREFINE_BEAM_CENTRE` @@ -418,6 +428,19 @@ value on most frames (already invalid on every frame, but not in the static mask them. A CCD (no sensor depth: read with an offset, not Poisson) is left alone. One log line says how many pixels were masked and why. +### 1.7 The saturation level + +A detector that saturates clips every value beyond its ceiling down to it, so on a frame's bright tail - +the cores of Bragg peaks, which otherwise fall off steadily with value - the values pile up just below +where they stop. Where each pixel has its own ceiling (an ADC limit less that pixel's dark), the +pile-up is a band, and its lower edge is where clipping starts. A file's own statement of the limit is +often missing or wrong for detectors outside the facility that wrote it, so on stills Rugnux reads it +off the values: a histogram of pixel values on a sample of frames, binned on a log scale, and the +lowest bin that rises above the tail below it. Pixels from there on are treated as overloaded where that +level is below the file's; the file's is kept for a file Jungfraujoch wrote, whose limit is the +detector's own, and when `--saturation` is given. The log states both values. Where an XFEL file +states no limit at all, the reader holds the values to their integer type's range. + --- ## 2. Azimuthal integration (radial profiles) diff --git a/docs/CPU_DATA_ANALYSIS_INDEXING.md b/docs/CPU_DATA_ANALYSIS_INDEXING.md index f600a7951..35292acf0 100644 --- a/docs/CPU_DATA_ANALYSIS_INDEXING.md +++ b/docs/CPU_DATA_ANALYSIS_INDEXING.md @@ -371,3 +371,32 @@ The **distance** is different: it follows from $r=F\tan2\theta$ with $\sin\theta A plain radial profile — one azimuthal sector — has averaged the ring over every direction and carries no centre at all, so the profile route requires at least four sectors and uses 32 by default. Sixteen to thirty-two are enough; beyond that the limit is the ring's own texture, not counting statistics. The extraction window around a ring is capped at half the gap to its neighbour, because the background under a peak is taken from the ends of that window: hexagonal ice has a triplet of rings (1.947, 1.916 and 1.882 Å) whose neighbours sit only 0.05–0.06 Å⁻¹ apart in $q = 2\pi/d$, which a fixed window merges into a single peak. Where only one ring is in reach the two tilts are held at their input values rather than fitted, since on a single ring they are degenerate with the centre (above) and the fit would otherwise trade the centre away for them. + +### 7.7 Stills: a given cell and the detector modules + +**A given cell is a starting point.** On stills a cell and a distance are degenerate - a spot's +radius fixes only their ratio. A cell taken from a deposited structure is routinely a percent off the +crystal at hand (another temperature, another batch), while a facility's stated distance is usually +better than that, so the stills geometry refinement holds the distance and refines the cell. Only where +the frames have rejected the file's distance (§1.4), or the file states the distance only nominally, is +the given cell anchored and the distance refined from it. Frames are matched to the given cell in any +setting of the same lattice, and are described on its axes. + +**One crystal form.** A sample can hold two crystal forms, and a known-cell indexer fits both to the +given cell's orientation search; their refined cells then fall into two volume populations, and +merging both mixes two structures. The frames merged are the population the given cell sits in: the +peak of the histogram of $v/v_\mathrm{given}-1$ nearest to zero, out to the valley on either side that +separates it from another population, with a bin that follows the spread of the data, so neither a +population's width nor its distance to the next is a fixed number. A run with one population has no +such valley and keeps every frame. The log says how many frames were left out on either side. + +**Detector modules.** Where the detector is described module by module +([Module geometry](RUGNUX_FORMATS.md#module-geometry)), the same refinement places each module with +enough spots on its frames' lattices as a rigid block - a shift and an in-plane turn - where that +predicts held-out spots better: the strong frames are split in half, each half fitted with the modules +fixed and with them placed, and scored on the other. The modules move only relative to one another: a +gauge holds their spot-weighted mean shift, turn and outward push, and with the detector tilt refined +also the two tilt-like gradients across the detector, at zero, so the beam centre, distance, tilt and +crystal orientations carry what all modules share. A coarse search of each module's position comes +first, scored like the beam centre by the spots that land on a lattice. The placement is used from +there on for a foreign multi-panel detector; for a JUNGFRAU master it is measured and reported only. diff --git a/docs/RUGNUX_OVERVIEW.md b/docs/RUGNUX_OVERVIEW.md index 9fde32503..7899448a6 100644 --- a/docs/RUGNUX_OVERVIEW.md +++ b/docs/RUGNUX_OVERVIEW.md @@ -84,6 +84,14 @@ space group or the twin hypothesis' subgroup, that merge is written beside the a (with the known-cell `ffbidx` indexer where a cell is given), partiality comes from a per-crystal orientation-tilt post-refinement rather than a rocking curve, and a merohedral indexing ambiguity has to be broken per image, at integration time, against a reference or a model -([Advanced ▸ the indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity)). +([Advanced ▸ the indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity)). The input may be +several files given together - Jungfraujoch masters, or the files an XFEL pipeline writes, read without +a geometry file ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). Before the images are +processed the beam centre is decided on a sample of frames, the file's against the measured ones, and a +given unit cell also decides the detector distance there; the cell is a starting point the frames +refine, and only the frames of its crystal form are merged. With a cell, the geometry refinement places +each detector module as well ([§1.4](CPU_DATA_ANALYSIS_IMAGE.md#14-the-beam-centre), +[§7.7](CPU_DATA_ANALYSIS_INDEXING.md#77-stills-a-given-cell-and-the-detector-modules)). The report adds +R_split and a CC1/2 from frame half-sets, as CrystFEL computes them ([Report](RUGNUX_REPORT.md)). -![The stills pipeline: every image indexed on its own, the geometry refined first where the cell is known, and the crystals scaled and merged together](images/rugnux_pipeline_stills.svg) +![The stills pipeline: the beam centre (and, with a cell, the distance and the module placement) decided on the frames first, every image indexed on its own, and the crystals scaled and merged together](images/rugnux_pipeline_stills.svg) diff --git a/docs/images/rugnux_pipeline_fig.py b/docs/images/rugnux_pipeline_fig.py index a2f57c4f7..3a94bb767 100644 --- a/docs/images/rugnux_pipeline_fig.py +++ b/docs/images/rugnux_pipeline_fig.py @@ -188,10 +188,10 @@ def stills(): # ---------- Input, geometry refinement, pre-scan - box(IX, IY, IW, IH, "Input", "HDF5, CBF, SMV, marCCD", INK, bs=14) + box(IX, IY, IW, IH, "Input", "HDF5, CXI, XTC, CBF, SMV", INK, bs=14) GX, SX = 560, 900 - box(GX, SY, 240, BH+22, "Geometry", ["with a known cell:", "beam, distance, cell"], TEAL, autodiff=True, dashed=True) - box(SX, SY, 240, BH+22, "Pre-scan", ["beam-stop shadow,", "bad pixels"], TEAL, gpu=True) + box(GX, SY, 240, BH+22, "Geometry", ["beam centre from the frames;", "with a cell: distance, panels"], TEAL, autodiff=True) + box(SX, SY, 240, BH+22, "Pre-scan", ["beam-stop shadow,", "bad pixels, saturation"], TEAL, gpu=True) path(f"M{IX+IW},{IY+40} L{GX-2},{IY+40}") label((IX+IW+GX)/2, IY+32, "strong frames", size=13) @@ -204,7 +204,7 @@ def stills(): label(SX+132, SY+BH+22+28, "masks", anchor="start", size=13) # ---------- Whole-dataset chain - dataset_row(["consensus cell,", "per-image scale"], "space group, twinning, tNCS", + dataset_row(["consensus cell, crystal form,", "per-image scale"], "space group, twinning, tNCS", ["in space group, resolution cut", "partiality, error model"], gpu=False) path(f"M{C[1]+110},{R2Y+BH} L{C[1]+110},{OUTY} L{QX[0]+120},{OUTY} L{QX[0]+120},{QY-2}") label(QX[0]+135, OUTY-8, "every crystal", anchor="start", size=13) diff --git a/docs/images/rugnux_pipeline_stills.svg b/docs/images/rugnux_pipeline_stills.svg index 1211c6863..3ca41a8c9 100644 --- a/docs/images/rugnux_pipeline_stills.svg +++ b/docs/images/rugnux_pipeline_stills.svg @@ -50,17 +50,17 @@ Input -HDF5, CBF, SMV, marCCD - +HDF5, CXI, XTC, CBF, SMV + Geometry -with a known cell: -beam, distance, cell +beam centre from the frames; +with a cell: distance, panels autodiff Pre-scan beam-stop shadow, -bad pixels +bad pixels, saturation CUDA @@ -75,7 +75,7 @@ Whole dataset Scale & merge (P1) -consensus cell, +consensus cell, crystal form, per-image scale Symmetry