Beside --beam-x/--beam-y/--detector-distance/--wavelength, the remaining metadata a foreign XFEL
file can get wrong are given on the command line rather than through a hand-built geometry file:
--pixel-size (um); --saturation (the pixel value from which a pixel is overloaded, in the units
the frames are read in), which also replaces the level stills read off the frames; and --gain
(detector units per photon), for CXI and --crystfel-geometry input, whose reader converts the
frames to photons with it - refused for other inputs, which are read in the units they state.
Single-panel Rayonix CXI, 1316 frames, -C/-S, --detector-distance 105: --gain 0.37 (the pre-scan's
background measurement) gives rugnux R-free 0.344 against 0.358 in detector units.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Foreign stills files state their saturation level wrongly or not at all: an MPCCD Cheetah file is
dark-subtracted per pixel, so each pixel clips at the ADC limit less its own dark, and its
CrystFEL geometry leaves max_adu commented out - the run reported "Overloads: 0" while the inner
panels saturated in most frames.
saturation_ceiling (image_preprocessing/SaturationCeiling) histograms pixel values on a log
scale (64 bins per octave from 256) and returns the lower edge of the lowest bin holding over
twice what any of the 8 bins below it held (and at least 50 values): a bright tail only falls, a
clipped one piles up below where it stops, as a band where the level differs per pixel.
DetectSaturation runs it on 40 frames spread over every stills run and logs it beside the file's
limit; for any input but a Jungfraujoch-written file (ProcessConfig::detect_saturation) a level
below the file's replaces it. The CrystFEL/CXI reader, where the geometry states no max_adu, now
takes the stored integer type's maximum (an int16 Cheetah file clips at 32767).
The geometry-file-free CXI path (previous commit) is limited to square frames: a tiled detector's
Cheetah slab (CSPAD, AGIPD, MPCCD, ...) read as one flat panel would place every pixel wrongly
without a word, and those slabs are not square.
Measured:
MPCCD stills, 1339 frames: pile-up from 1218 photons (by hand 12000 ADU ~ 1200 ph); 5.5% of
reflections coarser than 4 A flagged; lowest-shell CC1/2 0.545 -> 0.674, overall 0.741 ->
0.793; same dimple refinement against the deposited model R-free 0.2586 -> 0.2556
single-panel Rayonix CXI, 1316 frames: no pile-up (20 pixels at the int16 limit in 40 frames);
the int16 limit flags 123 reflections; CC1/2 0.737 -> 0.787, rugnux R-free 0.382 -> 0.358
JUNGFRAU stills (Jungfraujoch files: lysozyme and two private sets): no pile-up below the stated
limit, results bit-identical
Cost 0.5 s per run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stills never used the beam centre the pre-scan measures - the "second first pass indexes both
centres" exists only in the rotation path - so a header centre a dozen pixels off indexed a few
percent of the frames, or a wrong cell.
DecideStillsGeometry runs after the pre-scan: spots are found once on 96 spread frames, and each
hypothesis re-indexes those spots (IndexFrameOnly) at its geometry; the score is the number of
found spots on the lattice of their frame. Centres: the file's, the background-isotropy
measurement and the point-symmetry capture (kept beside it now; the two estimators fail
differently - on a single-panel Rayonix with a tape shadow the isotropy fit reads 4 px off, the
capture 1.4 px). With a reference cell the distance is a hypothesis too: a 2.5% grid over
60-160% of the file's on the 32 strongest frames, a 1% one about its best on all 96, then the
centre decided again at the new distance. The file's value is kept unless another puts at least
1/0.9 times as many spots on the lattice. Where the geometry moved, the stills bundle adjustment
(-C) runs again from there, since its bounds are +-8 px / +-8 mm about its start. Without a cell
the distance is not searched: a wrong distance indexes as a cell of the wrong size.
The CrystFEL reader also opens a Cheetah CXI file (or a directory of them) with no .geom: one
flat panel, pixel size, photon energy and camera length from the file, centred on the detector.
Measured (single-panel Rayonix XFEL stills, a P6122 crystal, 1316 frames, -C/-S, ice rings on):
hand-built .geom (measured centre, 105 mm) 54.56% indexed, rugnux R-free 0.376
plain CXI + --detector-distance 105 54.71%, 0.382 (centre found: 733.2,717.8)
plain CXI, file's wrong 89.4 mm kept 54.41%, 0.367 (distance found 105.5 -> 104.97 refined)
A second single-panel Rayonix set, P6522, 1000 frames, -C/-S, centre moved off the .expt value:
0 px 59.26% R-free 0.286; 14 px 59.16% 0.298; 28 px 59.16% 0.285 (before: 5-10% indexed)
Unchanged (bit-identical merge statistics and R-free): JUNGFRAU lysozyme stills, 4,000 frames (file centre
kept, +1.6 s) and mpCCD 1339 frames (file centre kept, +2.5 s). The trial costs 1.6-2.5 s without
a cell and 2-11 s with one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A CrystFEL .geom (current and old syntax) is parsed for panels (pixel ranges, fs/ss, corners),
clen/coffset (a number, or an HDF5 path read as mm), res, adu_per_eV/adu_per_photon, max_adu,
no_index, bad regions (panel, whole-frame or lab-plane) and the HDF5 mask (mask/mask_file,
mask_good/mask_bad). JFJochCrystFELReader reads the frames the geometry's data path names - a 3D
stack (Cheetah CXI), one 2D frame per file, or one group per shot (/%/data) - from a file or a
directory, puts the panels on one flat image at their rounded positions with gaps and bad regions
masked, and converts to photons where a gain is stated (otherwise the pre-scan measures it). One
wavelength and one camera length per run: values taken from the files are averaged over the first
file. Panel positions are used as given. Frames stored as one deflate(+shuffle) chunk are read raw
under the HDF5 lock and decoded outside it.
Replaces the MPCCD-specific reader of the previous commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leonarski_fandClaude Opus 5.5