mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 87 Wiki Activity
Files
f199f23c54fc6c9e29b8560ab7cd3c2893dd94de
Jungfraujoch/docs/EXTERNAL_TEST_DATA_STILLS.md
T
leonarski_fandClaude Opus 5.5 0a515bb086 rc179: everything on rc178 after its merge into main (#88), except the CHANGELOG 
The rc178 branch continued after #88 was squash-merged; this is its content from b9ada7914 (the tree
#88 squashed) to 094eee31a, as one change on main. The individual commits stay on the rc178 branch:

- still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets
- DetectorModules: panel look-up by id, plane -> panel, plane bounds and a per-pixel walk; mask size checked
- tests: [PanelLayout] equivalence harness, stacked vs assembled panel layout
- Dense per-pixel maps through the panel table (panel layout stage 1)
- Panel id on spots and the panel table in the start message
- Viewer and preview show a panel detector as it lies on the detector plane
- Panel layout stage 3: per-panel spot finding (CPU+GPU, CCL) and raster algorithms on the plane raster
- Panel layout stage 2: prediction and Bragg integration through the panel table
- XTC reader: frames handed up as stored, streams indexed in parallel
- XFEL reader: integer frames stay integer, one gain, placement by panel
- EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"
- still battery: EuXFEL sets read the depositors' Cheetah CXI files directly; AGIPD converter removed
- float32_to_int16.py: a non-finite pixel becomes -32768, the int16 error marker, not a zero count
- CrystFEL geometry: read the pixel masks and bad regions instead of ignoring them
- Panel layout stage 5: the XFEL reader hands up the stored frame and its panel table
- Panel table persisted in NXmx; modulation grid and pre-scan memo key on the plane
- still battery: Alvra sets take their pixel mask from the CrystFEL geometry, not --mask-good
- still battery: take the deposited model as mmCIF where the RCSB has no PDB-format file
- still battery: dimple gets PDB-format coordinates converted from an mmCIF-only deposition
- stills fixed cost: geometry refinement and pre-scan do the same work in less wall time
- stills geometry sample and trial spot finding run as many workers as the image pass
- Stills integration: background ring reach measured on empty apertures, radial curvature from the whole frame
- StillsPartialityRefine: the partiality width is the run's, measured on the crystals, not the prediction window's
- MergeOnTheFly: capture term and multiplicative outlier band for the stills merge (both opt-in)
- Stills given a cell: the written cell is the mean over the frames that are merged
- Stills: the radial curvature correction decided per run on the same empty apertures
- StillsPartialityRefine: measure the run width only on shells with signal
- Stills pre-scan: report the paired z of the radial-correction comparison
- module placement: the five concurrent fits each keep every thread
- still battery: score R-free against the deposited data through the same dimple protocol
- still battery: the R-free target is the lower of the deposition's dimple and published values
- Stills: background clip follows the radial curve; signal disk from the lattice spacing
- Stills keep an overloaded observation as a lower bound instead of dropping it
- Stills: drop the lattice-spacing signal disk; the correction decision back on the plain clip
- battery: R-free against the deposition without refinement only, one protocol for rotation and stills
- still battery: EuXFEL sets run on a 2,000-image subset for now
- stills geometry refinement: fit the detector distance and the cell together
- stills geometry refinement: a distance given with --detector-distance is held in the joint fit
- Metal integration engine: back to its panel-s5 kernels; Mac stills keep the old clip, no saturation count
- Metal integration test: the CPU comparison runs on a rotation experiment
- Stills: kept overloads go back to the drop where they are not concentrated at low resolution
- stills distance search: an absolute 30-3000 mm scan, a sample found without d-limits, the fit iterated
- battery results: XFEL still battery 20261022-1358 on rc178 0cf62452c - 12 sets, cold, R_free unrefined
- battery results: still summary gains the resolution of our merge and of the deposition
- Pre-scan masks a pixel at the saturation limit on most frames, as one holding the error value
- Stills keep an overload as a lower bound only where it has pixels at the limit; run-level gate removed
- XFEL reader: a per-frame mask is read the way the frames are, its chunk inflated outside the HDF5 lock
- stills distance search: every centre scanned with a wider acceptance, a larger sample where nothing indexes
- rugnux usage: --beam-center-check describes the absolute 30-3000 mm stills distance scan, not the old 60-160 % grid
- stills: detector tilt refined in the geometry pre-pass, kept where held-out frames agree
- CLAUDE.md: the changelog is written once, just before the merge into main - condensed one-liners of what users get
- broker: serve the placed model and its maps of the last rugnux live run (/rugnux/model.cif, /rugnux/map.ccp4)
- frontend: Mol* viewer of the placed model and its density on the Rugnux live page
- writer: _model.cif from rugnux live keeps the input model's metadata, as the served model.cif does
- rugnux live maps: one unit cell centred on the model; the viewer opens on all of it
- Anomalous difference map: ANOM/PHANOM in _maps.mtz, /rugnux/map.ccp4?type=anom, a magenta mesh in the viewer
- Mol* viewer: open with the maps drawn within 10 A of the centre of view
- Mol* viewer: aaredb's structure-viewer setup, on a black stage
- rugnux: the gain of a stills file that states none is read off its single-photon peak where the background resolves photons
- Mol* viewer: in line with AareDB's StructureView at its current head (8ece2c2)
- ACKNOWLEDGEMENT: the web frontend's Mol* viewer follows AareDB's setup (G. Gotthard, PSI)
- stills tilt decision: the modules stay where the run placed them
- still battery: no distance pins and no per-set tuning flags; gain from the data except where a file's baseline hides it
- battery results: rc178 6f6f69 - full still battery and full rotation battery with REFMAC
- Bragg integration: a CCD rotation sweep's sigmas at the measured gain, intensities in photons
- BackgroundGain: the gain of a frame that resolves photons is the spacing of its photon peaks, so a baseline left in the frame no longer hides it
- ShadowFinder: measure the background's azimuthal modulation instead of predicting it from the polarization
- BackgroundGain: one gain estimate for rotation and stills; the still battery no longer gives EuXFEL 152 its gain
- ShadowFinder: measure the modulation robustly under a wide one-sided shadow
- ShadowFinderTest: a one-sided shadow is not fitted as part of the background's modulation
- ShadowFinder: back to the predicted polarization factor, in the plane the background shows
- MXAnalysisAfterFPGA: the Bragg engine's edge mask in the image's layout, so raw / untransformed FPGA runs start again
- battery results: still battery on rc178 e54f55 replaces the earlier rc178 still tables
- XFELReader: per-frame mask and photon conversion without a library call or a branch per pixel
- stills geometry trial: the hypotheses a search asks are indexed side by side
- stills tilt decision: the fits start beside the pre-scan and the geometry trial
- XFELReader: a deflated frame is inflated and converted a piece at a time, while it is in the cache
- XFELReader: a byte-shuffled frame is unshuffled a piece at a time, a whole pixel per step
- GeometryRefiner: residuals carry only the blocks a fit moves; beam and distance one block
- GeometryRefiner: a spot's residual differentiates its observed and predicted halves apart
- docs: EXTERNAL_TEST_DATA lists every open-arm battery set, with corrected values and counts
- battery: doc_check.py cross-checks the manifests against the dataset pages
- stills module placement: the placement fit gets half the threads, each held-out fit an eighth
- still battery: the EuXFEL sets run on all their frames, as the published still runs did
- battery results: still battery on rc178 a0ce6c, cold on a quiet machine, replaces the earlier rc178 table
- macOS build: explicit size_t in the XFEL reader's row bounds; parentheses in the CPU preprocessor's select

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-23 17:56:42 +02:00

77 lines
9.3 KiB
Markdown
Raw Blame History

# External test data: serial crystallography (XFEL stills)
The serial datasets of the still battery (`tools/battery/still_battery.py`, manifest
`tools/battery/still_manifest.json`). Like the rotation sets on
[External test data](EXTERNAL_TEST_DATA.md), they were collected by other people, on detectors and in
file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them
come from the [Coherent X-ray Imaging Data Bank](https://cxidb.org) (CXIDB).
**None of these data were collected by us.** If you use any of them, cite the dataset DOI below and
the paper of the entry.
## Where the values come from
- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from
the RCSB data API. Every entry is a dark or ground state refined against observed data.
- **Images used** are the frames in the files we use, counted in the files themselves. Where that is
only part of the entry, the table says so and gives what the entry or the deposition holds.
## Datasets
| Set | CXIDB, dataset DOI | Facility / beamline | Detector, format | PDB | Space group, d<sub>min</sub> (Å) | Paper |
|---|---|---|---|---|---|---|
| `sacla243` | [243](https://cxidb.org/id-243.html), [10.11577/17003933](https://doi.org/10.11577/17003933) | SACLA BL2 | MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 | [31JT](https://www.rcsb.org/structure/31JT) | P 2 21 21, 1.63 | Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026) |
| `sacla237` | [237](https://cxidb.org/id-237.html), [10.11577/2999572](https://doi.org/10.11577/2999572) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9S06](https://www.rcsb.org/structure/9S06) | P 21 21 21, 2.05 | Rios-Santacruz et al., Nature (2026) |
| `sacla236` | [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9V3D](https://www.rcsb.org/structure/9V3D) | P 43 21 2, 1.68 | Kang et al., J. Appl. Cryst. (2026) |
| `pal86` | [86](https://cxidb.org/id-86.html), [10.11577/1502065](https://doi.org/10.11577/1502065) | PAL-XFEL NCI | Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI | [6IG7](https://www.rcsb.org/structure/6IG7) | P 61 2 2, 1.80 | Park et al., Sci. Rep. (2019) |
| `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) |
| `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) |
| `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) |
| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) |
| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) |
Every set of the manifest is in the table and every set in the table is in the manifest, with the
same CXIDB entry, PDB entry, space group and d<sub>min</sub>; `tools/battery/doc_check.py` checks
this. The beamline of `pal90` is the CXIDB entry's (NCI); the PDB entry 6IRK records CSI.
## The files we use
Paths are relative to the still data root of the site config (`tools/battery/README.md`). CXIDB
serves `https://cxidb.org/data/<entry>/...`; most of it accepts HTTP range requests, so a
broken download resumes with `curl -C -`.
| Set | Files | Images used | Of the entry | Size |
|---|---|---|---|---|
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | all (the deposition's 52,151) | 198 GB |
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` | 10,792 | **subset**: 18 of about 1,000 `dark_ref` files, about 1.8% of ~600,000 hits | 48 GB |
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | all (the deposition's 16,083) | 67 GB |
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | all but file 1 (24 of 25 files) | 111 GB |
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | all (both files) | 25 GB |
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | **subset**: run 165 of the runs 165-168 behind 8TDQ | 72 GB |
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB |
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB |
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB |
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry; the set reads `swissfel_103/ref/jf-16M_mask.geom`, a copy with `mask_file` pointed at that mask (see below) | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB |
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs, through `swissfel_213/ref/mbco_march_mask.geom`, a copy with `mask_file` pointed at it (see below) | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB |
**The masks of the SwissFEL geometry files.** rugnux takes the pixel masks and bad regions a CrystFEL
geometry file names. The SwissFEL files name their mask with an absolute path on the facility's disk,
so each set reads a copy whose `mask_file` is relative to the copy's own directory (a relative mask
file is looked for in the working directory first, as CrystFEL does, then beside the geometry file):
`swissfel_103/ref/jf-16M_mask.geom` is `jf-16M.geom` with `mask_file = run47.JF06T32V01.mask.h5`, and
`swissfel_213/ref/mbco_march_mask.geom` is `mbco_march.geom` with
`mask_file = ../raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5`. Only
that line differs.
**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
`_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided
by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them
with `--gain 1` and merges them to the same numbers. Make them with
`tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV>` (h5py and hdf5plugin),
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's
directory tree), named `<input>_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy.
Reference in new issue View Git Blame Copy Permalink