Files
Jungfraujoch/docs/EXTERNAL_TEST_DATA_STILLS.md
leonarski_f 7f2ebd8209
Build Packages / Build documentation (push) Successful in 46s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m25s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m46s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 13m25s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 16m41s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m46s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m37s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 24m31s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m9s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m47s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 26m21s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 27m36s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 29m27s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m46s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 25m22s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 30m36s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 31m49s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 23m57s
Build Packages / Unit tests (push) Successful in 1h47m52s
Build Packages / Validation Battery (push) Successful in 44m55s
v1.0.0-rc.184 (#95)
* Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner.
* Rugnux: More reliable rotation processing and indexing, including small molecules and supercells.
* Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models.
* jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run.
* jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes.
* jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored.
* jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line.
* jfjoch_viewer: Build and display fixes; the README is rewritten.

Reviewed-on: #95
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-10 22:06:51 +01:00

11 KiB

External test data: serial crystallography (XFEL stills)

The serial datasets of the still battery (tools/battery/still_battery.py, manifest tools/battery/still_manifest.json). Like the rotation sets on External test data, they were collected by other people, on detectors and in file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them come from the Coherent X-ray Imaging Data Bank (CXIDB). They are the eleven datasets of the Rugnux paper's Extended Data Table 2, which gives the published results on them.

None of these data were collected by us. If you use any of them, cite the dataset DOI below and the paper of the entry.

Where the values come from

  • Dataset DOI: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
  • Space group, cell, resolution and R-free are the values deposited with the PDB entry, read from the RCSB data API. Every entry is a dark or ground state refined against observed data.
  • Images used are the frames in the files we use, counted in the files themselves. Where that is only part of the entry, the table says so and gives what the entry or the deposition holds.

Datasets

Set CXIDB, dataset DOI Facility / beamline Detector, format PDB Space group, dmin (Å) Paper
sacla243 243, 10.11577/17003933 SACLA BL2 MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 31JT P 2 21 21, 1.63 Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026)
sacla237 237, 10.11577/2999572 SACLA BL2 MPCCD; Cheetah-SACLA HDF5 9S06 P 21 21 21, 2.05 Rios-Santacruz et al., Nature (2026)
sacla236 236, 10.11577/3014306 SACLA BL2 MPCCD; Cheetah-SACLA HDF5 9V3D P 43 21 2, 1.68 Kang et al., J. Appl. Cryst. (2026)
pal86 86, 10.11577/1502065 PAL-XFEL NCI Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI 6IG7 P 61 2 2, 1.80 Park et al., Sci. Rep. (2019)
pal90 90, 10.11577/1511590 PAL-XFEL NCI Rayonix MX225-HS; Cheetah CXI 6IRK I 2 2 2, 1.75 Lee et al., Sci. Rep. (2019)
lcls222 222, 10.11577/2204603 LCLS MFX Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC 8TDQ P 65 2 2, 1.65 Nguyen et al., J. Am. Chem. Soc. (2023)
lcls242_s, lcls242_t 242, 10.11577/3374214 LCLS MFX Rayonix MX340-HS; LCLS-I XTC 9O4S (small cell), 9O4T (large cell) P 1 21 1, 1.95 Wolff et al., bioRxiv (2026)
euxfel83 83, 10.11577/1464102 European XFEL SPB/SFX AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI 6GTH P 32 2 1, 1.69 Wiedorn et al., Nat. Commun. (2018)
euxfel152 152, 10.11577/1660970 European XFEL SPB/SFX AGIPD 1M; Cheetah CXI 6U57 I 2 3, 2.80 Echelmeier et al., Nat. Commun. (2020)
alvra103_int16 103, 10.11577/1650020 SwissFEL Alvra JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) 6S0L C 2 2 21, 2.65 Nass et al., IUCrJ (2020)
alvra213_int16 213, 10.11577/2282689 SwissFEL Alvra JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) 8BKH (dark state) P 1 21 1, 1.35 Barends et al., Nature (2024)

Every set of the manifest is in the table and every set in the table is in the manifest, with the same CXIDB entry, PDB entry, space group and dmin; tools/battery/doc_check.py checks this. The beamline of pal90 is the CXIDB entry's (NCI); the PDB entry 6IRK records CSI.

How the battery runs them

Each set is one rugnux command, the way a serial user runs: the deposited cell and space group given (-C, -S), the wavelength where the files do not state it, --crystfel-geometry for the AGIPD and JUNGFRAU sets (the geometry file of the entry; the SACLA, PAL and LCLS files are placed without one), and the deposited model with --model, so that the report carries its R-free against the merge. The exact arguments are the args of each set in tools/battery/still_manifest.json; the stills section of the tutorial explains them.

The files we use

Paths are relative to the still data root of the site config (tools/battery/README.md). CXIDB serves https://cxidb.org/data/<entry>/...; most of it accepts HTTP range requests, so a broken download resumes with curl -C -.

Set Files Images used Of the entry Size
sacla243 sacla_243/raw_all/run*.h5: all 87 files of 243/Complete_dark/ 52,151 all (the deposition's 52,151) 198 GB
sacla237 sacla_237/raw/: the 18 files run130637-0 to run130642-2 of 237/SACLA_raw/2022Nov/dark_ref/ 10,792 subset: 18 of about 1,000 dark_ref files, about 1.8% of ~600,000 hits 48 GB
sacla236 sacla_236/raw/: the 18 files run157084-0 to run157089-2 of 236/Data/lyso01-apo-bigXtal/ 16,083 all (the deposition's 16,083) 67 GB
pal86 pal_86/raw/, pal_86/raw_more/: CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi (file 1 is truncated on the server and cannot be opened) 49,193 all but file 1 (24 of 25 files) 111 GB
pal90 pal_90/raw/: CXIDB_90-GI_SFX-filter-r00{01..14}-c00.cxi 79,805 all 14 filtered files of the entry 159 GB
lcls222 lcls_222/xtc/, lcls_222/xtc_more/: mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc (run 165, the substrate-bound state of 8TDQ) 8,534 subset: run 165 of the runs 165-168 behind 8TDQ 72 GB
lcls242_* lcls_242/xtc/, lcls_242/xtc_more/: xtc/mfxl1013721-r0256 to r0267 streams (the two crystal forms are in every run; each set keeps its own form) 82,388 subset: runs 256-267 (47 of about 220 stream files; ~570 GB of ~3 TB) ~570 GB
euxfel83 euxfel_83/raw/: the 30 Cheetah hit files XFEL-r0243-c00.cxi to XFEL-r0292-c00.cxi (runs 243-254, 270-276, 278-281 and 286-292, the r<run>-blac/XFEL-r<run>-c00.cxi of the entry's cheetah-calib-mar-v1/hdf5/ tree on the NERSC archive); euxfel_83/ref/agipd_mar18_v10.geom from the entry 14,445 all (the deposited stream's 14,445) 25 GB
euxfel152 euxfel_152/raw/: all 221 Cheetah hit files XFEL2042-r0099-c00.cxi to XFEL2042-r0420-c00.cxi of XFEL2042-runs-cxi.tar (extracted with range requests on the tar members); euxfel_152/ref/kdo_embedded.geom, the geometry embedded in the header of the entry's KDO8PS.stream 33,528 all the hit files of the entry's tar; the deposited stream has 37,006 frames 53 GB
alvra103_int16 swissfel_103/conv/run*_int16.h5: int16 copies (see below) of 59 runs of the entry's Raw_Files/<day>/a2a/run<N>.JF06T32V01.h5, run9 to run67 of 2018-08-07 and run1 to run3 of 2018-08-08 (named run0808_1 to _3); swissfel_103/ref/jf-16M.geom and the pixel mask swissfel_103/ref/run47.JF06T32V01.mask.h5 from the entry; the set reads swissfel_103/ref/jf-16M_mask.geom, a copy with mask_file pointed at that mask (see below) 53,734 subset: 59 runs of two days; the deposition indexed 199,136 of 2,251,787 images 172 GB of copies
alvra213_int16 swissfel_213/conv/march_dark_MbCO_II-March-06/data/dark/run_*_int16.h5: int16 copies (see below) of swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5, from march/march_dark_MbCO_II-March-06.tar.gz (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); swissfel_213/ref/mbco_march.geom from the JUNGFRAU geometry template of 2019-MARCH-SwissFEL-MbCO-supplement.tar.gz; the pixel mask of run 243 (data/pixel_mask/ in the same tarball) is used for all runs, through swissfel_213/ref/mbco_march_mask.geom, a copy with mask_file pointed at it (see below) 28,428 all 23 dark runs with frames of the March 2019 series 46 GB of copies (430 GB of source)

The masks of the SwissFEL geometry files. rugnux takes the pixel masks and bad regions a CrystFEL geometry file names. The SwissFEL files name their mask with an absolute path on the facility's disk, so each set reads a copy whose mask_file is relative to the copy's own directory (a relative mask file is looked for in the working directory first, as CrystFEL does, then beside the geometry file): swissfel_103/ref/jf-16M_mask.geom is jf-16M.geom with mask_file = run47.JF06T32V01.mask.h5, and swissfel_213/ref/mbco_march_mask.geom is mbco_march.geom with mask_file = ../raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5. Only that line differs.

The _int16 sets. The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV, 68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The _int16 sets are copies with the same layout in which /data/data holds whole photons (keV divided by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller. The battery runs only the copies: they merge to the same numbers as the float32 originals (rugnux measures the gain from the frames), and the originals are kept only as the source of the conversion and as a test of the format. Make them with tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV> (h5py and hdf5plugin), into swissfel_103/conv/ (4.57 keV) and swissfel_213/conv/ (12.037 keV, keeping the tarball's directory tree), named <input>_int16.h5. One MbCO dark run (run_000238) holds no frames and has no copy.