* Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner. * Rugnux: More reliable rotation processing and indexing, including small molecules and supercells. * Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models. * jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run. * jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes. * jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored. * jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line. * jfjoch_viewer: Build and display fixes; the README is rewritten. Reviewed-on: #95 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
11 KiB
External test data: serial crystallography (XFEL stills)
The serial datasets of the still battery (tools/battery/still_battery.py, manifest
tools/battery/still_manifest.json). Like the rotation sets on
External test data, they were collected by other people, on detectors and in
file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them
come from the Coherent X-ray Imaging Data Bank (CXIDB). They are the eleven
datasets of the Rugnux paper's Extended Data Table 2, which gives the published results on them.
None of these data were collected by us. If you use any of them, cite the dataset DOI below and the paper of the entry.
Where the values come from
- Dataset DOI: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
- Space group, cell, resolution and R-free are the values deposited with the PDB entry, read from the RCSB data API. Every entry is a dark or ground state refined against observed data.
- Images used are the frames in the files we use, counted in the files themselves. Where that is only part of the entry, the table says so and gives what the entry or the deposition holds.
Datasets
| Set | CXIDB, dataset DOI | Facility / beamline | Detector, format | PDB | Space group, dmin (Å) | Paper |
|---|---|---|---|---|---|---|
sacla243 |
243, 10.11577/17003933 | SACLA BL2 | MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 | 31JT | P 2 21 21, 1.63 | Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026) |
sacla237 |
237, 10.11577/2999572 | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | 9S06 | P 21 21 21, 2.05 | Rios-Santacruz et al., Nature (2026) |
sacla236 |
236, 10.11577/3014306 | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | 9V3D | P 43 21 2, 1.68 | Kang et al., J. Appl. Cryst. (2026) |
pal86 |
86, 10.11577/1502065 | PAL-XFEL NCI | Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI | 6IG7 | P 61 2 2, 1.80 | Park et al., Sci. Rep. (2019) |
pal90 |
90, 10.11577/1511590 | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | 6IRK | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) |
lcls222 |
222, 10.11577/2204603 | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | 8TDQ | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) |
lcls242_s, lcls242_t |
242, 10.11577/3374214 | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | 9O4S (small cell), 9O4T (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) |
euxfel83 |
83, 10.11577/1464102 | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI | 6GTH | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
euxfel152 |
152, 10.11577/1660970 | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | 6U57 | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
alvra103_int16 |
103, 10.11577/1650020 | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | 6S0L | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) |
alvra213_int16 |
213, 10.11577/2282689 | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | 8BKH (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) |
Every set of the manifest is in the table and every set in the table is in the manifest, with the
same CXIDB entry, PDB entry, space group and dmin; tools/battery/doc_check.py checks
this. The beamline of pal90 is the CXIDB entry's (NCI); the PDB entry 6IRK records CSI.
How the battery runs them
Each set is one rugnux command, the way a serial user runs: the deposited cell and space group given
(-C, -S), the wavelength where the files do not state it, --crystfel-geometry for the AGIPD and
JUNGFRAU sets (the geometry file of the entry; the SACLA, PAL and LCLS files are placed without one),
and the deposited model with --model, so that the report carries its R-free against the merge. The
exact arguments are the args of each set in tools/battery/still_manifest.json; the
stills section of the tutorial explains them.
The files we use
Paths are relative to the still data root of the site config (tools/battery/README.md). CXIDB
serves https://cxidb.org/data/<entry>/...; most of it accepts HTTP range requests, so a
broken download resumes with curl -C -.
| Set | Files | Images used | Of the entry | Size |
|---|---|---|---|---|
sacla243 |
sacla_243/raw_all/run*.h5: all 87 files of 243/Complete_dark/ |
52,151 | all (the deposition's 52,151) | 198 GB |
sacla237 |
sacla_237/raw/: the 18 files run130637-0 to run130642-2 of 237/SACLA_raw/2022Nov/dark_ref/ |
10,792 | subset: 18 of about 1,000 dark_ref files, about 1.8% of ~600,000 hits |
48 GB |
sacla236 |
sacla_236/raw/: the 18 files run157084-0 to run157089-2 of 236/Data/lyso01-apo-bigXtal/ |
16,083 | all (the deposition's 16,083) | 67 GB |
pal86 |
pal_86/raw/, pal_86/raw_more/: CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi (file 1 is truncated on the server and cannot be opened) |
49,193 | all but file 1 (24 of 25 files) | 111 GB |
pal90 |
pal_90/raw/: CXIDB_90-GI_SFX-filter-r00{01..14}-c00.cxi |
79,805 | all 14 filtered files of the entry | 159 GB |
lcls222 |
lcls_222/xtc/, lcls_222/xtc_more/: mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc (run 165, the substrate-bound state of 8TDQ) |
8,534 | subset: run 165 of the runs 165-168 behind 8TDQ | 72 GB |
lcls242_* |
lcls_242/xtc/, lcls_242/xtc_more/: xtc/mfxl1013721-r0256 to r0267 streams (the two crystal forms are in every run; each set keeps its own form) |
82,388 | subset: runs 256-267 (47 of about 220 stream files; ~570 GB of ~3 TB) | ~570 GB |
euxfel83 |
euxfel_83/raw/: the 30 Cheetah hit files XFEL-r0243-c00.cxi to XFEL-r0292-c00.cxi (runs 243-254, 270-276, 278-281 and 286-292, the r<run>-blac/XFEL-r<run>-c00.cxi of the entry's cheetah-calib-mar-v1/hdf5/ tree on the NERSC archive); euxfel_83/ref/agipd_mar18_v10.geom from the entry |
14,445 | all (the deposited stream's 14,445) | 25 GB |
euxfel152 |
euxfel_152/raw/: all 221 Cheetah hit files XFEL2042-r0099-c00.cxi to XFEL2042-r0420-c00.cxi of XFEL2042-runs-cxi.tar (extracted with range requests on the tar members); euxfel_152/ref/kdo_embedded.geom, the geometry embedded in the header of the entry's KDO8PS.stream |
33,528 | all the hit files of the entry's tar; the deposited stream has 37,006 frames | 53 GB |
alvra103_int16 |
swissfel_103/conv/run*_int16.h5: int16 copies (see below) of 59 runs of the entry's Raw_Files/<day>/a2a/run<N>.JF06T32V01.h5, run9 to run67 of 2018-08-07 and run1 to run3 of 2018-08-08 (named run0808_1 to _3); swissfel_103/ref/jf-16M.geom and the pixel mask swissfel_103/ref/run47.JF06T32V01.mask.h5 from the entry; the set reads swissfel_103/ref/jf-16M_mask.geom, a copy with mask_file pointed at that mask (see below) |
53,734 | subset: 59 runs of two days; the deposition indexed 199,136 of 2,251,787 images | 172 GB of copies |
alvra213_int16 |
swissfel_213/conv/march_dark_MbCO_II-March-06/data/dark/run_*_int16.h5: int16 copies (see below) of swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5, from march/march_dark_MbCO_II-March-06.tar.gz (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); swissfel_213/ref/mbco_march.geom from the JUNGFRAU geometry template of 2019-MARCH-SwissFEL-MbCO-supplement.tar.gz; the pixel mask of run 243 (data/pixel_mask/ in the same tarball) is used for all runs, through swissfel_213/ref/mbco_march_mask.geom, a copy with mask_file pointed at it (see below) |
28,428 | all 23 dark runs with frames of the March 2019 series | 46 GB of copies (430 GB of source) |
The masks of the SwissFEL geometry files. rugnux takes the pixel masks and bad regions a CrystFEL
geometry file names. The SwissFEL files name their mask with an absolute path on the facility's disk,
so each set reads a copy whose mask_file is relative to the copy's own directory (a relative mask
file is looked for in the working directory first, as CrystFEL does, then beside the geometry file):
swissfel_103/ref/jf-16M_mask.geom is jf-16M.geom with mask_file = run47.JF06T32V01.mask.h5, and
swissfel_213/ref/mbco_march_mask.geom is mbco_march.geom with
mask_file = ../raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5. Only
that line differs.
The _int16 sets. The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
_int16 sets are copies with the same layout in which /data/data holds whole photons (keV divided
by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller. The battery runs only the
copies: they merge to the same numbers as the float32 originals (rugnux measures the gain from the
frames), and the originals are kept only as the source of the conversion and as a test of the format. Make them with
tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV> (h5py and hdf5plugin),
into swissfel_103/conv/ (4.57 keV) and swissfel_213/conv/ (12.037 keV, keeping the tarball's
directory tree), named <input>_int16.h5. One MbCO dark run (run_000238) holds no frames and has no copy.