# External test data: serial crystallography (XFEL stills)

The serial datasets of the still battery (`tools/battery/still_battery.py`, manifest
`tools/battery/still_manifest.json`). Like the rotation sets on
[External test data](EXTERNAL_TEST_DATA.md), they were collected by other people, on detectors and in
file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them
come from the [Coherent X-ray Imaging Data Bank](https://cxidb.org) (CXIDB).

**None of these data were collected by us.** If you use any of them, cite the dataset DOI below and
the paper of the entry.

## Where the values come from

- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from
  the RCSB data API. Every entry is a dark or ground state refined against observed data.
- **Images used** are the frames in the files we use, counted in the files themselves. Where that is
  only part of the entry, the table says so and gives what the entry or the deposition holds.

## Datasets

| Set | CXIDB, dataset DOI | Facility / beamline | Detector, format | PDB | Space group, d<sub>min</sub> (Å) | Paper |
|---|---|---|---|---|---|---|
| `sacla243` | [243](https://cxidb.org/id-243.html), [10.11577/17003933](https://doi.org/10.11577/17003933) | SACLA BL2 | MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 | [31JT](https://www.rcsb.org/structure/31JT) | P 2 21 21, 1.63 | Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026) |
| `sacla237` | [237](https://cxidb.org/id-237.html), [10.11577/2999572](https://doi.org/10.11577/2999572) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9S06](https://www.rcsb.org/structure/9S06) | P 21 21 21, 2.05 | Rios-Santacruz et al., Nature (2026) |
| `sacla236` | [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9V3D](https://www.rcsb.org/structure/9V3D) | P 43 21 2, 1.68 | Kang et al., J. Appl. Cryst. (2026) |
| `pal86` | [86](https://cxidb.org/id-86.html), [10.11577/1502065](https://doi.org/10.11577/1502065) | PAL-XFEL NCI | Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI | [6IG7](https://www.rcsb.org/structure/6IG7) | P 61 2 2, 1.80 | Park et al., Sci. Rep. (2019) |
| `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) |
| `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) |
| `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) |
| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) |
| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) |

Every set of the manifest is in the table and every set in the table is in the manifest, with the
same CXIDB entry, PDB entry, space group and d<sub>min</sub>; `tools/battery/doc_check.py` checks
this. The beamline of `pal90` is the CXIDB entry's (NCI); the PDB entry 6IRK records CSI.

## The files we use

Paths are relative to the still data root of the site config (`tools/battery/README.md`). CXIDB
serves `https://cxidb.org/data/<entry>/...`; most of it accepts HTTP range requests, so a
broken download resumes with `curl -C -`.

| Set | Files | Images used | Of the entry | Size |
|---|---|---|---|---|
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | all (the deposition's 52,151) | 198 GB |
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` | 10,792 | **subset**: 18 of about 1,000 `dark_ref` files, about 1.8% of ~600,000 hits | 48 GB |
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | all (the deposition's 16,083) | 67 GB |
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | all but file 1 (24 of 25 files) | 111 GB |
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | all (both files) | 25 GB |
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | **subset**: run 165 of the runs 165-168 behind 8TDQ | 72 GB |
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB |
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB |
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB |
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry; the set reads `swissfel_103/ref/jf-16M_mask.geom`, a copy with `mask_file` pointed at that mask (see below) | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB |
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs, through `swissfel_213/ref/mbco_march_mask.geom`, a copy with `mask_file` pointed at it (see below) | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB |

**The masks of the SwissFEL geometry files.** rugnux takes the pixel masks and bad regions a CrystFEL
geometry file names. The SwissFEL files name their mask with an absolute path on the facility's disk,
so each set reads a copy whose `mask_file` is relative to the copy's own directory (a relative mask
file is looked for in the working directory first, as CrystFEL does, then beside the geometry file):
`swissfel_103/ref/jf-16M_mask.geom` is `jf-16M.geom` with `mask_file = run47.JF06T32V01.mask.h5`, and
`swissfel_213/ref/mbco_march_mask.geom` is `mbco_march.geom` with
`mask_file = ../raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5`. Only
that line differs.

**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
`_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided
by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them
with `--gain 1` and merges them to the same numbers. Make them with
`tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV>` (h5py and hdf5plugin),
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's
directory tree), named `<input>_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy.