EXTERNAL_TEST_DATA_STILLS.md: images used vs the entry for every set; drop "Not in the battery"

Each set's row now says how many images we use and, where that is a subset, what the entry or its
deposition holds (SACLA 237 ~1.8% of dark_ref, LCLS 222 one of four runs, LCLS 242 runs 256-267, both
AGIPD sets and A2A against their deposited frame counts). The section listing what is not used is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-21 11:44:25 +02:00
1 parent e4eaeba874
commit dafb112d93
1 file changed
+16 -24
+16 -24
View File
@@ -13,10 +13,9 @@ the paper of the entry.
- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from
the RCSB data API. Every entry is a dark or ground state refined against observed data; entries
refined against extrapolated structure factors (the time points of a pump-probe series) are not
used, because a merge of the measured images cannot be compared with them.
- **Images** are the frames in the files we use, counted in the files themselves.
the RCSB data API. Every entry is a dark or ground state refined against observed data.
- **Images used** are the frames in the files we use, counted in the files themselves. Where that is
only part of the entry, the table says so and gives what the entry or the deposition holds.
## Datasets
@@ -40,19 +39,19 @@ Paths are relative to the still data root of the site config (`tools/battery/REA
serves `https://cxidb.org/data/<entry>/...`; most of it accepts HTTP range requests, so a
broken download resumes with `curl -C -`.
| Set | Files | Images | Size |
|---|---|---|---|
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | 198 GB |
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` (the dark reference; the time points are not used) | 10,792 | 48 GB |
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | 67 GB |
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | 111 GB |
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB |
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB |
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB |
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | 4.7 GB (1.3 GB converted) |
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | 5.7 GB (1.1 GB converted) |
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | ~750 GB |
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | ~430 GB |
| Set | Files | Images used | Of the entry | Size |
|---|---|---|---|---|
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | all (the deposition's 52,151) | 198 GB |
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` | 10,792 | **subset**: 18 of about 1,000 `dark_ref` files, about 1.8% of ~600,000 hits | 48 GB |
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | all (the deposition's 16,083) | 67 GB |
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | all but file 1 (24 of 25 files) | 111 GB |
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | all (both files) | 25 GB |
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | **subset**: run 165 of the runs 165-168 behind 8TDQ | 72 GB |
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB |
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB (1.3 GB converted) |
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB (1.1 GB converted) |
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB |
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB |
**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
@@ -84,10 +83,3 @@ geometry `conv/agipd_int16.geom`; read them with `--gain 1`. Neither file states
The files' own photon energy (9.6 keV) and distance are not the experiment's: the manifest gives
`--wavelength`, and the geometry the camera length. In the CXIDB 152 files the dark offset is not removed
(the zero-photon peak is at about 2 photons), which is left as background.
## Not in the battery
- **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time
point was refined against extrapolated structure factors.
- **Raw JUNGFRAU data** (ADC values with gain bits that need pedestal and gain correction, e.g. LCLS
CXIDB 189 and 214): only converted detector data are read.