diff --git a/docs/EXTERNAL_TEST_DATA_STILLS.md b/docs/EXTERNAL_TEST_DATA_STILLS.md index f6d3747c5..57306a03b 100644 --- a/docs/EXTERNAL_TEST_DATA_STILLS.md +++ b/docs/EXTERNAL_TEST_DATA_STILLS.md @@ -13,10 +13,9 @@ the paper of the entry. - **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down. - **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from - the RCSB data API. Every entry is a dark or ground state refined against observed data; entries - refined against extrapolated structure factors (the time points of a pump-probe series) are not - used, because a merge of the measured images cannot be compared with them. -- **Images** are the frames in the files we use, counted in the files themselves. + the RCSB data API. Every entry is a dark or ground state refined against observed data. +- **Images used** are the frames in the files we use, counted in the files themselves. Where that is + only part of the entry, the table says so and gives what the entry or the deposition holds. ## Datasets @@ -40,19 +39,19 @@ Paths are relative to the still data root of the site config (`tools/battery/REA serves `https://cxidb.org/data//...`; most of it accepts HTTP range requests, so a broken download resumes with `curl -C -`. -| Set | Files | Images | Size | -|---|---|---|---| -| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | 198 GB | -| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` (the dark reference; the time points are not used) | 10,792 | 48 GB | -| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | 67 GB | -| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | 111 GB | -| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB | -| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB | -| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB | -| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | 4.7 GB (1.3 GB converted) | -| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | 5.7 GB (1.1 GB converted) | -| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | ~750 GB | -| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | ~430 GB | +| Set | Files | Images used | Of the entry | Size | +|---|---|---|---|---| +| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | all (the deposition's 52,151) | 198 GB | +| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` | 10,792 | **subset**: 18 of about 1,000 `dark_ref` files, about 1.8% of ~600,000 hits | 48 GB | +| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | all (the deposition's 16,083) | 67 GB | +| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | all but file 1 (24 of 25 files) | 111 GB | +| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | all (both files) | 25 GB | +| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | **subset**: run 165 of the runs 165-168 behind 8TDQ | 72 GB | +| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB | +| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB (1.3 GB converted) | +| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB (1.1 GB converted) | +| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB | +| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB | **The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV, 68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The @@ -84,10 +83,3 @@ geometry `conv/agipd_int16.geom`; read them with `--gain 1`. Neither file states The files' own photon energy (9.6 keV) and distance are not the experiment's: the manifest gives `--wavelength`, and the geometry the camera length. In the CXIDB 152 files the dark offset is not removed (the zero-photon peak is at about 2 photons), which is left as background. - -## Not in the battery - -- **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time - point was refined against extrapolated structure factors. -- **Raw JUNGFRAU data** (ADC values with gain bits that need pedestal and gain correction, e.g. LCLS - CXIDB 189 and 214): only converted detector data are read.