From 59332d46949a83b90ba0c327365ce1bb607662e7 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 20 Oct 2026 23:04:43 +0200 Subject: [PATCH] tools/battery: the still battery - serial (XFEL stills) datasets against their depositions still_battery.py runs the public serial datasets of still_manifest.json with their deposition's cell and space group, refines the deposited model against each merge with dimple and the deposition's own free set, and reports per set: facility, beamline, detector, PDB, images, indexing rate, R_split, our R-free beside the deposition's published R-free, the wall time of the rugnux command, ms per image, the input's I/O floor (bytes over 5 GB/s) and how much of the input was already in the page cache. It shares battery.py's site config (a "still" section), verbs (run, compare, report, list, keep) and run-directory layout; run directories carry "still-" in their label. The datasets (CXIDB 243, 237, 236, 86, 90, 222, 242, 83, 152, 103, 213; dark and ground-state entries only) are listed with their verified DataCite DOIs in docs/EXTERNAL_TEST_DATA_STILLS.md, linked from EXTERNAL_TEST_DATA.md; CXIDB is credited in ACKNOWLEDGEMENT.md. The SwissFEL sets also run as int16 bitshuffle/LZ4 copies, made with still/float32_to_int16.py. The stray scripts in the repository root move under tools/battery: rugnux_stills_ab.py and its dataset template to still/, rugnux_anomalous.py beside battery.py; docs/TESTS.md follows. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu --- docs/ACKNOWLEDGEMENT.md | 6 + docs/EXTERNAL_TEST_DATA.md | 3 + docs/EXTERNAL_TEST_DATA_STILLS.md | 70 +++ docs/TESTS.md | 17 +- docs/index.rst | 1 + tools/battery/README.md | 52 +++ .../battery/rugnux_anomalous.py | 0 tools/battery/site.example.json | 1 + tools/battery/still/float32_to_int16.py | 47 ++ .../battery/still/rugnux_stills_ab.py | 5 +- .../battery/still/stills_datasets.tsv | 0 tools/battery/still_battery.py | 422 ++++++++++++++++++ tools/battery/still_manifest.json | 17 + 13 files changed, 634 insertions(+), 7 deletions(-) create mode 100644 docs/EXTERNAL_TEST_DATA_STILLS.md rename rugnux_anomalous.py => tools/battery/rugnux_anomalous.py (100%) create mode 100644 tools/battery/still/float32_to_int16.py rename rugnux_stills_ab.py => tools/battery/still/rugnux_stills_ab.py (99%) rename stills_datasets.tsv => tools/battery/still/stills_datasets.tsv (100%) create mode 100644 tools/battery/still_battery.py create mode 100644 tools/battery/still_manifest.json diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 45afd765c..6d17f558a 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -598,6 +598,12 @@ by the groups that collected them. European Organization for Nuclear Research an deposits are described in IUCrData Raw Data Letters; the letters are cited on the [EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md) page, beside the datasets they describe. +**[CXIDB](https://cxidb.org/)**, the Coherent X-ray Imaging Data Bank, is the source of the serial +(XFEL stills) datasets of the still battery, listed with their DOIs on +[EXTERNAL_TEST_DATA_STILLS](EXTERNAL_TEST_DATA_STILLS.md). F. R. N. C. Maia, "The Coherent X-ray +Imaging Data Bank" (2012), Nat. Methods 9, 854-855 +[doi:10.1038/nmeth.2110](https://doi.org/10.1038/nmeth.2110). + **[MXRDR](https://mxrdr.icm.edu.pl/)**, the Macromolecular Xtallography Raw Data Repository (ICM, University of Warsaw), releases its data under CC0 and asks that the DOI of the dataset be cited. diff --git a/docs/EXTERNAL_TEST_DATA.md b/docs/EXTERNAL_TEST_DATA.md index 770a08db5..68a410c86 100644 --- a/docs/EXTERNAL_TEST_DATA.md +++ b/docs/EXTERNAL_TEST_DATA.md @@ -12,6 +12,9 @@ exactly this kind of reuse, and this page is where we credit them. the table below; the repositories themselves are cited in [ACKNOWLEDGEMENT](ACKNOWLEDGEMENT.md). +This page lists the rotation datasets. The serial (XFEL stills) datasets of the still battery are on +[External test data: serial crystallography](EXTERNAL_TEST_DATA_STILLS.md). + ## Where the values come from - **Source** is the repository we downloaded from and that repository's own citable DOI for diff --git a/docs/EXTERNAL_TEST_DATA_STILLS.md b/docs/EXTERNAL_TEST_DATA_STILLS.md new file mode 100644 index 000000000..583765bf0 --- /dev/null +++ b/docs/EXTERNAL_TEST_DATA_STILLS.md @@ -0,0 +1,70 @@ +# External test data: serial crystallography (XFEL stills) + +The serial datasets of the still battery (`tools/battery/still_battery.py`, manifest +`tools/battery/still_manifest.json`). Like the rotation sets on +[External test data](EXTERNAL_TEST_DATA.md), they were collected by other people, on detectors and in +file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them +come from the [Coherent X-ray Imaging Data Bank](https://cxidb.org) (CXIDB). + +**None of these data were collected by us.** If you use any of them, cite the dataset DOI below and +the paper of the entry. + +## Where the values come from + +- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down. +- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from + the RCSB data API. Every entry is a dark or ground state refined against observed data; entries + refined against extrapolated structure factors (the time points of a pump-probe series) are not + used, because a merge of the measured images cannot be compared with them. +- **Images** are the frames in the files we use, counted in the files themselves. + +## Datasets + +| Set | CXIDB, dataset DOI | Facility / beamline | Detector, format | PDB | Space group, dmin (Å) | Paper | +|---|---|---|---|---|---|---| +| `sacla243` | [243](https://cxidb.org/id-243.html), [10.11577/17003933](https://doi.org/10.11577/17003933) | SACLA BL2 | MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 | [31JT](https://www.rcsb.org/structure/31JT) | P 2 21 21, 1.63 | Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026) | +| `sacla237` | [237](https://cxidb.org/id-237.html), [10.11577/2999572](https://doi.org/10.11577/2999572) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9S06](https://www.rcsb.org/structure/9S06) | P 21 21 21, 2.05 | Rios-Santacruz et al., Nature (2026) | +| `sacla236` | [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9V3D](https://www.rcsb.org/structure/9V3D) | P 43 21 2, 1.68 | Kang et al., J. Appl. Cryst. (2026) | +| `pal86` | [86](https://cxidb.org/id-86.html), [10.11577/1502065](https://doi.org/10.11577/1502065) | PAL-XFEL NCI | Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI | [6IG7](https://www.rcsb.org/structure/6IG7) | P 61 2 2, 1.80 | Park et al., Sci. Rep. (2019) | +| `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) | +| `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) | +| `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) | +| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 x (512 x 128), 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) | +| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) | +| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) | +| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) | + +## The files we use + +Paths are relative to the still data root of the site config (`tools/battery/README.md`). CXIDB +serves `https://cxidb.org/data//...`; most of it accepts HTTP range requests, so a +broken download resumes with `curl -C -`. + +| Set | Files | Images | Size | +|---|---|---|---| +| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | 198 GB | +| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` (the dark reference; the time points are not used) | 10,792 | 48 GB | +| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | 67 GB | +| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | 111 GB | +| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB | +| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB | +| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB | +| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | 4.7 GB | +| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | 5.7 GB | +| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` from the entry | ~11,000 | ~750 GB | +| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz` | ~38,000 | ~430 GB | + +**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV, +68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The +`_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided +by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them +with `--gain 1` and merges them to the same numbers. Make them with +`tools/battery/still/float32_to_int16.py ` (h5py and hdf5plugin), +into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV), named `_int16.h5`. + +## Not in the battery + +- **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time + point was refined against extrapolated structure factors. +- **Raw JUNGFRAU data** (ADC values with gain bits that need pedestal and gain correction, e.g. LCLS + CXIDB 189 and 214): only converted detector data are read. diff --git a/docs/TESTS.md b/docs/TESTS.md index f1e04f93b..8fc489f8e 100644 --- a/docs/TESTS.md +++ b/docs/TESTS.md @@ -43,9 +43,14 @@ pipeline is exercised on, and the DOI to cite for each, are listed in and compare protocol and the report are described in `tools/battery/README.md`. Each set runs once, with one command per arm; [Measuring Rugnux](RUGNUX_BENCHMARKING.md) explains the protocol, the REFMAC check and the timing benchmark, and how to take timings that compare. -* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial - dataset by what it does to the merge. Takes its dataset list from outside the repository. -* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below. +* `tools/battery/still_battery.py` - the still battery: public serial (XFEL stills) datasets + run with their deposition's cell and space group, each merge refined against the deposited model + and reported beside the deposition's R-free, with rugnux's wall time and the input's I/O floor. + See "The still battery" in `tools/battery/README.md`. +* `tools/battery/still/rugnux_stills_ab.py` - A/B comparison of argument sets on a fixed stills + subset: what one change does to the merge, shell by shell. Takes its dataset list from outside + the repository. +* `tools/battery/rugnux_anomalous.py` - the anomalous-peak-height arbiter, below. ### The anomalous-peak-height arbiter @@ -64,16 +69,16 @@ statistics we normally reach for: units of the map's own sigma, so a uniform intensity rescale cancels exactly, and it is referenced to the structure rather than to another program's partiality model. -`rugnux_anomalous.py` measures it: `shelxc` + `anode -a` (CCP4) on each arm's merged reflections, +`tools/battery/rugnux_anomalous.py` measures it: `shelxc` + `anode -a` (CCP4) on each arm's merged reflections, against a model that is placed **once** and then held fixed. It reports, per dataset, the mean site height and the off-site noise floor, and, between arms, the **paired per-site** change. ``` # compare two arms (each a directory of /.hkl + .mtz) -./rugnux_anomalous.py --config .json base= test= +tools/battery/rugnux_anomalous.py --config
.json base= test= # a parameter scan: numeric labels turn the arms into a curve with a per-dataset optimum -./rugnux_anomalous.py --config
.json \ +tools/battery/rugnux_anomalous.py --config
.json \ 0.85='/{name}/s0p85.hkl' 1.00='/{name}/s1.hkl' 1.20='/{name}/s1p2.hkl' ``` diff --git a/docs/index.rst b/docs/index.rst index 48a3d1cac..9c2a3dc52 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -83,6 +83,7 @@ Jungfraujoch is distributed under the GPLv3 license. ACKNOWLEDGEMENT EXTERNAL_TEST_DATA + EXTERNAL_TEST_DATA_STILLS LICENSE THIRD_PARTY_NOTICES VERSIONING diff --git a/tools/battery/README.md b/tools/battery/README.md index 6054d3d35..fcbdc0bee 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -934,3 +934,55 @@ sets that ran with other processes on the GPU (`gpu_others` > 0) are drawn as th segment of their bar rather than dropped; `--uncontended RUN` names a run of the same sets with the GPU to itself and `summary.txt` then reports their median slowdown against it. + +## The still battery + +`still_battery.py` is the serial (XFEL stills) counterpart of the rotation battery, for the public +datasets on [External test data: serial crystallography](../../docs/EXTERNAL_TEST_DATA_STILLS.md) +(manifest `still_manifest.json`). It shares the site config, the verbs and the run-directory layout +with `battery.py`, and reports something different: for stills the cell and space group are given, the +way a serial user runs (`-C`/`-S`), so there is no lattice or symmetry verdict to score. Each set runs +once and its row says what the merge is worth and what it cost: + +| column | meaning | +|---|---| +| facility, beamline, detector, dataset, PDB | from the manifest | +| images | frames in the input files (rugnux's `IMAGES_PROCESSED`); images (deposition) the deposition's own count, where it is known | +| indexed %, d_min, R_split | rugnux's report (`INDEXING_RATE`, `INCLUDE_RESOLUTION_RANGE`, `R_SPLIT`: whole frames go to one half, as in CrystFEL) | +| R_free rugnux | `dimple` of the deposited model against our merge, reindexed into the deposition's space group, with the deposition's own free set, at the deposition's resolution | +| R_free deposited | the entry's published R-free (dark and ground-state entries only) | +| time, ms/image | wall clock of the rugnux command, which ends with the model validation | +| I/O floor | the input's bytes over `--io-gbps` (default 5 GB/s): the time reading alone would take from an SSD | +| page cache | share of the input already in memory when the run started (`fincore`): cold, warm or partial | + +The I/O floor and the page-cache state are there because several of these formats are uncompressed +(XTC, float32 SwissFEL HDF5): a run whose time is at its floor measured the disk, not rugnux, and a +warm run of a set smaller than the RAM measured neither. For honest timings drop the page cache +before the run (`sync; echo 3 | sudo tee /proc/sys/vm/drop_caches`) and check that the column says +cold. + +``` +still_battery.py run --rugnux build/rugnux/rugnux [--only pal90,sacla236 | --tier smoke] [--threads N] +still_battery.py compare RUN_A RUN_B +still_battery.py report RUN [--out DIR] +still_battery.py list +still_battery.py keep RUN --as workstation|server +``` + +A run writes `/__still-