Merge still-battery into rc178: the still battery and the external stills test data

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
2026-10-21 00:10:43 +02:00
commit 2bd3616388
@@ -598,6 +598,12 @@ by the groups that collected them. European Organization for Nuclear Research an
 deposits are described in IUCrData Raw Data Letters; the letters are cited on the
 [EXTERNAL_TEST_DATA](EXTERNAL_TEST_DATA.md) page, beside the datasets they describe.

+**[CXIDB](https://cxidb.org/)**, the Coherent X-ray Imaging Data Bank, is the source of the serial
+(XFEL stills) datasets of the still battery, listed with their DOIs on
+[EXTERNAL_TEST_DATA_STILLS](EXTERNAL_TEST_DATA_STILLS.md). F. R. N. C. Maia, "The Coherent X-ray
+Imaging Data Bank" (2012), Nat. Methods 9, 854-855
+[doi:10.1038/nmeth.2110](https://doi.org/10.1038/nmeth.2110).
+
 **[MXRDR](https://mxrdr.icm.edu.pl/)**, the Macromolecular Xtallography Raw Data Repository
 (ICM, University of Warsaw), releases its data under CC0 and asks that the DOI of the dataset be
 cited.
@@ -12,6 +12,9 @@ exactly this kind of reuse, and this page is where we credit them.
 the table below; the repositories themselves are cited in
 [ACKNOWLEDGEMENT](ACKNOWLEDGEMENT.md).

+This page lists the rotation datasets. The serial (XFEL stills) datasets of the still battery are on
+[External test data: serial crystallography](EXTERNAL_TEST_DATA_STILLS.md).
+
 ## Where the values come from

 - **Source** is the repository we downloaded from and that repository's own citable DOI for
@@ -0,0 +1,70 @@
+# External test data: serial crystallography (XFEL stills)
+
+The serial datasets of the still battery (`tools/battery/still_battery.py`, manifest
+`tools/battery/still_manifest.json`). Like the rotation sets on
+[External test data](EXTERNAL_TEST_DATA.md), they were collected by other people, on detectors and in
+file formats we do not produce ourselves, and published for exactly this kind of reuse. All of them
+come from the [Coherent X-ray Imaging Data Bank](https://cxidb.org) (CXIDB).
+
+**None of these data were collected by us.** If you use any of them, cite the dataset DOI below and
+the paper of the entry.
+
+## Where the values come from
+
+- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
+- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from
+  the RCSB data API. Every entry is a dark or ground state refined against observed data; entries
+  refined against extrapolated structure factors (the time points of a pump-probe series) are not
+  used, because a merge of the measured images cannot be compared with them.
+- **Images** are the frames in the files we use, counted in the files themselves.
+
+## Datasets
+
+| Set | CXIDB, dataset DOI | Facility / beamline | Detector, format | PDB | Space group, d<sub>min</sub> (Å) | Paper |
+|---|---|---|---|---|---|---|
+| `sacla243` | [243](https://cxidb.org/id-243.html), [10.11577/17003933](https://doi.org/10.11577/17003933) | SACLA BL2 | MPCCD, 8 panels of 1024 x 512, 50 µm; Cheetah-SACLA HDF5 | [31JT](https://www.rcsb.org/structure/31JT) | P 2 21 21, 1.63 | Bibi, De Vrieze et al., J. Am. Chem. Soc. (2026) |
+| `sacla237` | [237](https://cxidb.org/id-237.html), [10.11577/2999572](https://doi.org/10.11577/2999572) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9S06](https://www.rcsb.org/structure/9S06) | P 21 21 21, 2.05 | Rios-Santacruz et al., Nature (2026) |
+| `sacla236` | [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | SACLA BL2 | MPCCD; Cheetah-SACLA HDF5 | [9V3D](https://www.rcsb.org/structure/9V3D) | P 43 21 2, 1.68 | Kang et al., J. Appl. Cryst. (2026) |
+| `pal86` | [86](https://cxidb.org/id-86.html), [10.11577/1502065](https://doi.org/10.11577/1502065) | PAL-XFEL NCI | Rayonix MX225-HS, 1440 x 1440, 156 µm; Cheetah CXI | [6IG7](https://www.rcsb.org/structure/6IG7) | P 61 2 2, 1.80 | Park et al., Sci. Rep. (2019) |
+| `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) |
+| `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) |
+| `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) |
+| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 x (512 x 128), 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
+| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
+| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) |
+| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) |
+
+## The files we use
+
+Paths are relative to the still data root of the site config (`tools/battery/README.md`). CXIDB
+serves `https://cxidb.org/data/<entry>/...`; most of it accepts HTTP range requests, so a
+broken download resumes with `curl -C -`.
+
+| Set | Files | Images | Size |
+|---|---|---|---|
+| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | 198 GB |
+| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` (the dark reference; the time points are not used) | 10,792 | 48 GB |
+| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | 67 GB |
+| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | 111 GB |
+| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB |
+| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB |
+| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB |
+| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | 4.7 GB |
+| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | 5.7 GB |
+| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` from the entry | ~11,000 | ~750 GB |
+| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz` | ~38,000 | ~430 GB |
+
+**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
+68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
+`_int16` sets are copies with the same layout in which `/data/data` holds whole photons (keV divided
+by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them
+with `--gain 1` and merges them to the same numbers. Make them with
+`tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV>` (h5py and hdf5plugin),
+into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV), named `<input>_int16.h5`.
+
+## Not in the battery
+
+- **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time
+  point was refined against extrapolated structure factors.
+- **Raw JUNGFRAU data** (ADC values with gain bits that need pedestal and gain correction, e.g. LCLS
+  CXIDB 189 and 214): only converted detector data are read.
@@ -43,9 +43,14 @@ pipeline is exercised on, and the DOI to cite for each, are listed in
  and compare protocol and the report are described in `tools/battery/README.md`. Each set runs
  once, with one command per arm; [Measuring Rugnux](RUGNUX_BENCHMARKING.md) explains the protocol,
  the REFMAC check and the timing benchmark, and how to take timings that compare.
-* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial
-  dataset by what it does to the merge. Takes its dataset list from outside the repository.
-* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.
+* `tools/battery/still_battery.py` - the still battery: public serial (XFEL stills) datasets
+  run with their deposition's cell and space group, each merge refined against the deposited model
+  and reported beside the deposition's R-free, with rugnux's wall time and the input's I/O floor.
+  See "The still battery" in `tools/battery/README.md`.
+* `tools/battery/still/rugnux_stills_ab.py` - A/B comparison of argument sets on a fixed stills
+  subset: what one change does to the merge, shell by shell. Takes its dataset list from outside
+  the repository.
+* `tools/battery/rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.

 ### The anomalous-peak-height arbiter

@@ -64,16 +69,16 @@ statistics we normally reach for:
 units of the map's own sigma, so a uniform intensity rescale cancels exactly, and it is referenced
 to the structure rather than to another program's partiality model.

-`rugnux_anomalous.py` measures it: `shelxc` + `anode -a` (CCP4) on each arm's merged reflections,
+`tools/battery/rugnux_anomalous.py` measures it: `shelxc` + `anode -a` (CCP4) on each arm's merged reflections,
 against a model that is placed **once** and then held fixed. It reports, per dataset, the mean site
 height and the off-site noise floor, and, between arms, the **paired per-site** change.

 ```
 # compare two arms (each a directory of <id>/<id>.hkl + .mtz)
-./rugnux_anomalous.py --config <table>.json  base=<dir-A>  test=<dir-B>
+tools/battery/rugnux_anomalous.py --config <table>.json  base=<dir-A>  test=<dir-B>

 # a parameter scan: numeric labels turn the arms into a curve with a per-dataset optimum
-./rugnux_anomalous.py --config <table>.json \
+tools/battery/rugnux_anomalous.py --config <table>.json \
    0.85='<scan>/{name}/s0p85.hkl' 1.00='<scan>/{name}/s1.hkl' 1.20='<scan>/{name}/s1p2.hkl'
 ```

@@ -83,6 +83,7 @@ Jungfraujoch is distributed under the GPLv3 license.

   ACKNOWLEDGEMENT
   EXTERNAL_TEST_DATA
+   EXTERNAL_TEST_DATA_STILLS
   LICENSE
   THIRD_PARTY_NOTICES
   VERSIONING
@@ -934,3 +934,55 @@ sets that ran with other processes on the GPU (`gpu_others` > 0) are drawn as th
 segment of their bar rather than dropped; `--uncontended RUN` names a run of the same sets with the
 GPU to itself and `summary.txt` then reports their median slowdown against it.

+
+## The still battery
+
+`still_battery.py` is the serial (XFEL stills) counterpart of the rotation battery, for the public
+datasets on [External test data: serial crystallography](../../docs/EXTERNAL_TEST_DATA_STILLS.md)
+(manifest `still_manifest.json`). It shares the site config, the verbs and the run-directory layout
+with `battery.py`, and reports something different: for stills the cell and space group are given, the
+way a serial user runs (`-C`/`-S`), so there is no lattice or symmetry verdict to score. Each set runs
+once and its row says what the merge is worth and what it cost:
+
+| column | meaning |
+|---|---|
+| facility, beamline, detector, dataset, PDB | from the manifest |
+| images | frames in the input files (rugnux's `IMAGES_PROCESSED`); images (deposition) the deposition's own count, where it is known |
+| indexed %, d_min, R_split | rugnux's report (`INDEXING_RATE`, `INCLUDE_RESOLUTION_RANGE`, `R_SPLIT`: whole frames go to one half, as in CrystFEL) |
+| R_free rugnux | `dimple` of the deposited model against our merge, reindexed into the deposition's space group, with the deposition's own free set, at the deposition's resolution |
+| R_free deposited | the entry's published R-free (dark and ground-state entries only) |
+| time, ms/image | wall clock of the rugnux command, which ends with the model validation |
+| I/O floor | the input's bytes over `--io-gbps` (default 5 GB/s): the time reading alone would take from an SSD |
+| page cache | share of the input already in memory when the run started (`fincore`): cold, warm or partial |
+
+The I/O floor and the page-cache state are there because several of these formats are uncompressed
+(XTC, float32 SwissFEL HDF5): a run whose time is at its floor measured the disk, not rugnux, and a
+warm run of a set smaller than the RAM measured neither. For honest timings drop the page cache
+before the run (`sync; echo 3 | sudo tee /proc/sys/vm/drop_caches`) and check that the column says
+cold.
+
+```
+still_battery.py run --rugnux build/rugnux/rugnux [--only pal90,sacla236 | --tier smoke] [--threads N]
+still_battery.py compare RUN_A RUN_B
+still_battery.py report RUN [--out DIR]
+still_battery.py list
+still_battery.py keep RUN --as workstation|server
+```
+
+A run writes `<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_still-<label>/` with `manifest.json`,
+`results.json`, `report.md` and `work/<set>/` (the rugnux outputs and the dimple run), made
+read-only when done; `keep` copies the three files into `results/<role>/<run>/`, like a rotation run.
+`--tier smoke` runs the sets marked so in the manifest on their first few thousand frames. Data
+paths come from the `"still"` section of the site config:
+
+```json
+"still": {"manifest": "still_manifest.json", "data_root": "/ssd/data/still_test", "private_manifest": null}
+```
+
+`private_manifest` names a local-only manifest of user data (`--private`); such a run gets its own
+run directory and `report_PRIVATE.md`, and `keep` refuses it. Needs `gemmi` (Python), CCP4's
+`dimple` and `reindex`, and `fincore` (util-linux) for the page-cache column.
+
+Data preparation for the `_int16` sets is `still/float32_to_int16.py` (see the stills data page).
+`still/rugnux_stills_ab.py` is the older A/B harness: it compares two argument sets on a fixed
+subset of one dataset, shell by shell, and takes its dataset list from outside the repository.
@@ -9,6 +9,7 @@
 "pdb_cache": "/path/to/battery/pdb_cache",
 "cod_cache": "/path/to/battery/cod_cache",
 "shelxl": null,
+ "still": {"manifest": "still_manifest.json", "data_root": "/path/to/battery/still", "private_manifest": null},
 "arms": {
  "open":    {"manifest": "open.json",    "reference": "deposition", "data_root": "/path/to/battery/open"},
  "inhouse": {"manifest": "inhouse.json", "reference": "xds",        "data_root": "/path/to/battery/inhouse"}
@@ -0,0 +1,47 @@
+"""Data preparation for the still battery (not product code): copy a SwissFEL converted JUNGFRAU file, replacing /data/data (float32 keV) by int16 photon counts,
+bitshuffle/LZ4, one chunk per frame. Everything else is copied unchanged. NaN -> 0 (what rugnux's reader makes of it).
+usage: float32_to_int16.py <in.h5> <out.h5> <photon_energy_keV>
+Needs h5py and hdf5plugin. The copies are the alvra*_int16 sets of still_manifest.json; read them with --gain 1.
+See docs/EXTERNAL_TEST_DATA_STILLS.md."""
+import os, sys, time, h5py, hdf5plugin, numpy as np
+
+src, dst, e_kev = sys.argv[1], sys.argv[2], float(sys.argv[3])
+t0 = time.time()
+with h5py.File(src, 'r') as fi, h5py.File(dst + '.part', 'w') as fo:
+    for k, v in fi.attrs.items():
+        fo.attrs[k] = v
+    def copy(name, obj):
+        if name == 'data/data':
+            return
+        if isinstance(obj, h5py.Group):
+            g = fo.require_group(name)
+            for k, v in obj.attrs.items():
+                g.attrs[k] = v
+        else:
+            fo.copy(obj, name if '/' not in name else fo.require_group(name.rsplit('/', 1)[0]), name=name.rsplit('/', 1)[-1])
+    fi.visititems(copy)
+    d = fi['data/data']
+    n, h, w = d.shape
+    out = fo.create_dataset('data/data', shape=(n, h, w), dtype='int16', chunks=(1, h, w),
+                            **hdf5plugin.Bitshuffle(nelems=0, cname='lz4'))
+    for k, v in d.attrs.items():
+        out.attrs[k] = v
+    out.attrs['photon_energy_keV_used_for_conversion'] = e_kev
+    out.attrs['conversion'] = 'int16 photons = round(float32 keV / photon_energy_keV), NaN -> 0, clipped to int16; original: ' + src
+    nan = clipped = 0
+    hist = np.zeros(4, dtype=np.int64)  # <0, 0, 1, >=2 photons
+    for i in range(n):
+        x = d[i]
+        bad = np.isnan(x)
+        nan += int(bad.sum())
+        p = np.rint(np.where(bad, 0, x) / e_kev)
+        clipped += int(((p > 32767) | (p < -32768)).sum())
+        p = np.clip(p, -32768, 32767).astype(np.int16)
+        if i % 50 == 0:
+            hist += [(p < 0).sum(), (p == 0).sum(), (p == 1).sum(), (p >= 2).sum()]
+        out[i] = p
+os.rename(dst + '.part', dst)
+s_in, s_out = os.path.getsize(src), os.path.getsize(dst)
+print(f'{os.path.basename(src)} frames {n} in {s_in/1e9:.2f} GB out {s_out/1e9:.3f} GB ratio {s_in/s_out:.1f} '
+      f'MB/frame {s_in/n/1e6:.2f}->{s_out/n/1e6:.3f} time {time.time()-t0:.0f} s nan {nan} clipped {clipped} '
+      f'sampled photon hist <0/0/1/>=2 {hist.tolist()}', flush=True)
@@ -1,6 +1,9 @@
 #!/usr/bin/env python3
 """
-rugnux_stills_ab.py -- the stills analogue of the rotation battery (tools/battery/).
+rugnux_stills_ab.py -- A/B comparison of two (or more) rugnux argument sets on a fixed stills subset.
+
+The still battery (tools/battery/still_battery.py) reports each public serial dataset against its
+deposition; this script answers the narrower question of whether one change moved the merge.

 The battery scores a ROTATION change against a reference: one crystal per directory, one
 deposition or CORRECT.LP to compare with, space-group agreement as the headline. None of