From e4eaeba87475e2722da23e7f88f313b1d31ffb92 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 21 Oct 2026 09:21:33 +0200 Subject: [PATCH] AGIPD: converter to one rigid panel per module, int16 photons, mask folded in; still battery reads the converted sets tools/battery/still/agipd_to_int16.py converts an AGIPD 1M Cheetah CXI file into the only form rugnux supports for AGIPD: each module's eight 64-row ASICs placed 66 rows apart (the 2-pixel ASIC gaps inserted), one 526 x 128 panel per module; the per-frame mask and mask_shared (bad bits = the geometry's mask_bad) and the geometry's bad regions folded into the data as -32768; values round(value / ADU per photon) as int16 (clipped to +-32767; no pixel clipped in either set); bitshuffle/LZ4, one chunk per frame at /entry_1/data_1/data. It writes the matching CrystFEL geometry (16 panels, each the rigid fit of its ASICs, module coffset kept) and refuses a module whose ASICs are not on one straight line at a whole-pixel pitch within 0.5 px (worst here: 0.16 px for CXIDB 83, 0.002 px for 152). Reader: a stored value equal to the smallest value of its signed integer type (INT16_MIN, INT32_MIN) is no measurement. The reader decodes to float and divides by the gain, so -32768 used to come out as a count; the CrystFEL parser takes only module geometry, so a flag_lessthan in the .geom could not say it. Still battery: euxfel83 / euxfel152 read conv/*_int16.h5 with conv/agipd_int16.geom and --gain 1. Full sets, -N 16, --refine-geometry=off, --model (other jobs on the machine, load 7-23): indexed d_min CC1/2 CC_HALF_SPLIT R_split R_FREE main pass wall CXIDB 83 original 90.9% 1.84 0.641 0.617 0.482 0.280 89.3 s (31/s) 105 s converted 94.1% 1.84 0.686 0.644 0.470 0.277 6.5 s (429/s) 20 s CXIDB 152 original 17.5% 5.37 0.521 0.479 0.590 0.308 131.7 s (29/s) 153 s converted 62.8% 2.92 0.876 0.891 0.354 0.284 13.2 s (287/s) 27 s Files 4.7 -> 1.3 GB and 5.7 -> 1.1 GB. The 152 gain is the mask: the reader took only Cheetah's generic bad bits, while the geometry's mask_bad (0xffff) also marks three broken modules (bit 32) and its bad region a fourth; the converter run with the generic bits instead gives 17.5% / 3.41 A again. With geometry refinement on (16 rigid modules): 83 94.2%, 1.75 A, CC1/2 0.665, R_FREE 0.284, 49 s; 152 65.1%, 2.94 A, CC1/2 0.843, R_FREE 0.278, 27 s - mixed, the manifest keeps it off. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu --- docs/EXTERNAL_TEST_DATA_STILLS.md | 30 +++++- reader/JFJochXFELReader.cpp | 11 +- tests/XFELReaderTest.cpp | 4 + tools/battery/README.md | 3 +- tools/battery/still/agipd_to_int16.py | 150 ++++++++++++++++++++++++++ tools/battery/still_manifest.json | 4 +- 6 files changed, 192 insertions(+), 10 deletions(-) create mode 100644 tools/battery/still/agipd_to_int16.py diff --git a/docs/EXTERNAL_TEST_DATA_STILLS.md b/docs/EXTERNAL_TEST_DATA_STILLS.md index d4712c440..f6d3747c5 100644 --- a/docs/EXTERNAL_TEST_DATA_STILLS.md +++ b/docs/EXTERNAL_TEST_DATA_STILLS.md @@ -29,8 +29,8 @@ the paper of the entry. | `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) | | `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) | | `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) | -| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 x (512 x 128), 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) | -| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) | +| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI (converted) | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) | +| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI (converted) | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) | | `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) | | `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) | @@ -49,8 +49,8 @@ broken download resumes with `curl -C -`. | `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB | | `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB | | `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB | -| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | 4.7 GB | -| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | 5.7 GB | +| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | 4.7 GB (1.3 GB converted) | +| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | 5.7 GB (1.1 GB converted) | | `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | ~750 GB | | `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | ~430 GB | @@ -63,6 +63,28 @@ with `--gain 1` and merges them to the same numbers. Make them with into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's directory tree), named `_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy. +**The AGIPD sets.** rugnux reads AGIPD 1M data only in one converted form: one rigid panel per module, +with the gaps between its ASICs inserted as pixels, every masked pixel folded into the data, and int16 +photon counts. The Cheetah CXI files store each 512 x 128 module as eight 64 x 128 ASICs packed without +their 2-pixel gaps, place each ASIC as its own panel in the CrystFEL geometry, and keep the pixel mask in +separate per-frame and shared datasets of Cheetah bits. The converted file has +`/entry_1/data_1/data` {N, 16 x 526, 128} int16 with bitshuffle/LZ4, one chunk per frame: each module is +526 rows (8 x 64 rows of ASICs, 66 rows apart), the gap rows and every masked pixel (a bad bit of the +geometry's `mask_bad` in the per-frame mask or in `mask_shared`, and the geometry's bad regions) hold +-32768, the smallest int16, which rugnux reads as no measurement; every other pixel holds +round(value / ADU per photon), clipped to ±32767. The beam energy, distance and pixel size datasets are +copied unchanged. The converter writes the matching CrystFEL geometry (16 panels of 526 x 128, each the +rigid fit of its module's ASICs, with the module's coffset), and refuses a geometry whose ASICs do not lie +on one straight line at a whole-pixel pitch within half a pixel. Make them with +`tools/battery/still/agipd_to_int16.py --geom --adu-per-photon --out-geom ` +(h5py and hdf5plugin), into `euxfel_83/conv/` (geometry `ref/agipd_mar18_v10.geom`, 60 ADU per photon) +and `euxfel_152/conv/` (`ref/kdo_embedded.geom`, 62 ADU per photon), named `_int16.h5`, with the +geometry `conv/agipd_int16.geom`; read them with `--gain 1`. Neither file states its gain (the geometries' +`adu_per_eV` is marked as a guess); both values were measured on the photon peaks of the pixel histogram. +The files' own photon energy (9.6 keV) and distance are not the experiment's: the manifest gives +`--wavelength`, and the geometry the camera length. In the CXIDB 152 files the dark offset is not removed +(the zero-photon peak is at about 2 photons), which is left as background. + ## Not in the battery - **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time diff --git a/reader/JFJochXFELReader.cpp b/reader/JFJochXFELReader.cpp index d59e524b4..271e969b3 100644 --- a/reader/JFJochXFELReader.cpp +++ b/reader/JFJochXFELReader.cpp @@ -820,9 +820,14 @@ void JFJochXFELReader::DecodeInto(int64_t image_number, Buffer &buffer) const { } } - // And so is one whose value the layout flags as no measurement. - const auto flagged = [this](float v) { - return (geom_.flag_lessthan && v < *geom_.flag_lessthan) || (geom_.flag_morethan && v > *geom_.flag_morethan) + // And so is one whose value the layout flags as no measurement, or that holds the smallest value of its + // signed integer type - what a converted file (tools/battery/still/agipd_to_int16.py) writes for a + // masked pixel or a gap. + const float type_min = layout_.stored_signed && !layout_.stored_float + ? -std::exp2(8.0f * static_cast(layout_.stored_size) - 1.0f) : -INFINITY; + const auto flagged = [this, type_min](float v) { + return v == type_min + || (geom_.flag_lessthan && v < *geom_.flag_lessthan) || (geom_.flag_morethan && v > *geom_.flag_morethan) || (geom_.flag_equal && v == *geom_.flag_equal); }; diff --git a/tests/XFELReaderTest.cpp b/tests/XFELReaderTest.cpp index 8bd0caa49..d74e93f81 100644 --- a/tests/XFELReaderTest.cpp +++ b/tests/XFELReaderTest.cpp @@ -2,6 +2,7 @@ // SPDX-License-Identifier: GPL-3.0-only #include +#include #include #include #include @@ -82,6 +83,7 @@ TEST_CASE("JFJochXFELReader_SACLA", "[portable]") { frame[8 * 4] = -5; // second sensor (fs 0, ss 0): below zero frame[8 * 4 + 1] = 33; // second sensor (fs 1, ss 0) frame[9 * 4] = 44; // second sensor (fs 0, ss 1) + frame[2 * 4] = INT16_MIN; // first sensor (fs 0, ss 2): no measurement WriteFrames(g, "data", H5T_NATIVE_INT16, frame.data(), {16, 4}); WriteScalar(g, "photon_energy_ev", 10000.0 + 10.0 * shot); H5Gclose(g); @@ -109,6 +111,8 @@ TEST_CASE("JFJochXFELReader_SACLA", "[portable]") { CHECK(PixelAt(reader, 0, 21, 2) == -5); CHECK(PixelAt(reader, 0, 21, 3) == 33); CHECK(PixelAt(reader, 0, 20, 2) == 44); + // The smallest int16 is no measurement, not a count. + CHECK(PixelAt(reader, 0, 2, 4) == INT32_MIN); // The file's mask holds the first sensor's (fs 1, ss 0), at column 3, row 2. const auto &mask = reader.GetDataset()->pixel_mask->GetMask(); CHECK(mask[2 * x.GetXPixelsNum() + 3] != 0); diff --git a/tools/battery/README.md b/tools/battery/README.md index 4fe3d9b79..766137372 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -983,6 +983,7 @@ paths come from the `"still"` section of the site config: run directory and `report_PRIVATE.md`, and `keep` refuses it. Needs `gemmi` (Python), CCP4's `dimple` and `reindex`, and `fincore` (util-linux) for the page-cache column. -Data preparation for the `_int16` sets is `still/float32_to_int16.py` (see the stills data page). +Data preparation for the `_int16` sets is `still/float32_to_int16.py`, and for the AGIPD sets (`euxfel*`, +read only in that converted form) `still/agipd_to_int16.py` (see the stills data page). `still/rugnux_stills_ab.py` is the older A/B harness: it compares two argument sets on a fixed subset of one dataset, shell by shell, and takes its dataset list from outside the repository. diff --git a/tools/battery/still/agipd_to_int16.py b/tools/battery/still/agipd_to_int16.py new file mode 100644 index 000000000..3dba8be50 --- /dev/null +++ b/tools/battery/still/agipd_to_int16.py @@ -0,0 +1,150 @@ +"""Data preparation for the still battery (not product code): convert an AGIPD 1M Cheetah CXI file into the form +rugnux reads AGIPD in - one rigid panel per module with the ASIC gaps inserted, int16 photon counts, every masked or +gap pixel written as -32768 (the smallest int16, which rugnux reads as no measurement). + +usage: agipd_to_int16.py --geom --adu-per-photon [--out-geom ] + +The input frame {N, 16*512, 128} holds 16 modules of 8 ASICs (64 x 128) stacked along ss; the CrystFEL geometry places +every ASIC as its own panel. Each module's ASICs must lie on one straight line along their ss axis at one whole-pixel +pitch (66 = 64 + a 2-pixel gap) - a rigid module - or the file is refused. The output frame is {N, 16*526, 128} at +/entry_1/data_1/data, bitshuffle/LZ4, one chunk per frame; the gap rows are -32768. Masked: a pixel with a bad bit +(the geometry's mask_bad/mask_good) in the per-frame mask or in mask_shared, and the geometry's bad regions. +Values: round(value / adu_per_photon), clipped to [-32767, 32767]. The beam energy, distance and pixel size datasets +are copied unchanged. --out-geom writes the matching geometry: one panel per module (the rigid fit of its ASICs). +Read with --crystfel-geometry --gain 1. Needs h5py and hdf5plugin. See docs/EXTERNAL_TEST_DATA_STILLS.md.""" +import argparse, os, re, time, h5py, hdf5plugin, numpy as np + +DET = 'entry_1/instrument_1/detector_1/' +COPY = ['entry_1/experiment_identifier', 'entry_1/instrument_1/source_1/energy', DET + 'description', DET + 'distance', + DET + 'x_pixel_size', DET + 'y_pixel_size', 'instrument'] +TOLERANCE_PX = 0.5 # how far an ASIC corner may lie from its rigid module's + +ap = argparse.ArgumentParser() +ap.add_argument('src') +ap.add_argument('dst') +ap.add_argument('--geom', required=True) +ap.add_argument('--adu-per-photon', type=float, required=True) +ap.add_argument('--out-geom') +a = ap.parse_args() + + +def direction(text): + v = {'x': 0.0, 'y': 0.0, 'z': 0.0} + for sign, num, axis in re.findall(r'([+-]?)\s*([0-9.]*(?:[eE][+-]?[0-9]+)?)\s*([xyz])', text): + v[axis] = (-1 if sign == '-' else 1) * (float(num) if num else 1.0) + if v['z'] != 0: + raise SystemExit(f'{a.geom}: a panel tilted out of the detector plane ({text})') + return np.array([v['x'], v['y']]) + + +# The geometry: global keys, ASIC panels pa, bad regions bad*. +glob, asics, bad = {}, {}, {} +for line in open(a.geom): + line = line.split(';')[0] + if '=' not in line: + continue + key, value = (s.strip() for s in line.split('=', 1)) + if '/' not in key: + glob[key] = value + continue + name, prop = key.split('/', 1) + m = re.fullmatch(r'p(\d+)a(\d+)', name) + if m: + asics.setdefault(int(m[1]), {}).setdefault(int(m[2]), {})[prop] = value + elif name.startswith('bad'): + bad.setdefault(name, {})[prop] = value + elif not name.startswith('rigid_group'): + raise SystemExit(f'{a.geom}: panel {name} is not an AGIPD ASIC pa') + +# Every module a rigid row of ASICs: same axes and coffset, consecutive ss ranges of the same height covering all fs, +# corners on one line along ss at one whole-pixel pitch. +nmod, nasic = len(asics), len(asics[0]) +first = asics[0][0] +height, width = int(first['max_ss']) - int(first['min_ss']) + 1, int(first['max_fs']) - int(first['min_fs']) + 1 +c0, c1 = (np.array([float(asics[0][k]['corner_x']), float(asics[0][k]['corner_y'])]) for k in (0, 1)) +pitch = int(round((c1 - c0) @ direction(first['ss']))) +modules, worst = [], 0.0 +for mod in range(nmod): + p = asics[mod] + fs, ss = direction(p[0]['fs']), direction(p[0]['ss']) + corners = [] + for k in range(nasic): + q = p[k] + ok = ((direction(q['fs']) == fs).all() and (direction(q['ss']) == ss).all() + and q.get('coffset') == p[0].get('coffset') and q['min_fs'] == '0' and int(q['max_fs']) == width - 1 + and int(q['min_ss']) == (mod * nasic + k) * height and int(q['max_ss']) == (mod * nasic + k + 1) * height - 1) + if not ok: + raise SystemExit(f'{a.geom}: module {mod} ASIC {k} is not laid out as the next {height} rows of its module ' + 'with the same axes and coffset; not a rigid module, not converted') + corners.append(np.array([float(q['corner_x']), float(q['corner_y'])]) - k * pitch * ss) + corner = np.mean(corners, axis=0) + dev = max(np.hypot(*(c - corner)) for c in corners) + worst = max(worst, dev) + if dev > TOLERANCE_PX: + raise SystemExit(f'{a.geom}: module {mod} ASICs lie up to {dev:.2f} px off one straight line at a {pitch}-pixel ' + 'pitch; not a rigid module, not converted') + modules.append((fs, ss, corner, p[0].get('coffset'))) +gap = pitch - height +rows = nasic * height + (nasic - 1) * gap # output rows per module + +if 'mask_good' in glob and int(glob['mask_good'], 0) != 0: + raise SystemExit(f'{a.geom}: mask_good is set; only mask_bad is converted') +mask_bad = int(glob['mask_bad'], 0) +bad_frame = np.zeros((nmod * nasic * height, width), dtype=bool) +for name, r in bad.items(): + if set(r) != {'min_fs', 'max_fs', 'min_ss', 'max_ss'}: + raise SystemExit(f'{a.geom}: bad region {name} is not a whole-frame fs/ss range; not converted') + bad_frame[int(r['min_ss']):int(r['max_ss']) + 1, int(r['min_fs']):int(r['max_fs']) + 1] = True + +# Stored row -> output row. +out_row = np.array([mod * rows + k * pitch + i for mod in range(nmod) for k in range(nasic) for i in range(height)]) + +if a.out_geom: + with open(a.out_geom, 'w') as f: + f.write(f'; AGIPD as rugnux reads it: written by agipd_to_int16.py from {os.path.basename(a.geom)}.\n' + f'; One rigid panel per module ({rows} x {width}: {nasic} ASICs of {height} rows, {gap}-row gaps\n' + '; inserted), int16 photon counts, -32768 = masked or gap. Read with --gain 1.\n') + for key in ('clen', 'res', 'photon_energy'): + f.write(f'{key} = {glob[key]}\n') + f.write('adu_per_photon = 1\ndata = /entry_1/data_1/data\ndim0 = %\ndim1 = ss\ndim2 = fs\n\n') + for mod, (fs, ss, corner, coffset) in enumerate(modules): + f.write(f'p{mod}/min_fs = 0\np{mod}/max_fs = {width - 1}\np{mod}/min_ss = {mod * rows}\n' + f'p{mod}/max_ss = {(mod + 1) * rows - 1}\np{mod}/fs = {fs[0]:+.6f}x {fs[1]:+.6f}y\n' + f'p{mod}/ss = {ss[0]:+.6f}x {ss[1]:+.6f}y\np{mod}/corner_x = {corner[0]:.3f}\n' + f'p{mod}/corner_y = {corner[1]:.3f}\n' + (f'p{mod}/coffset = {coffset}\n' if coffset else '') + '\n') + +t0 = time.time() +with h5py.File(a.src, 'r') as fi, h5py.File(a.dst + '.part', 'w') as fo: + for path in COPY: + if path in fi: + fo.copy(fi[path], fo.require_group(os.path.dirname(path) or '/'), name=os.path.basename(path)) + d = fi[DET + 'detector_corrected/data'] + m = fi[DET + 'detector_corrected/mask'] + shared_bad = bad_frame | ((fi[DET + 'detector_corrected/mask_shared'][()] & mask_bad) != 0) + n = d.shape[0] + if d.shape[1:] != bad_frame.shape: + raise SystemExit(f'{a.src}: frame {d.shape[1:]} is not the {bad_frame.shape} the geometry lays out') + out = fo.create_dataset('entry_1/data_1/data', shape=(n, nmod * rows, width), dtype='int16', + chunks=(1, nmod * rows, width), **hdf5plugin.Bitshuffle(nelems=0, cname='lz4')) + out.attrs['adu_per_photon'] = a.adu_per_photon + out.attrs['conversion'] = (f'int16 photons = round(value / {a.adu_per_photon}), clipped to +-32767; -32768 = ' + f'masked (mask_bad {mask_bad:#x}, mask_shared, bad regions) or ASIC gap; ' + f'original: {a.src}, geometry {a.geom}') + clipped = masked = 0 + frame = np.full((nmod * rows, width), -32768, dtype=np.int16) + for i in range(n): + p = np.rint(d[i] / a.adu_per_photon) + bad_i = shared_bad | ((m[i] & mask_bad) != 0) + over = (np.abs(p) > 32767) & ~bad_i + clipped += int(over.sum()) + masked += int(bad_i.sum()) + p = np.clip(p, -32767, 32767).astype(np.int16) + p[bad_i] = -32768 + frame[out_row] = p + out[i] = frame +os.rename(a.dst + '.part', a.dst) +s_in, s_out = os.path.getsize(a.src), os.path.getsize(a.dst) +print(f'{os.path.basename(a.src)} frames {n} in {s_in / 1e9:.2f} GB out {s_out / 1e9:.3f} GB time {time.time() - t0:.0f} s ' + f'masked {masked / n / bad_frame.size:.2%} of stored pixels per frame, clipped {clipped}; ' + f'{nmod} modules {rows} x {width}, ASIC pitch {pitch}, worst ASIC corner off its rigid module {worst:.3f} px', + flush=True) diff --git a/tools/battery/still_manifest.json b/tools/battery/still_manifest.json index ca4896275..633c89717 100644 --- a/tools/battery/still_manifest.json +++ b/tools/battery/still_manifest.json @@ -8,8 +8,8 @@ {"id": "lcls222", "cxidb": 222, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "8TDQ", "sg": "P 65 2 2", "d_cut": 1.65, "r_free_deposited": 0.2012, "images_deposited": null, "inputs": ["lcls_222/xtc/*.xtc", "lcls_222/xtc_more/*.xtc"], "args": ["--detector-distance", "193.43", "--polarization", "0.998", "-C", "78.616,78.616,265.275,90,90,120", "-S", "P6522"], "tiers": {"smoke": {"images": 1500}}}, {"id": "lcls242_s", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4S", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2102, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "92.044,93.019,49.014,90,92.69,90", "-S", "P21"], "tiers": {}}, {"id": "lcls242_t", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4T", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2348, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "96.031,94.536,50.529,90,91.18,90", "-S", "P21"], "tiers": {}}, - {"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_83/ref/agipd_mar18_v10.geom", "--gain", "60", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {}}, - {"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_152/ref/kdo_embedded.geom", "--gain", "62", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {}}, + {"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI, converted: int16 photons, one panel per module", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/conv/*_int16.h5"], "args": ["--crystfel-geometry", "{data}/euxfel_83/conv/agipd_int16.geom", "--gain", "1", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {}}, + {"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI, converted: int16 photons, one panel per module", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/conv/*_int16.h5"], "args": ["--crystfel-geometry", "{data}/euxfel_152/conv/agipd_int16.geom", "--gain", "1", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {}}, {"id": "alvra103", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/raw/run*.JF06T32V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "4.57", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}}, {"id": "alvra103_int16", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/conv/run*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "1", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}}, {"id": "alvra213", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "12.037", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21", "--mask-good", "{data}/swissfel_213/raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5:/mask"], "tiers": {}},