Merge branch 'rc178-still-results' into geom-mask
This commit is contained in:
leonarski_f committed
commit
30ea607686
4 files changed
+21
-202
No files matched your search
@@ -13,10 +13,9 @@ the paper of the entry.
|
||||
|
||||
- **Dataset DOI**: the CXIDB entry's own DOI, resolved against DataCite before it was written down.
|
||||
- **Space group, cell, resolution and R-free** are the values deposited with the PDB entry, read from
|
||||
the RCSB data API. Every entry is a dark or ground state refined against observed data; entries
|
||||
refined against extrapolated structure factors (the time points of a pump-probe series) are not
|
||||
used, because a merge of the measured images cannot be compared with them.
|
||||
- **Images** are the frames in the files we use, counted in the files themselves.
|
||||
the RCSB data API. Every entry is a dark or ground state refined against observed data.
|
||||
- **Images used** are the frames in the files we use, counted in the files themselves. Where that is
|
||||
only part of the entry, the table says so and gives what the entry or the deposition holds.
|
||||
|
||||
## Datasets
|
||||
|
||||
@@ -29,8 +28,8 @@ the paper of the entry.
|
||||
| `pal90` | [90](https://cxidb.org/id-90.html), [10.11577/1511590](https://doi.org/10.11577/1511590) | PAL-XFEL NCI | Rayonix MX225-HS; Cheetah CXI | [6IRK](https://www.rcsb.org/structure/6IRK) | I 2 2 2, 1.75 | Lee et al., Sci. Rep. (2019) |
|
||||
| `lcls222` | [222](https://cxidb.org/id-222.html), [10.11577/2204603](https://doi.org/10.11577/2204603) | LCLS MFX | Rayonix MX340-HS, 1920 x 1920, 177 µm; LCLS-I XTC | [8TDQ](https://www.rcsb.org/structure/8TDQ) | P 65 2 2, 1.65 | Nguyen et al., J. Am. Chem. Soc. (2023) |
|
||||
| `lcls242_s`, `lcls242_t` | [242](https://cxidb.org/id-242.html), [10.11577/3374214](https://doi.org/10.11577/3374214) | LCLS MFX | Rayonix MX340-HS; LCLS-I XTC | [9O4S](https://www.rcsb.org/structure/9O4S) (small cell), [9O4T](https://www.rcsb.org/structure/9O4T) (large cell) | P 1 21 1, 1.95 | Wolff et al., bioRxiv (2026) |
|
||||
| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI (converted) | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
|
||||
| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI (converted) | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
|
||||
| `euxfel83` | [83](https://cxidb.org/id-83.html), [10.11577/1464102](https://doi.org/10.11577/1464102) | European XFEL SPB/SFX | AGIPD 1M, 16 modules of 512 x 128, 200 µm; Cheetah CXI | [6GTH](https://www.rcsb.org/structure/6GTH) | P 32 2 1, 1.69 | Wiedorn et al., Nat. Commun. (2018) |
|
||||
| `euxfel152` | [152](https://cxidb.org/id-152.html), [10.11577/1660970](https://doi.org/10.11577/1660970) | European XFEL SPB/SFX | AGIPD 1M; Cheetah CXI | [6U57](https://www.rcsb.org/structure/6U57) | I 2 3, 2.80 | Echelmeier et al., Nat. Commun. (2020) |
|
||||
| `alvra103`, `alvra103_int16` | [103](https://cxidb.org/id-103.html), [10.11577/1650020](https://doi.org/10.11577/1650020) | SwissFEL Alvra | JUNGFRAU 16M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [6S0L](https://www.rcsb.org/structure/6S0L) | C 2 2 21, 2.65 | Nass et al., IUCrJ (2020) |
|
||||
| `alvra213`, `alvra213_int16` | [213](https://cxidb.org/id-213.html), [10.11577/2282689](https://doi.org/10.11577/2282689) | SwissFEL Alvra | JUNGFRAU 4M, 75 µm; SwissFEL HDF5, float32 keV (converted) | [8BKH](https://www.rcsb.org/structure/8BKH) (dark state) | P 1 21 1, 1.35 | Barends et al., Nature (2024) |
|
||||
|
||||
@@ -40,19 +39,19 @@ Paths are relative to the still data root of the site config (`tools/battery/REA
|
||||
serves `https://cxidb.org/data/<entry>/...`; most of it accepts HTTP range requests, so a
|
||||
broken download resumes with `curl -C -`.
|
||||
|
||||
| Set | Files | Images | Size |
|
||||
|---|---|---|---|
|
||||
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | 198 GB |
|
||||
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` (the dark reference; the time points are not used) | 10,792 | 48 GB |
|
||||
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | 67 GB |
|
||||
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | 111 GB |
|
||||
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | 25 GB |
|
||||
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | 72 GB |
|
||||
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB |
|
||||
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry; read as the converted copies in `euxfel_83/conv/` (below) | 2,790 | 4.7 GB (1.3 GB converted) |
|
||||
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream`; read as the converted copies in `euxfel_152/conv/` (below) | 3,801 | 5.7 GB (1.1 GB converted) |
|
||||
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | ~750 GB |
|
||||
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | ~430 GB |
|
||||
| Set | Files | Images used | Of the entry | Size |
|
||||
|---|---|---|---|---|
|
||||
| `sacla243` | `sacla_243/raw_all/run*.h5`: all 87 files of `243/Complete_dark/` | 52,151 | all (the deposition's 52,151) | 198 GB |
|
||||
| `sacla237` | `sacla_237/raw/`: the 18 files `run130637-0` to `run130642-2` of `237/SACLA_raw/2022Nov/dark_ref/` | 10,792 | **subset**: 18 of about 1,000 `dark_ref` files, about 1.8% of ~600,000 hits | 48 GB |
|
||||
| `sacla236` | `sacla_236/raw/`: the 18 files `run157084-0` to `run157089-2` of `236/Data/lyso01-apo-bigXtal/` | 16,083 | all (the deposition's 16,083) | 67 GB |
|
||||
| `pal86` | `pal_86/raw/`, `pal_86/raw_more/`: `CXIDB_86_themolysin_{2..25}_SFX_Nam.cxi` (file 1 is truncated on the server and cannot be opened) | 49,193 | all but file 1 (24 of 25 files) | 111 GB |
|
||||
| `pal90` | `pal_90/raw/`: `CXIDB_90-GI_SFX-filter-r000{1,2}-c00.cxi` | 11,947 | all (both files) | 25 GB |
|
||||
| `lcls222` | `lcls_222/xtc/`, `lcls_222/xtc_more/`: `mfx/mfxlu5017/xtc/e1277-r0165-{s00-c00,s00-c01,s01-c00,s01-c01,s02-c00}.xtc` (run 165, the substrate-bound state of 8TDQ) | 8,534 | **subset**: run 165 of the runs 165-168 behind 8TDQ | 72 GB |
|
||||
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | **subset**: runs 256-267 (44 of about 220 stream files; ~610 GB of ~3 TB) | ~700 GB |
|
||||
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | **subset**: 3 files; the deposited stream has 14,445 frames | 4.7 GB |
|
||||
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | **subset**: 9 runs; the deposited stream has 37,006 frames | 5.7 GB |
|
||||
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | **subset**: 15 runs of one day; the deposition indexed 199,136 of 2,251,787 images | ~750 GB |
|
||||
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | all 24 dark runs of the March 2019 series | ~430 GB |
|
||||
|
||||
**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
|
||||
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
|
||||
@@ -62,32 +61,3 @@ with `--gain 1` and merges them to the same numbers. Make them with
|
||||
`tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV>` (h5py and hdf5plugin),
|
||||
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's
|
||||
directory tree), named `<input>_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy.
|
||||
|
||||
**The AGIPD sets.** rugnux reads AGIPD 1M data only in one converted form: one rigid panel per module,
|
||||
with the gaps between its ASICs inserted as pixels, every masked pixel folded into the data, and int16
|
||||
photon counts. The Cheetah CXI files store each 512 x 128 module as eight 64 x 128 ASICs packed without
|
||||
their 2-pixel gaps, place each ASIC as its own panel in the CrystFEL geometry, and keep the pixel mask in
|
||||
separate per-frame and shared datasets of Cheetah bits. The converted file has
|
||||
`/entry_1/data_1/data` {N, 16 x 526, 128} int16 with bitshuffle/LZ4, one chunk per frame: each module is
|
||||
526 rows (8 x 64 rows of ASICs, 66 rows apart), the gap rows and every masked pixel (a bad bit of the
|
||||
geometry's `mask_bad` in the per-frame mask or in `mask_shared`, and the geometry's bad regions) hold
|
||||
-32768, the smallest int16, which rugnux reads as no measurement; every other pixel holds
|
||||
round(value / ADU per photon), clipped to ±32767. The beam energy, distance and pixel size datasets are
|
||||
copied unchanged. The converter writes the matching CrystFEL geometry (16 panels of 526 x 128, each the
|
||||
rigid fit of its module's ASICs, with the module's coffset), and refuses a geometry whose ASICs do not lie
|
||||
on one straight line at a whole-pixel pitch within half a pixel. Make them with
|
||||
`tools/battery/still/agipd_to_int16.py <in.cxi> <out.h5> --geom <in.geom> --adu-per-photon <g> --out-geom <out.geom>`
|
||||
(h5py and hdf5plugin), into `euxfel_83/conv/` (geometry `ref/agipd_mar18_v10.geom`, 60 ADU per photon)
|
||||
and `euxfel_152/conv/` (`ref/kdo_embedded.geom`, 62 ADU per photon), named `<input>_int16.h5`, with the
|
||||
geometry `conv/agipd_int16.geom`; read them with `--gain 1`. Neither file states its gain (the geometries'
|
||||
`adu_per_eV` is marked as a guess); both values were measured on the photon peaks of the pixel histogram.
|
||||
The files' own photon energy (9.6 keV) and distance are not the experiment's: the manifest gives
|
||||
`--wavelength`, and the geometry the camera length. In the CXIDB 152 files the dark offset is not removed
|
||||
(the zero-photon peak is at about 2 photons), which is left as background.
|
||||
|
||||
## Not in the battery
|
||||
|
||||
- **Time points of pump-probe series** (e.g. the light data of CXIDB 237 and 213): every deposited time
|
||||
point was refined against extrapolated structure factors.
|
||||
- **Raw JUNGFRAU data** (ADC values with gain bits that need pedestal and gain correction, e.g. LCLS
|
||||
CXIDB 189 and 214): only converted detector data are read.
|
||||
@@ -983,7 +983,6 @@ paths come from the `"still"` section of the site config:
|
||||
run directory and `report_PRIVATE.md`, and `keep` refuses it. Needs `gemmi` (Python), CCP4's
|
||||
`dimple` and `reindex`, and `fincore` (util-linux) for the page-cache column.
|
||||
|
||||
Data preparation for the `_int16` sets is `still/float32_to_int16.py`, and for the AGIPD sets (`euxfel*`,
|
||||
read only in that converted form) `still/agipd_to_int16.py` (see the stills data page).
|
||||
Data preparation for the `_int16` sets is `still/float32_to_int16.py` (see the stills data page).
|
||||
`still/rugnux_stills_ab.py` is the older A/B harness: it compares two argument sets on a fixed
|
||||
subset of one dataset, shell by shell, and takes its dataset list from outside the repository.
|
||||
@@ -1,150 +0,0 @@
|
||||
"""Data preparation for the still battery (not product code): convert an AGIPD 1M Cheetah CXI file into the form
|
||||
rugnux reads AGIPD in - one rigid panel per module with the ASIC gaps inserted, int16 photon counts, every masked or
|
||||
gap pixel written as -32768 (the smallest int16, which rugnux reads as no measurement).
|
||||
|
||||
usage: agipd_to_int16.py <in.cxi> <out.h5> --geom <in.geom> --adu-per-photon <g> [--out-geom <out.geom>]
|
||||
|
||||
The input frame {N, 16*512, 128} holds 16 modules of 8 ASICs (64 x 128) stacked along ss; the CrystFEL geometry places
|
||||
every ASIC as its own panel. Each module's ASICs must lie on one straight line along their ss axis at one whole-pixel
|
||||
pitch (66 = 64 + a 2-pixel gap) - a rigid module - or the file is refused. The output frame is {N, 16*526, 128} at
|
||||
/entry_1/data_1/data, bitshuffle/LZ4, one chunk per frame; the gap rows are -32768. Masked: a pixel with a bad bit
|
||||
(the geometry's mask_bad/mask_good) in the per-frame mask or in mask_shared, and the geometry's bad regions.
|
||||
Values: round(value / adu_per_photon), clipped to [-32767, 32767]. The beam energy, distance and pixel size datasets
|
||||
are copied unchanged. --out-geom writes the matching geometry: one panel per module (the rigid fit of its ASICs).
|
||||
Read with --crystfel-geometry <out.geom> --gain 1. Needs h5py and hdf5plugin. See docs/EXTERNAL_TEST_DATA_STILLS.md."""
|
||||
import argparse, os, re, time, h5py, hdf5plugin, numpy as np
|
||||
|
||||
DET = 'entry_1/instrument_1/detector_1/'
|
||||
COPY = ['entry_1/experiment_identifier', 'entry_1/instrument_1/source_1/energy', DET + 'description', DET + 'distance',
|
||||
DET + 'x_pixel_size', DET + 'y_pixel_size', 'instrument']
|
||||
TOLERANCE_PX = 0.5 # how far an ASIC corner may lie from its rigid module's
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('src')
|
||||
ap.add_argument('dst')
|
||||
ap.add_argument('--geom', required=True)
|
||||
ap.add_argument('--adu-per-photon', type=float, required=True)
|
||||
ap.add_argument('--out-geom')
|
||||
a = ap.parse_args()
|
||||
|
||||
|
||||
def direction(text):
|
||||
v = {'x': 0.0, 'y': 0.0, 'z': 0.0}
|
||||
for sign, num, axis in re.findall(r'([+-]?)\s*([0-9.]*(?:[eE][+-]?[0-9]+)?)\s*([xyz])', text):
|
||||
v[axis] = (-1 if sign == '-' else 1) * (float(num) if num else 1.0)
|
||||
if v['z'] != 0:
|
||||
raise SystemExit(f'{a.geom}: a panel tilted out of the detector plane ({text})')
|
||||
return np.array([v['x'], v['y']])
|
||||
|
||||
|
||||
# The geometry: global keys, ASIC panels p<module>a<asic>, bad regions bad*.
|
||||
glob, asics, bad = {}, {}, {}
|
||||
for line in open(a.geom):
|
||||
line = line.split(';')[0]
|
||||
if '=' not in line:
|
||||
continue
|
||||
key, value = (s.strip() for s in line.split('=', 1))
|
||||
if '/' not in key:
|
||||
glob[key] = value
|
||||
continue
|
||||
name, prop = key.split('/', 1)
|
||||
m = re.fullmatch(r'p(\d+)a(\d+)', name)
|
||||
if m:
|
||||
asics.setdefault(int(m[1]), {}).setdefault(int(m[2]), {})[prop] = value
|
||||
elif name.startswith('bad'):
|
||||
bad.setdefault(name, {})[prop] = value
|
||||
elif not name.startswith('rigid_group'):
|
||||
raise SystemExit(f'{a.geom}: panel {name} is not an AGIPD ASIC p<module>a<asic>')
|
||||
|
||||
# Every module a rigid row of ASICs: same axes and coffset, consecutive ss ranges of the same height covering all fs,
|
||||
# corners on one line along ss at one whole-pixel pitch.
|
||||
nmod, nasic = len(asics), len(asics[0])
|
||||
first = asics[0][0]
|
||||
height, width = int(first['max_ss']) - int(first['min_ss']) + 1, int(first['max_fs']) - int(first['min_fs']) + 1
|
||||
c0, c1 = (np.array([float(asics[0][k]['corner_x']), float(asics[0][k]['corner_y'])]) for k in (0, 1))
|
||||
pitch = int(round((c1 - c0) @ direction(first['ss'])))
|
||||
modules, worst = [], 0.0
|
||||
for mod in range(nmod):
|
||||
p = asics[mod]
|
||||
fs, ss = direction(p[0]['fs']), direction(p[0]['ss'])
|
||||
corners = []
|
||||
for k in range(nasic):
|
||||
q = p[k]
|
||||
ok = ((direction(q['fs']) == fs).all() and (direction(q['ss']) == ss).all()
|
||||
and q.get('coffset') == p[0].get('coffset') and q['min_fs'] == '0' and int(q['max_fs']) == width - 1
|
||||
and int(q['min_ss']) == (mod * nasic + k) * height and int(q['max_ss']) == (mod * nasic + k + 1) * height - 1)
|
||||
if not ok:
|
||||
raise SystemExit(f'{a.geom}: module {mod} ASIC {k} is not laid out as the next {height} rows of its module '
|
||||
'with the same axes and coffset; not a rigid module, not converted')
|
||||
corners.append(np.array([float(q['corner_x']), float(q['corner_y'])]) - k * pitch * ss)
|
||||
corner = np.mean(corners, axis=0)
|
||||
dev = max(np.hypot(*(c - corner)) for c in corners)
|
||||
worst = max(worst, dev)
|
||||
if dev > TOLERANCE_PX:
|
||||
raise SystemExit(f'{a.geom}: module {mod} ASICs lie up to {dev:.2f} px off one straight line at a {pitch}-pixel '
|
||||
'pitch; not a rigid module, not converted')
|
||||
modules.append((fs, ss, corner, p[0].get('coffset')))
|
||||
gap = pitch - height
|
||||
rows = nasic * height + (nasic - 1) * gap # output rows per module
|
||||
|
||||
if 'mask_good' in glob and int(glob['mask_good'], 0) != 0:
|
||||
raise SystemExit(f'{a.geom}: mask_good is set; only mask_bad is converted')
|
||||
mask_bad = int(glob['mask_bad'], 0)
|
||||
bad_frame = np.zeros((nmod * nasic * height, width), dtype=bool)
|
||||
for name, r in bad.items():
|
||||
if set(r) != {'min_fs', 'max_fs', 'min_ss', 'max_ss'}:
|
||||
raise SystemExit(f'{a.geom}: bad region {name} is not a whole-frame fs/ss range; not converted')
|
||||
bad_frame[int(r['min_ss']):int(r['max_ss']) + 1, int(r['min_fs']):int(r['max_fs']) + 1] = True
|
||||
|
||||
# Stored row -> output row.
|
||||
out_row = np.array([mod * rows + k * pitch + i for mod in range(nmod) for k in range(nasic) for i in range(height)])
|
||||
|
||||
if a.out_geom:
|
||||
with open(a.out_geom, 'w') as f:
|
||||
f.write(f'; AGIPD as rugnux reads it: written by agipd_to_int16.py from {os.path.basename(a.geom)}.\n'
|
||||
f'; One rigid panel per module ({rows} x {width}: {nasic} ASICs of {height} rows, {gap}-row gaps\n'
|
||||
'; inserted), int16 photon counts, -32768 = masked or gap. Read with --gain 1.\n')
|
||||
for key in ('clen', 'res', 'photon_energy'):
|
||||
f.write(f'{key} = {glob[key]}\n')
|
||||
f.write('adu_per_photon = 1\ndata = /entry_1/data_1/data\ndim0 = %\ndim1 = ss\ndim2 = fs\n\n')
|
||||
for mod, (fs, ss, corner, coffset) in enumerate(modules):
|
||||
f.write(f'p{mod}/min_fs = 0\np{mod}/max_fs = {width - 1}\np{mod}/min_ss = {mod * rows}\n'
|
||||
f'p{mod}/max_ss = {(mod + 1) * rows - 1}\np{mod}/fs = {fs[0]:+.6f}x {fs[1]:+.6f}y\n'
|
||||
f'p{mod}/ss = {ss[0]:+.6f}x {ss[1]:+.6f}y\np{mod}/corner_x = {corner[0]:.3f}\n'
|
||||
f'p{mod}/corner_y = {corner[1]:.3f}\n' + (f'p{mod}/coffset = {coffset}\n' if coffset else '') + '\n')
|
||||
|
||||
t0 = time.time()
|
||||
with h5py.File(a.src, 'r') as fi, h5py.File(a.dst + '.part', 'w') as fo:
|
||||
for path in COPY:
|
||||
if path in fi:
|
||||
fo.copy(fi[path], fo.require_group(os.path.dirname(path) or '/'), name=os.path.basename(path))
|
||||
d = fi[DET + 'detector_corrected/data']
|
||||
m = fi[DET + 'detector_corrected/mask']
|
||||
shared_bad = bad_frame | ((fi[DET + 'detector_corrected/mask_shared'][()] & mask_bad) != 0)
|
||||
n = d.shape[0]
|
||||
if d.shape[1:] != bad_frame.shape:
|
||||
raise SystemExit(f'{a.src}: frame {d.shape[1:]} is not the {bad_frame.shape} the geometry lays out')
|
||||
out = fo.create_dataset('entry_1/data_1/data', shape=(n, nmod * rows, width), dtype='int16',
|
||||
chunks=(1, nmod * rows, width), **hdf5plugin.Bitshuffle(nelems=0, cname='lz4'))
|
||||
out.attrs['adu_per_photon'] = a.adu_per_photon
|
||||
out.attrs['conversion'] = (f'int16 photons = round(value / {a.adu_per_photon}), clipped to +-32767; -32768 = '
|
||||
f'masked (mask_bad {mask_bad:#x}, mask_shared, bad regions) or ASIC gap; '
|
||||
f'original: {a.src}, geometry {a.geom}')
|
||||
clipped = masked = 0
|
||||
frame = np.full((nmod * rows, width), -32768, dtype=np.int16)
|
||||
for i in range(n):
|
||||
p = np.rint(d[i] / a.adu_per_photon)
|
||||
bad_i = shared_bad | ((m[i] & mask_bad) != 0)
|
||||
over = (np.abs(p) > 32767) & ~bad_i
|
||||
clipped += int(over.sum())
|
||||
masked += int(bad_i.sum())
|
||||
p = np.clip(p, -32767, 32767).astype(np.int16)
|
||||
p[bad_i] = -32768
|
||||
frame[out_row] = p
|
||||
out[i] = frame
|
||||
os.rename(a.dst + '.part', a.dst)
|
||||
s_in, s_out = os.path.getsize(a.src), os.path.getsize(a.dst)
|
||||
print(f'{os.path.basename(a.src)} frames {n} in {s_in / 1e9:.2f} GB out {s_out / 1e9:.3f} GB time {time.time() - t0:.0f} s '
|
||||
f'masked {masked / n / bad_frame.size:.2%} of stored pixels per frame, clipped {clipped}; '
|
||||
f'{nmod} modules {rows} x {width}, ASIC pitch {pitch}, worst ASIC corner off its rigid module {worst:.3f} px',
|
||||
flush=True)
|
||||
@@ -8,8 +8,8 @@
|
||||
{"id": "lcls222", "cxidb": 222, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "8TDQ", "sg": "P 65 2 2", "d_cut": 1.65, "r_free_deposited": 0.2012, "images_deposited": null, "inputs": ["lcls_222/xtc/*.xtc", "lcls_222/xtc_more/*.xtc"], "args": ["--detector-distance", "193.43", "--polarization", "0.998", "-C", "78.616,78.616,265.275,90,90,120", "-S", "P6522"], "tiers": {"smoke": {"images": 1500}}},
|
||||
{"id": "lcls242_s", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4S", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2102, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "92.044,93.019,49.014,90,92.69,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "lcls242_t", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4T", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2348, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "96.031,94.536,50.529,90,91.18,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI, converted: int16 photons, one panel per module", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/conv/*_int16.h5"], "args": ["--crystfel-geometry", "{data}/euxfel_83/conv/agipd_int16.geom", "--gain", "1", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {}},
|
||||
{"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI, converted: int16 photons, one panel per module", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/conv/*_int16.h5"], "args": ["--crystfel-geometry", "{data}/euxfel_152/conv/agipd_int16.geom", "--gain", "1", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {}},
|
||||
{"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_83/ref/agipd_mar18_v10.geom", "--gain", "60", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {}},
|
||||
{"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_152/ref/kdo_embedded.geom", "--gain", "62", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {}},
|
||||
{"id": "alvra103", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/raw/run*.JF06T32V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "4.57", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}},
|
||||
{"id": "alvra103_int16", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/conv/run*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "1", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}},
|
||||
{"id": "alvra213", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "12.037", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21", "--mask-good", "{data}/swissfel_213/raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5:/mask"], "tiers": {}},
|
||||
|
||||
Reference in new issue
Block a user