still battery: --cold evicts each set's input from the page cache; Alvra pixel masks; manifest fixes
- still_battery.py run --cold: posix_fadvise(DONTNEED) on a set's input files just before its run, so every timed row is a cold read without root (the page-cache column confirms it). - Alvra sets use the depositors' pixel masks (--mask-good); the MbCO int16 glob now finds the copies, which keep the tarball's directory tree; LCLS 242 states its wavelength (the XTC does not). - EXTERNAL_TEST_DATA_STILLS.md names the mask files and the one MbCO dark run that holds no frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu
This commit is contained in:
1 parent
b9ada7914b
commit
43e4207aaa
4 files changed
+27
-15
No files matched your search
@@ -51,8 +51,8 @@ broken download resumes with `curl -C -`.
|
||||
| `lcls242_*` | `lcls_242/xtc/`, `lcls_242/xtc_more/`: `xtc/mfxl1013721-r0256` to `r0267` streams (the two crystal forms are in every run; each set keeps its own form) | ~90,000 | ~700 GB |
|
||||
| `euxfel83` | `euxfel_83/raw/`: `XFEL-r0243-c00.cxi`, `XFEL-r0278-c00.cxi`, `XFEL-r0280-c00.cxi` (Cheetah hits, from the entry's `cheetah-calib-mar-v1/hdf5/` tree on the NERSC archive); `euxfel_83/ref/agipd_mar18_v10.geom` from the entry | 2,790 | 4.7 GB |
|
||||
| `euxfel152` | `euxfel_152/raw/`: runs r0108, 0110, 0112, 0114, 0115, 0116, 0118, 0123, 0125 extracted from `XFEL2042-runs-cxi.tar` (range requests on the tar members); `euxfel_152/ref/kdo_embedded.geom`, the geometry embedded in the header of the entry's `KDO8PS.stream` | 3,801 | 5.7 GB |
|
||||
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` from the entry | ~11,000 | ~750 GB |
|
||||
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz` | ~38,000 | ~430 GB |
|
||||
| `alvra103` | `swissfel_103/raw/`: `Raw_Files/2018-08-07/a2a/run{10..24}.JF06T32V01.h5`; `swissfel_103/ref/jf-16M.geom` and the pixel mask `swissfel_103/ref/run47.JF06T32V01.mask.h5` from the entry | ~11,000 | ~750 GB |
|
||||
| `alvra213` | `swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5`, from `march/march_dark_MbCO_II-March-06.tar.gz` (425 GB; download it to disk and extract, a streamed extraction does not survive a broken connection); `swissfel_213/ref/mbco_march.geom` from the JUNGFRAU geometry template of `2019-MARCH-SwissFEL-MbCO-supplement.tar.gz`; the pixel mask of run 243 (`data/pixel_mask/` in the same tarball) is used for all runs | ~38,000 | ~430 GB |
|
||||
|
||||
**The `_int16` sets.** The SwissFEL files store converted JUNGFRAU data as uncompressed float32 keV,
|
||||
68 MB (JF16M) or 17 MB (JF4M) a frame, which caps reading at about 74 or 295 frames/s at 5 GB/s. The
|
||||
@@ -60,7 +60,8 @@ broken download resumes with `curl -C -`.
|
||||
by the photon energy, rounded) as int16 with bitshuffle/LZ4, about 25 times smaller; rugnux reads them
|
||||
with `--gain 1` and merges them to the same numbers. Make them with
|
||||
`tools/battery/still/float32_to_int16.py <in.h5> <out.h5> <photon energy keV>` (h5py and hdf5plugin),
|
||||
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV), named `<input>_int16.h5`.
|
||||
into `swissfel_103/conv/` (4.57 keV) and `swissfel_213/conv/` (12.037 keV, keeping the tarball's
|
||||
directory tree), named `<input>_int16.h5`. One MbCO dark run (`run_000238`) holds no frames and has no copy.
|
||||
|
||||
## Not in the battery
|
||||
|
||||
|
||||
@@ -957,12 +957,12 @@ once and its row says what the merge is worth and what it cost:
|
||||
|
||||
The I/O floor and the page-cache state are there because several of these formats are uncompressed
|
||||
(XTC, float32 SwissFEL HDF5): a run whose time is at its floor measured the disk, not rugnux, and a
|
||||
warm run of a set smaller than the RAM measured neither. For honest timings drop the page cache
|
||||
before the run (`sync; echo 3 | sudo tee /proc/sys/vm/drop_caches`) and check that the column says
|
||||
cold.
|
||||
warm run of a set smaller than the RAM measured neither. For honest timings run with `--cold`, which
|
||||
evicts each set's input from the page cache just before its run (`posix_fadvise`, no root needed), and
|
||||
check that the column says cold.
|
||||
|
||||
```
|
||||
still_battery.py run --rugnux build/rugnux/rugnux [--only pal90,sacla236 | --tier smoke] [--threads N]
|
||||
still_battery.py run --rugnux build/rugnux/rugnux [--only pal90,sacla236 | --tier smoke] [--threads N] [--cold]
|
||||
still_battery.py compare RUN_A RUN_B
|
||||
still_battery.py report RUN [--out DIR]
|
||||
still_battery.py list
|
||||
|
||||
@@ -10,7 +10,7 @@ time of the rugnux command with the input's I/O floor next to it.
|
||||
|
||||
Usage (README.md in this directory, section "The still battery", is the guide):
|
||||
still_battery.py run --rugnux BIN [--label L] [--only a,b | --tier smoke] [--threads N]
|
||||
[--no-refine] [--io-gbps G]
|
||||
[--cold] [--no-refine] [--io-gbps G]
|
||||
still_battery.py compare RUN_A RUN_B
|
||||
still_battery.py report RUN [--out DIR] re-render a run's report.md
|
||||
still_battery.py list still runs under the runs root
|
||||
@@ -117,6 +117,14 @@ def page_cache(files):
|
||||
return res / size if size else None
|
||||
|
||||
|
||||
def evict(files):
|
||||
"""Drop the input's pages from the page cache, so the run reads it from the disk."""
|
||||
for f in files:
|
||||
fd = os.open(f, os.O_RDONLY)
|
||||
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def cache_state(fraction):
|
||||
if fraction is None:
|
||||
return "unknown"
|
||||
@@ -230,6 +238,8 @@ def run_one(run_dir, binary, e, st, opts):
|
||||
model = model_check.fetch(model_check.RCSB + pdb + ".pdb", os.path.join(st["pdb_cache"], pdb + ".pdb"))
|
||||
cmd, short = command(binary, e, st["data_root"], opts, model)
|
||||
io_bytes = sum(os.path.getsize(f) for f in files)
|
||||
if opts["cold"]:
|
||||
evict(files)
|
||||
cached = page_cache(files)
|
||||
t0 = time.time()
|
||||
rc = battery.run_logged(cmd, wd, opts["timeout"])
|
||||
@@ -313,7 +323,7 @@ def cmd_run(a, st):
|
||||
shutil.copy2(a.rugnux, binary)
|
||||
info.update(copied_from=os.path.realpath(a.rugnux), sha256=battery.sha256(binary))
|
||||
opts = {"threads": a.threads, "timeout": a.timeout, "tier": a.tier, "refine": not a.no_refine,
|
||||
"io_gbps": a.io_gbps}
|
||||
"io_gbps": a.io_gbps, "cold": a.cold}
|
||||
man = {"kind": "still", "label": label, "private": a.private, "subset": a.only or (f"tier {a.tier}" if a.tier else None),
|
||||
"binary": info, "runner_git": battery.git(HERE, "rev-parse", "--short", "HEAD") or "unknown",
|
||||
"host": socket.gethostname(), "hardware": battery.hardware_info(),
|
||||
@@ -399,6 +409,7 @@ def main():
|
||||
r.add_argument("--tier")
|
||||
r.add_argument("--private", action="store_true", help="the site's local-only still manifest")
|
||||
r.add_argument("--threads", type=int)
|
||||
r.add_argument("--cold", action="store_true", help="evict each set's input from the page cache before its run")
|
||||
r.add_argument("--timeout", type=int, default=6 * 3600)
|
||||
r.add_argument("--build-dir")
|
||||
r.add_argument("--no-refine", action="store_true", help="skip the dimple refinement")
|
||||
|
||||
@@ -6,12 +6,12 @@
|
||||
{"id": "pal86", "cxidb": 86, "facility": "PAL-XFEL", "beamline": "NCI", "detector": "Rayonix MX225-HS (4x4 binned, 156 um)", "format": "Cheetah CXI", "pdb": "6IG7", "sg": "P 61 2 2", "d_cut": 1.8, "r_free_deposited": 0.2024, "images_deposited": null, "inputs": ["pal_86/raw/*.cxi", "pal_86/raw_more/*.cxi"], "args": ["--detector-distance", "105", "--detect-ice-rings=on", "-C", "92.663,92.663,128.591,90,90,120", "-S", "P6122"], "tiers": {}},
|
||||
{"id": "pal90", "cxidb": 90, "facility": "PAL-XFEL", "beamline": "NCI", "detector": "Rayonix MX225-HS (4x4 binned, 156 um)", "format": "Cheetah CXI", "pdb": "6IRK", "sg": "I 2 2 2", "d_cut": 1.75, "r_free_deposited": 0.203, "images_deposited": null, "inputs": ["pal_90/raw/*.cxi"], "args": ["-C", "93.050,99.000,101.920,90,90,90", "-S", "I222"], "tiers": {"smoke": {"images": 1500}}},
|
||||
{"id": "lcls222", "cxidb": 222, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "8TDQ", "sg": "P 65 2 2", "d_cut": 1.65, "r_free_deposited": 0.2012, "images_deposited": null, "inputs": ["lcls_222/xtc/*.xtc", "lcls_222/xtc_more/*.xtc"], "args": ["--detector-distance", "193.43", "--polarization", "0.998", "-C", "78.616,78.616,265.275,90,90,120", "-S", "P6522"], "tiers": {"smoke": {"images": 1500}}},
|
||||
{"id": "lcls242_s", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4S", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2102, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--polarization", "0.998", "-C", "92.044,93.019,49.014,90,92.69,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "lcls242_t", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4T", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2348, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--polarization", "0.998", "-C", "96.031,94.536,50.529,90,91.18,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "lcls242_s", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4S", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2102, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "92.044,93.019,49.014,90,92.69,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "lcls242_t", "cxidb": 242, "facility": "LCLS", "beamline": "MFX", "detector": "Rayonix MX340-HS (4x4 binned, 177 um)", "format": "LCLS-I XTC", "pdb": "9O4T", "sg": "P 1 21 1", "d_cut": 1.95, "r_free_deposited": 0.2348, "images_deposited": null, "inputs": ["lcls_242/xtc/*.xtc", "lcls_242/xtc_more/*.xtc"], "args": ["--detector-distance", "199.0", "--wavelength", "1.294", "--polarization", "0.998", "-C", "96.031,94.536,50.529,90,91.18,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "euxfel83", "cxidb": 83, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6GTH", "sg": "P 32 2 1", "d_cut": 1.69, "r_free_deposited": 0.2096, "images_deposited": 14445, "inputs": ["euxfel_83/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_83/ref/agipd_mar18_v10.geom", "--gain", "60", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "41.84,41.84,233.28,90,90,120", "-S", "P3221"], "tiers": {}},
|
||||
{"id": "euxfel152", "cxidb": 152, "facility": "European XFEL", "beamline": "SPB/SFX", "detector": "AGIPD 1M (200 um)", "format": "Cheetah CXI", "pdb": "6U57", "sg": "I 2 3", "d_cut": 2.8, "r_free_deposited": 0.2489, "images_deposited": 37006, "inputs": ["euxfel_152/raw/*.cxi"], "args": ["--crystfel-geometry", "{data}/euxfel_152/ref/kdo_embedded.geom", "--gain", "62", "--wavelength", "1.3332", "--refine-geometry=off", "-C", "118,118,118,90,90,90", "-S", "I23"], "tiers": {}},
|
||||
{"id": "alvra103", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/raw/run*.JF06T32V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "4.57", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221"], "tiers": {}},
|
||||
{"id": "alvra103_int16", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/conv/run*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "1", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221"], "tiers": {}},
|
||||
{"id": "alvra213", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "12.037", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21"], "tiers": {}},
|
||||
{"id": "alvra213_int16", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/conv/run_*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "1", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21"], "tiers": {}}
|
||||
{"id": "alvra103", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/raw/run*.JF06T32V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "4.57", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}},
|
||||
{"id": "alvra103_int16", "cxidb": 103, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 16M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "6S0L", "sg": "C 2 2 21", "d_cut": 2.65, "r_free_deposited": 0.2203, "images_deposited": 199136, "inputs": ["swissfel_103/conv/run*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_103/ref/jf-16M.geom", "--gain", "1", "--wavelength", "2.7130", "-C", "40.34,180.66,143.05,90,90,90", "-S", "C2221", "--mask-good", "{data}/swissfel_103/ref/run47.JF06T32V01.mask.h5:/mask"], "tiers": {}},
|
||||
{"id": "alvra213", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "SwissFEL HDF5, float32 keV", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/raw/march_dark_MbCO_II-March-06/data/dark/run_*.JF06T08V01.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "12.037", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21", "--mask-good", "{data}/swissfel_213/raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5:/mask"], "tiers": {}},
|
||||
{"id": "alvra213_int16", "cxidb": 213, "facility": "SwissFEL", "beamline": "Alvra", "detector": "JUNGFRAU 4M (75 um)", "format": "int16 photons, bitshuffle/LZ4 copy", "pdb": "8BKH", "sg": "P 1 21 1", "d_cut": 1.35, "r_free_deposited": 0.2271, "images_deposited": null, "inputs": ["swissfel_213/conv/march_dark_MbCO_II-March-06/data/dark/run_*_int16.h5"], "args": ["--crystfel-geometry", "{data}/swissfel_213/ref/mbco_march.geom", "--gain", "1", "--wavelength", "1.0300", "-C", "64.1,28.8,35.8,90,106.9,90", "-S", "P21", "--mask-good", "{data}/swissfel_213/raw/march_dark_MbCO_II-March-06/data/pixel_mask/run_000243.JF06T08V01.mask.h5:/mask"], "tiers": {}}
|
||||
]}
|
||||
Reference in new issue
Block a user