tools/battery: one canonical runner, derived XDS d_min, clean inhouse layout
- Site config: site.example.json is committed; the local site.json (git-ignored) or $JFJOCH_BATTERY_SITE / --site names this machine's paths and the private manifest. Data roots are the arms' own directories, so copying them and editing the site config moves the battery. - Run states: manifest records the runner's pid; a run is complete, running, aborted (Ctrl-C/SIGTERM, or `abort RUN`) or unfinished. compare and report refuse a run that did not finish unless --allow-incomplete, and the report says so. - `list` shows runs and their state; `run` defaults its label to the rugnux version and its baseline to the site's persisted one; a baseline must be complete and of the same privacy as the run; a private report cannot be written into the repo. - XDS reference d_min: when the finest shell of XDS's table has no significant CC1/2, XDS merged past its own signal (no resolution cut, or a cut at the detector edge) and its limit is the detector's reach. The reference is then where XDS's CC1/2 falls through 0.30 (rugnux's own target), and that is also the range forced on rugnux. The manifest keeps XDS's limit (dmin_xds) and the rule (dmin_rule). Forcing the old 1.08 A on the weak insulin set turned rugnux's I23 into I222 (ISa 12.5); forced at the derived 1.81 A it is I23 again (ISa 18.0). Changed in the inhouse arm: insu_I_weak 1.08 -> 1.81, lyso_half_image 0.80 -> 2.60, cytc_eiger 2.04 -> 2.27, lyso_strong 1.18 -> 1.24; every other inhouse limit stands. - inhouse.json follows the directory rename (ids and inputs, data root /home/data/inhouse); `remap` applies such a rename and keeps the old ids as aliases, so runs from before it still compare set by set. - Optional open-arm model check (model_check.py, when present): R-free of the deposited model against each merge, recorded and plotted. - results.json has a fixed, documented row schema (results_schema 1); the HTML report gains inline-SVG plots (verdicts per arm, d_min and R-free ratios per set). - README.md: arms, data layout and provenance, prerequisites, running, run directory, results schema, report, comparing, protocol, adding datasets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8afab32d38
commit
a7a3c2a00c
10 files changed
+773
-111
No files matched your search
+4
-1
@@ -12,7 +12,7 @@ tmp_venv/
|
||||
# gen_python_client.sh: the built sdist/wheel.
|
||||
/dist/
|
||||
|
||||
# rugnux_vs_xds.py output dirs (the default, and the ad-hoc ones used when comparing runs)
|
||||
# output dirs of the retired rugnux_vs_xds.py, still lying around in older working trees
|
||||
rugnux_cmp*/
|
||||
|
||||
# rugnux_anomalous.py ANODE work dir, when the config does not point it at the data
|
||||
@@ -42,3 +42,6 @@ docs/review/
|
||||
|
||||
# Python bytecode (tools/battery and the root scripts)
|
||||
__pycache__/
|
||||
|
||||
# tools/battery: the local site config (paths of this machine, private manifest); site.example.json is the template
|
||||
tools/battery/site.json
|
||||
@@ -0,0 +1,279 @@
|
||||
# Rugnux validation battery
|
||||
|
||||
`battery.py` is the one way to run the Rugnux validation battery: run `rugnux` over a fixed list of
|
||||
rotation datasets, score each result against a reference, and write a standard report. Two runs
|
||||
of the same manifests with the same options give results you can compare set by set. Do not use
|
||||
ad-hoc runner scripts instead. The old ones (`rugnux_vs_xds.py`, `battery_run.py`,
|
||||
`battery2_serial.py`, `batstats.py`, ...) are retired because their numbers could not be compared
|
||||
with each other.
|
||||
|
||||
## Arms
|
||||
|
||||
| arm | datasets | reference | manifest |
|
||||
|---|---|---|---|
|
||||
| **open** | public PDB depositions of raw diffraction data, plus a few published small-molecule sets | the deposited space group, cell and resolution | `open.json` (committed) |
|
||||
| **inhouse** | standard test crystals measured at the SLS (lysozyme, thaumatin, insulin, cytochrome C, myoglobin), plus no-crystal controls | XDS, from the `CORRECT.LP` beside each dataset | `inhouse.json` (committed) |
|
||||
| **private** | user data | XDS, like inhouse | outside the repository; the local site config gives its path |
|
||||
|
||||
Scoring checks four things in order, and the first one that fails decides the verdict: did it
|
||||
run, is the lattice right, is the symmetry right, is the merge usable (R_meas at most 60%, CC1/2 at
|
||||
least 0.5). The lattice test compares Niggli-reduced primitive cells: the primitive volume ratio
|
||||
must be 0.95-1.05 and the reduced edges must agree within 2%. On the open arm the space group is
|
||||
scored with `sgequiv`. On the XDS arms only the point group is scored, because XDS never tests a
|
||||
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
|
||||
`score.py` has the details.
|
||||
|
||||
The private arm is never mixed with the others. A private run gets its own run directory
|
||||
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
|
||||
public report or be written into the repository. Its names, paths and values never go into a
|
||||
commit, an issue or a public report.
|
||||
|
||||
## Getting the data
|
||||
|
||||
Each manifest row gives an `input` path relative to that arm's **data root**. The data root is
|
||||
set in the site config, and the first directory of each input is the dataset's directory:
|
||||
|
||||
```
|
||||
<open data root>/11if/BopeA_18500_a_B2-PEF_11if/data/PSL-1503_13949_master.h5
|
||||
<inhouse data root>/lyso_ref/hewl_35_001_master.h5
|
||||
```
|
||||
|
||||
To move the battery to another machine, copy the two data roots and point the new site config at
|
||||
the copies. The paths inside the manifests do not change.
|
||||
|
||||
- **open**: every dataset is listed with its source repository and dataset DOI in
|
||||
[`docs/EXTERNAL_TEST_DATA.md`](../../docs/EXTERNAL_TEST_DATA.md). The sources are IRRMC
|
||||
(proteindiffraction.org), SBGrid Data Bank, Zenodo and a few others, and the page also says what
|
||||
each archive holds. Please cite those DOIs. At PSI the data root is `/home/data/open`, a
|
||||
symlink to `/home/data/nonsls/raw`. 51 of the dataset directories in it are themselves symlinks
|
||||
into `/data/scout_staging2`, so **copy with symlinks followed**: `rsync -aL` or `cp -rL`.
|
||||
A plain `rsync -a` copies dangling links. The open arm is about 1.7 TB.
|
||||
- **inhouse**: these are our own measurements and cannot be downloaded publicly. Ask the
|
||||
maintainers for a copy. Each dataset directory holds the images and the XDS output
|
||||
(`CORRECT.LP`, `XDS_ASCII.HKL`, ...) that the reference comes from. At PSI the data root is
|
||||
`/home/data/inhouse` (about 130 GB). **Leave out `.master_backups_kek`** when copying it
|
||||
(`rsync -a --exclude .master_backups_kek`): it holds backups of private masters, not battery data.
|
||||
- **private**: its data root and manifest are named only in the local site config.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.9 or newer, with `gemmi` (scoring). `h5py` is needed only by `discover`.
|
||||
- A `rugnux` binary. Build it with the flags CI uses (`-march=x86-64-v3`, see `CLAUDE.md`),
|
||||
because CPU-bound results and timings differ between builds. The runner records the build
|
||||
flags from the `CMakeCache.txt` above the binary, or from the tree given with `--build-dir`.
|
||||
- XDS is not needed. The XDS references are already in the manifests (`refs` re-reads them from
|
||||
`CORRECT.LP`).
|
||||
|
||||
## Site config
|
||||
|
||||
Machine-specific settings live in a site config: the data roots, the runs root, the lock file, an
|
||||
optional GPU-queue wrapper, the private arm, and the persisted baselines. The runner looks for it
|
||||
in this order:
|
||||
|
||||
1. `--site PATH`
|
||||
2. `$JFJOCH_BATTERY_SITE`
|
||||
3. `tools/battery/site.json` (git-ignored)
|
||||
|
||||
`site.example.json` is the committed template. Copy it and fill in your paths. The local file is
|
||||
kept out of git because it names the private manifest. At PSI the config is
|
||||
`/data/battery/site.json`: link it (`ln -s /data/battery/site.json tools/battery/site.json`) or
|
||||
export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"runs_root": "/data/battery/runs",
|
||||
"lock_file": "/data/battery/full_run.lock",
|
||||
"gpulock": "/data/tmp/gpulock",
|
||||
"baseline": "/data/battery/runs/<a blessed full run>",
|
||||
"baseline_private": null,
|
||||
"arms": {
|
||||
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"},
|
||||
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"},
|
||||
"private": {"manifest": "/path/to/private.json", "reference": "xds",
|
||||
"data_root": "/path/to/private/data", "private": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
|
||||
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
|
||||
the shared GPU slots. Set it to `null` if you have no such wrapper.
|
||||
|
||||
## Running
|
||||
|
||||
```
|
||||
B=tools/battery/battery.py
|
||||
R=build/rugnux/rugnux
|
||||
|
||||
$B run --rugnux $R # full battery: open + inhouse
|
||||
$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases
|
||||
$B run --rugnux $R --only 5reo,lyso_ref # named sets
|
||||
$B run --rugnux $R --arm private # the private arm, on its own
|
||||
$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced
|
||||
```
|
||||
|
||||
Useful options:
|
||||
|
||||
- `--label` names the run. By default the label is the rugnux version.
|
||||
- `--threads N` passes `-N` to rugnux.
|
||||
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
|
||||
- `--extra "<args>"` adds rugnux arguments to every set.
|
||||
- `--baseline RUN|none` sets the run the report is compared against.
|
||||
- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below).
|
||||
|
||||
A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
|
||||
manifests (`"tiers": {"smoke": "<why this set>"}`).
|
||||
|
||||
**Full runs.** A run without `--only` or `--tier` takes the site lock (`flock`, non-blocking), so
|
||||
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
|
||||
everything else on the machine competes with it.
|
||||
|
||||
### The model check (open arm)
|
||||
|
||||
If `tools/battery/model_check.py` is present, each open-arm merge (`p.mtz`) is also scored
|
||||
against the deposited model: `model_check.check(mtz, pdb_id, workdir)` returns R-free and R-work
|
||||
of the deposited model against our data, plus the deposited R-free. The runner records these as
|
||||
`rfree`, `rwork`, `rfree_deposited`, `rfree_ratio`, `model_status` and `model_reason`, and the
|
||||
report plots their ratio. The check is on by default when the module exists. Without the module
|
||||
the fields stay null. The PDB entry is the set id up to the first `_`, so `6h2p_native` is checked
|
||||
as `6h2p`. Small-molecule sets are skipped.
|
||||
|
||||
`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same
|
||||
site config) and compares the R-free that rugnux itself reports with the published value. It needs
|
||||
network access to the RCSB and runs separately from `battery.py`.
|
||||
|
||||
## A run directory
|
||||
|
||||
`<runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>[_private]/` is never reused, and it is made
|
||||
read-only when the run completes.
|
||||
|
||||
| path | content |
|
||||
|---|---|
|
||||
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
|
||||
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
|
||||
| `results.json` | one row per set (the schema is below) |
|
||||
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
|
||||
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
|
||||
|
||||
**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
|
||||
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
|
||||
`unfinished` (it died without a trace, which is treated the same as aborted). An unfinished run
|
||||
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
|
||||
`--allow-incomplete`, and the report then says so at the top.
|
||||
|
||||
### results.json (schema 1)
|
||||
|
||||
This is a flat list with one object per set, and every arm uses the same keys. Missing values are
|
||||
`null`. `manifest.json` records the schema version as `results_schema`.
|
||||
|
||||
| key | meaning |
|
||||
|---|---|
|
||||
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
|
||||
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) |
|
||||
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` |
|
||||
| `reason` | one line for a human |
|
||||
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours / the reference's |
|
||||
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
|
||||
| `cell`, `cell_ref`, `cell_dev_pct`, `volume_ratio` | cells, the largest edge deviation in % (over axis orders), and the primitive volume ratio |
|
||||
| `d_min`, `d_min_ref`, `res_gain_pct` | our high-resolution limit, the reference's, and (ref - ours) / ref in % |
|
||||
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
|
||||
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt` |
|
||||
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref` | XDS's overall values, over XDS's own range |
|
||||
| `rfree`, `rwork`, `rfree_deposited`, `rfree_ratio`, `model_status`, `model_reason` | the model check (open arm) |
|
||||
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |
|
||||
|
||||
## The report
|
||||
|
||||
Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
|
||||
SVG, no external files):
|
||||
|
||||
- a header with the binary, build flags, host, subset, resolution mode and whether timing counts;
|
||||
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
|
||||
ISa, R_meas, time) with a verdict bar chart;
|
||||
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
|
||||
the main metrics;
|
||||
- plots per set: d_min(rugnux) / d_min(reference) for each arm whose resolution is not forced, and
|
||||
R_free / deposited R_free for the open arm (HTML only; hovering a point shows the set);
|
||||
- the **failures**, and, if there is a baseline, the **delta against it**: pass rates on the
|
||||
common sets and every set that moved beyond noise;
|
||||
- **one row per set** with all the numbers.
|
||||
|
||||
Re-render a report, for example against another baseline (run directories are read-only, so the
|
||||
output goes elsewhere and is named after the run):
|
||||
|
||||
```
|
||||
$B report RUN --out DIR [--baseline OTHER_RUN] # writes DIR/<run>.md and DIR/<run>.html
|
||||
```
|
||||
|
||||
**Keeping a report in the repository.** Public reports of blessed runs go to
|
||||
`tools/battery/reports/<run>.md` (Markdown only; the HTML stays with the run). `report` refuses
|
||||
to write a private run's report anywhere inside the repository.
|
||||
|
||||
## Comparing runs
|
||||
|
||||
```
|
||||
$B list # runs, their state, and the baselines
|
||||
$B compare RUN_A RUN_B [--all] # per-set table of what moved beyond noise
|
||||
$B compare RUN_A RUN_B --rerun-changed # rerun the changed sets with A's binary
|
||||
```
|
||||
|
||||
`compare` pairs sets by arm and id, following the manifests' `aliases` across renames, and lists
|
||||
every set whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
|
||||
R-free or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds
|
||||
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
|
||||
A's saved binary and options. A set that moves again under the same binary is noise. A set that
|
||||
reproduces A was really changed by B.
|
||||
|
||||
## Protocol
|
||||
|
||||
1. **Compare against a persisted baseline, never against a number from memory.** The site
|
||||
config's `baseline` (and `baseline_private`) names a complete run. Every new run is reported
|
||||
against it by default. Set a new baseline only when the owner blesses a run.
|
||||
2. **Keep the manifests unchanged between the two runs you compare.** A change to a manifest
|
||||
(new sets, a new reference, a different input) is committed on its own, and the baseline is
|
||||
re-run with it.
|
||||
3. **Pool fixes.** Run the full battery once on a branch that carries several fixes, not once per
|
||||
fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
|
||||
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
|
||||
reference". Only a run with the machine to itself can be used to judge speed.
|
||||
5. **Resolution on the XDS arms.** By default XDS's resolution range is forced on rugnux, so both
|
||||
programs merge the same reflections and the statistics are like for like. This answers "is our
|
||||
processing better than XDS's, reflection for reflection". `--unforced` leaves rugnux its own
|
||||
cut, which answers "what does a user get". A change can win one and lose the other, so at
|
||||
important moments run both. Never quote the forced run as what rugnux delivers.
|
||||
|
||||
### The XDS reference d_min
|
||||
|
||||
`CORRECT.LP` gives the range XDS merged. That range is a resolution limit only if someone chose
|
||||
it. With `INCLUDE_RESOLUTION_RANGE` left at `0.0`, or set at the detector edge, it is the detector's
|
||||
reach, not the crystal's. It then gives a meaningless reference, and forcing it on rugnux makes
|
||||
rugnux merge shells of pure noise. On one insulin set, forcing XDS's 1.08 A (where the data end
|
||||
near 1.8 A) turned rugnux's correct I23 into I222. XDS marks each shell's CC1/2 with `*` when it
|
||||
is significant, and that decides which case applies (`inputs.reference_dmin`):
|
||||
|
||||
- `xds_range`: the finest shell's CC1/2 is significant, so XDS's limit stands.
|
||||
- `cc_half_0.30`: it is not, meaning XDS merged past its own signal. The reference is then the
|
||||
point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
|
||||
1/d² between shell centres.
|
||||
|
||||
The derived value is both the score reference and the forced range. The manifest keeps XDS's own
|
||||
limit as `dmin_xds` and the rule used as `dmin_rule`. XDS's pooled R_meas, CC1/2 and completeness
|
||||
still cover its whole range.
|
||||
|
||||
## Adding or changing datasets
|
||||
|
||||
- **Open arm**: put the dataset directory, named by its PDB code, under the open data root, then
|
||||
run `battery.py discover`. It proposes an input for every directory the manifest does not have
|
||||
(the largest sweep, chosen by frame count), and with `--refresh` it shows where discovery now
|
||||
disagrees with an unpinned manifest input. Add the row to `open.json` with the deposited
|
||||
reference (`sg`, `sgno`, `cell`, `dmin`) and tags. Set `"pinned": true` when the choice between
|
||||
sweeps was deliberate. Record the source and DOI in `docs/EXTERNAL_TEST_DATA.md`.
|
||||
- **Inhouse / private**: add the row (`id`, `input`, tags, and `"expect": "no_lattice"` for a
|
||||
control), then run `battery.py refs --arm inhouse --write` to fill in `ref` from the
|
||||
`CORRECT.LP` beside the input. Without `--write`, `refs` only shows what would change.
|
||||
A reference measured by hand goes in `ref_override`, which `refs` leaves alone.
|
||||
- **Renamed data directories**: `battery.py remap --arm inhouse MAP.json --write`, where
|
||||
`MAP.json` is `{"old": "new"}` (renames the set and its directory) or
|
||||
`{"old id": {"id": "...", "input": "..."}}`. Each old id is recorded under `aliases` in the
|
||||
manifest, so runs made before the rename still compare set by set. Without `--write`, the
|
||||
command only shows the result and flags inputs that do not exist.
|
||||
+220
-43
@@ -10,18 +10,25 @@ Arms and where their truth comes from:
|
||||
private user data, scored like inhouse. Its manifest lives OUTSIDE the repository (path in the
|
||||
site config) and a private run gets its own run directory and report.
|
||||
|
||||
Usage:
|
||||
battery.py run --rugnux BIN --label L [--arm open,inhouse|private] [--only a,b | --tier smoke]
|
||||
[--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN]
|
||||
Usage (README.md in this directory is the full guide):
|
||||
battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke]
|
||||
[--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN|none]
|
||||
battery.py list runs under the runs root, with their state
|
||||
battery.py compare RUN_A RUN_B [--all] [--rerun-changed]
|
||||
battery.py report RUN --out DIR [--baseline RUN] re-render a run's report
|
||||
battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/<run>.{md,html}
|
||||
battery.py abort RUN [--reason TEXT] mark an unfinished run as aborted
|
||||
battery.py discover [--refresh] open arm: propose inputs for new directories
|
||||
battery.py refs --arm inhouse|private [--write] re-read the XDS references from CORRECT.LP
|
||||
battery.py remap --arm A MAP.json [--write] follow a rename of the data directories
|
||||
|
||||
The site config (data roots, runs root, lock, gpulock, private manifest, baselines) is --site, else
|
||||
$JFJOCH_BATTERY_SITE, else site.json beside this script (not committed; site.example.json is).
|
||||
|
||||
A run writes <runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>/ holding a copy of the binary,
|
||||
manifest.json (what was run, on what, with which references), results.json (one row per set),
|
||||
report.md + report.html and work/<arm>/<set>/ (log, report, cif). It is made read-only when done.
|
||||
A full-arm run (no --only / --tier) takes the site lock and refuses to start if another holds it.
|
||||
A run that did not finish is never compared or reported on unless asked (--allow-incomplete).
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
@@ -31,6 +38,7 @@ import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import stat
|
||||
import subprocess
|
||||
@@ -44,12 +52,24 @@ import inputs # noqa: E402
|
||||
import report # noqa: E402
|
||||
import score # noqa: E402
|
||||
|
||||
DEFAULT_SITE = os.path.join(HERE, "site.json")
|
||||
try:
|
||||
import model_check # optional: R-free of the deposited model against our merge (open arm)
|
||||
except ImportError:
|
||||
model_check = None
|
||||
|
||||
PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$")
|
||||
|
||||
LOCAL_SITE = os.path.join(HERE, "site.json")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ configuration and manifests
|
||||
|
||||
def load_site(path):
|
||||
path = path or os.environ.get("JFJOCH_BATTERY_SITE") or LOCAL_SITE
|
||||
if not os.path.exists(path):
|
||||
sys.exit(f"no site config at {path}: copy {os.path.join(HERE, 'site.example.json')} to "
|
||||
f"{LOCAL_SITE} (or anywhere, and point JFJOCH_BATTERY_SITE or --site at it) and "
|
||||
"fill in this machine's paths")
|
||||
site = json.load(open(path))
|
||||
for arm in site["arms"].values():
|
||||
arm["manifest"] = os.path.join(HERE, arm["manifest"]) # an absolute path stays as it is
|
||||
@@ -170,7 +190,8 @@ def run_one(run_dir, binary, e, opts):
|
||||
wd = os.path.join(run_dir, "work", e["arm"], e["id"])
|
||||
os.makedirs(wd)
|
||||
row = {"set": e["id"], "arm": e["arm"], "tags": e.get("tags", []), "input": e["input_path"],
|
||||
"gpu_others": gpu_others()}
|
||||
"gpu_others": gpu_others(), "rfree": None, "rwork": None, "rfree_deposited": None,
|
||||
"rfree_ratio": None, "model_status": None, "model_reason": None}
|
||||
if not os.path.exists(e["input_path"]):
|
||||
row.update(score.judge(e, {}, "no input"), exit_code=None, elapsed_s=None, wall_s=None, cmd=None)
|
||||
return row
|
||||
@@ -192,9 +213,32 @@ def run_one(run_dir, binary, e, opts):
|
||||
# rugnux's own WALL_TIME where it wrote one: the runner's clock also counts the wait for a
|
||||
# GPU slot under --gpulock
|
||||
row["wall_s"] = row["rugnux_wall_s"] if row["rugnux_wall_s"] is not None else row["elapsed_s"]
|
||||
if opts.get("model_check") and e["arm"] == "open":
|
||||
row.update(check_model(e, wd))
|
||||
return row
|
||||
|
||||
|
||||
def check_model(e, wd):
|
||||
"""R-free of the deposited model against this run's merge (model_check.py). A set id may
|
||||
carry a suffix (6h2p_native); the PDB entry is the part before it."""
|
||||
out = {"rfree": None, "rwork": None, "rfree_deposited": None, "rfree_ratio": None}
|
||||
pdb = e["id"].split("_")[0]
|
||||
mtz = os.path.join(wd, "p.mtz")
|
||||
if not PDB_ID.match(pdb):
|
||||
return dict(out, model_status="skipped", model_reason="not a PDB entry")
|
||||
if not os.path.exists(mtz):
|
||||
return dict(out, model_status="skipped", model_reason="no merged MTZ")
|
||||
try:
|
||||
res = model_check.check(mtz, pdb, os.path.join(wd, "model_check"))
|
||||
except Exception as ex: # a failing check must not lose the set's processing result
|
||||
return dict(out, model_status="error", model_reason=f"{type(ex).__name__}: {ex}"[:300])
|
||||
out.update({k: res.get(k) for k in ("rfree", "rwork", "rfree_deposited")},
|
||||
model_status=res.get("status"), model_reason=res.get("reason"))
|
||||
if out["rfree"] and out["rfree_deposited"]:
|
||||
out["rfree_ratio"] = round(out["rfree"] / out["rfree_deposited"], 4)
|
||||
return out
|
||||
|
||||
|
||||
def make_read_only(path):
|
||||
for dp, dirs, files in os.walk(path):
|
||||
for fn in files:
|
||||
@@ -219,12 +263,13 @@ def write_manifest(path, man):
|
||||
fh.write(head + ', "sets": [\n ' + ",\n ".join(lines) + "\n]}\n")
|
||||
|
||||
|
||||
def write_report(run_dir, baseline=None, out_dir=None):
|
||||
"""report.md and report.html (report_PRIVATE.* for a private run) into out_dir, by default
|
||||
the run directory itself."""
|
||||
def write_report(run_dir, baseline=None, out_dir=None, allow_incomplete=False):
|
||||
"""report.md and report.html (report_PRIVATE.* for a private run) in the run directory, or
|
||||
<run name>.md/.html (with _PRIVATE) in out_dir."""
|
||||
man = json.load(open(os.path.join(run_dir, "manifest.json")))
|
||||
base = os.path.join(out_dir or run_dir, "report_PRIVATE" if man.get("private") else "report")
|
||||
doc = report.build(run_dir, baseline)
|
||||
name = os.path.basename(os.path.normpath(run_dir)) if out_dir else "report"
|
||||
base = os.path.join(out_dir or run_dir, name + ("_PRIVATE" if man.get("private") else ""))
|
||||
doc = report.build(run_dir, baseline, allow_incomplete=allow_incomplete or not out_dir)
|
||||
open(base + ".md", "w").write(doc.markdown())
|
||||
open(base + ".html", "w").write(doc.html())
|
||||
return base + ".html"
|
||||
@@ -235,6 +280,8 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
private = any(site["arms"][a].get("private") for a in arms)
|
||||
if private and len(arms) > 1:
|
||||
sys.exit("the private arm runs on its own: its results must not share a run or a report")
|
||||
if baseline:
|
||||
report.check_baseline(baseline, private)
|
||||
lock_fh = None
|
||||
if lock:
|
||||
os.makedirs(os.path.dirname(lock), exist_ok=True)
|
||||
@@ -245,6 +292,7 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
sys.exit(f"another full battery run holds {lock}; refusing to start a second one")
|
||||
|
||||
info = binary_info(binary_src, build_dir)
|
||||
label = label or info.get("version") or "run"
|
||||
stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M")
|
||||
run_dir = os.path.join(opts["runs_root"],
|
||||
f"{stamp}_{info['git']}_{label}" + ("_private" if private else ""))
|
||||
@@ -261,22 +309,28 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
"binary": info,
|
||||
"runner_git": git(HERE, "rev-parse", "--short", "HEAD") or "unknown",
|
||||
"runner_dirty": bool(git(HERE, "status", "--porcelain", "--", ".")),
|
||||
"host": socket.gethostname(), "start": datetime.datetime.now().isoformat(timespec="seconds"),
|
||||
"end": None, "complete": False, "options": {k: v for k, v in opts.items() if k != "runs_root"},
|
||||
"host": socket.gethostname(), "pid": os.getpid(), "start": datetime.datetime.now().isoformat(timespec="seconds"),
|
||||
"end": None, "complete": False, "results_schema": report.RESULTS_SCHEMA, "options": {k: v for k, v in opts.items() if k != "runs_root"},
|
||||
"gpu_shared": bool(opts["gpulock"]), "sets": sets}
|
||||
write_json(os.path.join(run_dir, "manifest.json"), man)
|
||||
print(f"run directory {run_dir}", flush=True)
|
||||
|
||||
results = []
|
||||
for n, e in enumerate(sets, 1):
|
||||
row = run_one(run_dir, binary, e, opts)
|
||||
results.append(row)
|
||||
if row.get("gpu_others"):
|
||||
man["gpu_shared"] = True
|
||||
write_json(os.path.join(run_dir, "results.json"), results)
|
||||
print(f"[{n}/{len(sets)}] {e['arm']:8s} {e['id']:24s} {row['verdict']:8s} "
|
||||
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
|
||||
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
|
||||
# a kill by SIGTERM is recorded like Ctrl-C: the run directory then says it was aborted
|
||||
signal.signal(signal.SIGTERM, interrupted)
|
||||
try:
|
||||
for n, e in enumerate(sets, 1):
|
||||
row = run_one(run_dir, binary, e, opts)
|
||||
results.append(row)
|
||||
if row.get("gpu_others"):
|
||||
man["gpu_shared"] = True
|
||||
write_json(os.path.join(run_dir, "results.json"), results)
|
||||
print(f"[{n}/{len(sets)}] {e['arm']:8s} {e['id']:24s} {row['verdict']:8s} "
|
||||
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
|
||||
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
|
||||
except KeyboardInterrupt:
|
||||
mark_aborted(run_dir, f"interrupted after {len(results)} of {len(sets)} sets")
|
||||
sys.exit(f"aborted; {run_dir} is marked as such")
|
||||
|
||||
man.update(end=datetime.datetime.now().isoformat(timespec="seconds"), complete=True)
|
||||
write_json(os.path.join(run_dir, "manifest.json"), man)
|
||||
@@ -287,6 +341,25 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
return run_dir
|
||||
|
||||
|
||||
def interrupted(*_):
|
||||
raise KeyboardInterrupt
|
||||
|
||||
|
||||
def mark_aborted(run_dir, reason):
|
||||
man = json.load(open(os.path.join(run_dir, "manifest.json")))
|
||||
man["aborted"] = {"at": datetime.datetime.now().isoformat(timespec="seconds"), "reason": reason}
|
||||
write_json(os.path.join(run_dir, "manifest.json"), man)
|
||||
make_read_only(run_dir)
|
||||
|
||||
|
||||
def default_baseline(site, arms, given):
|
||||
"""--baseline, else the site's persisted baseline for this kind of run ('none' = no delta)."""
|
||||
if given:
|
||||
return None if given == "none" else given
|
||||
private = any(site["arms"][a].get("private") for a in arms)
|
||||
return site.get("baseline_private" if private else "baseline") or None
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ subcommands
|
||||
|
||||
def cmd_run(a, site):
|
||||
@@ -298,16 +371,47 @@ def cmd_run(a, site):
|
||||
if not sets:
|
||||
sys.exit("nothing selected")
|
||||
subset = a.only or (f"tier {a.tier}" if a.tier else None)
|
||||
if a.model_check and not model_check:
|
||||
sys.exit("--model-check: tools/battery/model_check.py is not there")
|
||||
use_model = a.model_check if a.model_check is not None else bool(model_check)
|
||||
opts = {"threads": a.threads, "xds_range": not a.unforced, "timeout": a.timeout,
|
||||
"model_check": use_model and "open" in arms,
|
||||
"gpulock": site["gpulock"] if a.gpulock else None, "extra_args": a.extra.split(),
|
||||
"runs_root": a.runs_root or site["runs_root"]}
|
||||
run_sets(site, sets, arms, a.rugnux, a.build_dir, a.label, opts, subset, a.baseline,
|
||||
lock=None if subset else site["lock_file"])
|
||||
if a.gpulock and not site.get("gpulock"):
|
||||
sys.exit("--gpulock: the site config names no gpulock")
|
||||
run_sets(site, sets, arms, a.rugnux, a.build_dir, a.label, opts, subset,
|
||||
default_baseline(site, arms, a.baseline), lock=None if subset else site["lock_file"])
|
||||
|
||||
|
||||
def cmd_list(a, site):
|
||||
root = a.runs_root or site["runs_root"]
|
||||
for d in sorted(os.listdir(root)):
|
||||
run = os.path.join(root, d)
|
||||
if not os.path.exists(os.path.join(run, "manifest.json")):
|
||||
continue
|
||||
man = json.load(open(os.path.join(run, "manifest.json")))
|
||||
state = report.run_state(man)
|
||||
n = len(json.load(open(os.path.join(run, "results.json")))) \
|
||||
if os.path.exists(os.path.join(run, "results.json")) else 0
|
||||
mark = " <- baseline" if run in (site.get("baseline"), site.get("baseline_private")) else ""
|
||||
print(f"{d:48s} {state:10s} {n:3d}/{len(man['sets']):3d} sets "
|
||||
f"{man.get('subset') or 'full arm':20.20s} {','.join(man['arms'])}{mark}")
|
||||
|
||||
|
||||
def cmd_abort(a, site):
|
||||
man = json.load(open(os.path.join(a.run, "manifest.json")))
|
||||
state = report.run_state(man)
|
||||
if state not in ("unfinished", "running") or (state == "running" and not a.force):
|
||||
sys.exit(f"{a.run} is {state}; only an unfinished run can be marked aborted "
|
||||
"(a live one needs --force)")
|
||||
mark_aborted(a.run, a.reason)
|
||||
print(f"{a.run}: marked aborted ({a.reason})")
|
||||
|
||||
|
||||
def cmd_compare(a, site):
|
||||
man_a, res_a = report.load_run(a.run_a)
|
||||
man_b, res_b = report.load_run(a.run_b)
|
||||
man_a, res_a = report.load_run(a.run_a, a.allow_incomplete)
|
||||
man_b, res_b = report.load_run(a.run_b, a.allow_incomplete)
|
||||
rows = report.compare_rows(res_a, res_b)
|
||||
doc = report.Doc(f"{os.path.basename(os.path.normpath(a.run_a))} vs "
|
||||
f"{os.path.basename(os.path.normpath(a.run_b))}")
|
||||
@@ -326,6 +430,7 @@ def cmd_compare(a, site):
|
||||
return
|
||||
sets = [e for e in man_a["sets"] if (e["arm"], e["id"]) in changed]
|
||||
opts = dict(man_a["options"], runs_root=a.runs_root or site["runs_root"])
|
||||
opts["gpulock"] = opts["gpulock"] and site.get("gpulock")
|
||||
label = "rerunA-" + man_a["label"]
|
||||
rerun = run_sets(site, sets, man_a["arms"], os.path.join(a.run_a, "bin", "rugnux"), None, label,
|
||||
opts, ",".join(e["id"] for e in sets))
|
||||
@@ -346,9 +451,14 @@ def cmd_compare(a, site):
|
||||
|
||||
|
||||
def cmd_report(a, site):
|
||||
# a finished run directory is read-only, so a re-render goes elsewhere
|
||||
# a finished run directory is read-only, so a re-render goes elsewhere, named after the run
|
||||
man, _ = report.load_run(a.run, a.allow_incomplete)
|
||||
repo = git(HERE, "rev-parse", "--show-toplevel")
|
||||
out = os.path.realpath(a.out)
|
||||
if man.get("private") and repo and (out + os.sep).startswith(os.path.realpath(repo) + os.sep):
|
||||
sys.exit("a private run's report must not be written into the repository")
|
||||
os.makedirs(a.out, exist_ok=True)
|
||||
print(write_report(a.run, a.baseline, a.out))
|
||||
print(write_report(a.run, a.baseline, a.out, a.allow_incomplete))
|
||||
|
||||
|
||||
def cmd_discover(a, site):
|
||||
@@ -356,20 +466,22 @@ def cmd_discover(a, site):
|
||||
--refresh, also show where today's discovery disagrees with the manifest's (unpinned) input."""
|
||||
cfg = site["arms"]["open"]
|
||||
root = cfg["data_root"]
|
||||
sets = {e["id"]: e for e in load_sets(site, "open")}
|
||||
by_dir = {} # a directory can hold several sets (6h2p_native, 6h2p_1p89A)
|
||||
for e in load_sets(site, "open"):
|
||||
by_dir.setdefault(e["input"].split("/")[0], []).append(e)
|
||||
for d in sorted(os.listdir(root)):
|
||||
if d.startswith(".") or not os.path.isdir(os.path.join(root, d)):
|
||||
continue
|
||||
e = sets.get(d)
|
||||
if e and (not a.refresh or e.get("pinned")):
|
||||
sets = by_dir.get(d, [])
|
||||
if (sets and not a.refresh) or len(sets) > 1 or (sets and sets[0].get("pinned")):
|
||||
continue
|
||||
path, fmt, note = inputs.find_input(os.path.join(root, d))
|
||||
rel = os.path.relpath(path, root) if path else None
|
||||
if not e:
|
||||
if not sets:
|
||||
print(f"NEW {d:10s} {rel} [{note}]")
|
||||
elif rel != e["input"]:
|
||||
print(f"DIFF {d:10s} manifest {e['input']}\n {'':10s} discover {rel} [{note}]")
|
||||
for d in sorted(set(sets) - set(os.listdir(root))):
|
||||
elif rel != sets[0]["input"]:
|
||||
print(f"DIFF {d:10s} manifest {sets[0]['input']}\n {'':10s} discover {rel} [{note}]")
|
||||
for d in sorted(set(by_dir) - set(os.listdir(root))):
|
||||
print(f"GONE {d:10s} in the manifest, not on disk")
|
||||
|
||||
|
||||
@@ -385,22 +497,65 @@ def cmd_refs(a, site):
|
||||
print(f"{e['id']:24s} no CORRECT.LP")
|
||||
continue
|
||||
new = inputs.parse_correct_lp(lp)
|
||||
if new != e.get("ref"):
|
||||
print(f"{e['id']:24s} {e.get('ref')}\n{'':24s} -> {new}")
|
||||
old = e.get("ref") or {}
|
||||
diff = [f"{k} {old.get(k)} -> {new.get(k)}" for k in sorted(set(old) | set(new))
|
||||
if old.get(k) != new.get(k)]
|
||||
if diff:
|
||||
print(f"{e['id']:24s} " + "; ".join(diff))
|
||||
e["ref"] = new
|
||||
if a.write:
|
||||
write_manifest(cfg["manifest"], man)
|
||||
|
||||
|
||||
def cmd_remap(a, site):
|
||||
"""Follow a rename of data directories. MAP is a JSON object {old: new}: a string renames the
|
||||
set of that id and, wherever it is the first directory of an input, that directory; an object
|
||||
{"id": ..., "input": ...} gives one set's new id and input outright. A renamed set's old id is
|
||||
kept under "aliases", so runs made before the rename still compare set by set."""
|
||||
cfg = site["arms"][a.arm]
|
||||
man = json.load(open(cfg["manifest"]))
|
||||
mp = json.load(open(a.map))
|
||||
aliases = man.setdefault("aliases", {})
|
||||
for e in man["sets"]:
|
||||
new = mp.get(e["id"])
|
||||
head, _, rest = e["input"].partition("/")
|
||||
if isinstance(new, dict):
|
||||
new_id, new_input = new.get("id", e["id"]), new.get("input", e["input"])
|
||||
else:
|
||||
new_id = new or e["id"]
|
||||
new_input = (mp[head] if isinstance(mp.get(head), str) else head) + "/" + rest
|
||||
if (new_id, new_input) == (e["id"], e["input"]):
|
||||
continue
|
||||
there = os.path.exists(os.path.join(cfg["data_root"], new_input))
|
||||
print(f"{e['id']:24s} -> {new_id:24s} {new_input}" + ("" if there else " NOT FOUND"))
|
||||
if new_id != e["id"]:
|
||||
for old in [k for k, v in aliases.items() if v == e["id"]]:
|
||||
aliases[old] = new_id
|
||||
aliases[e["id"]] = new_id
|
||||
e["id"], e["input"] = new_id, new_input
|
||||
if a.write:
|
||||
write_manifest(cfg["manifest"], man)
|
||||
|
||||
|
||||
def load_aliases(site):
|
||||
"""{arm: {old set id: current id}} from the manifests, applied when runs are compared."""
|
||||
out = {}
|
||||
for arm, cfg in site["arms"].items():
|
||||
if os.path.exists(cfg["manifest"]):
|
||||
out[arm] = json.load(open(cfg["manifest"])).get("aliases", {})
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
||||
ap.add_argument("--site", default=os.environ.get("BATTERY_SITE", DEFAULT_SITE),
|
||||
help="site config (data roots, runs root, private manifest, lock file)")
|
||||
ap.add_argument("--site", help="site config (default: $JFJOCH_BATTERY_SITE, else site.json "
|
||||
"beside this script)")
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
r = sub.add_parser("run")
|
||||
r.add_argument("--rugnux", required=True)
|
||||
r.add_argument("--label", required=True, help="short name, part of the run directory's name")
|
||||
r.add_argument("--label", help="short name, part of the run directory's name "
|
||||
"(default: the rugnux version)")
|
||||
r.add_argument("--arm", default="open,inhouse", help="comma-separated (default: open,inhouse)")
|
||||
r.add_argument("--only", help="comma-separated set ids")
|
||||
r.add_argument("--tier", help="named subset from the manifests, e.g. smoke")
|
||||
@@ -411,8 +566,12 @@ def main():
|
||||
r.add_argument("--unforced", action="store_true",
|
||||
help="do not force XDS's resolution range on the XDS-referenced arms")
|
||||
r.add_argument("--timeout", type=int, default=10800)
|
||||
r.add_argument("--model-check", action=argparse.BooleanOptionalAction, default=None,
|
||||
help="open arm: R-free of the deposited model against each merge "
|
||||
"(default: on when model_check.py is present)")
|
||||
r.add_argument("--extra", default="", help="extra rugnux arguments, applied to every set")
|
||||
r.add_argument("--baseline", help="earlier run directory to report the delta against")
|
||||
r.add_argument("--baseline", help="earlier run directory to report the delta against (default: "
|
||||
"the site's persisted baseline; 'none' for no delta)")
|
||||
r.add_argument("--runs-root")
|
||||
|
||||
c = sub.add_parser("compare")
|
||||
@@ -422,23 +581,41 @@ def main():
|
||||
c.add_argument("--rerun-changed", action="store_true",
|
||||
help="rerun the changed sets with A's saved binary to separate real change from noise")
|
||||
c.add_argument("--runs-root")
|
||||
c.add_argument("--allow-incomplete", action="store_true",
|
||||
help="compare a run that did not finish (aborted, killed or still running)")
|
||||
|
||||
p = sub.add_parser("report")
|
||||
p.add_argument("run")
|
||||
p.add_argument("--baseline")
|
||||
p.add_argument("--out", required=True, help="directory for the re-rendered report")
|
||||
p.add_argument("--allow-incomplete", action="store_true",
|
||||
help="render a run that did not finish; the report says so")
|
||||
|
||||
ls = sub.add_parser("list")
|
||||
ls.add_argument("--runs-root")
|
||||
|
||||
ab = sub.add_parser("abort")
|
||||
ab.add_argument("run")
|
||||
ab.add_argument("--reason", default="aborted by hand")
|
||||
ab.add_argument("--force", action="store_true", help="even if its runner still seems alive")
|
||||
|
||||
d = sub.add_parser("discover")
|
||||
d.add_argument("--refresh", action="store_true")
|
||||
|
||||
m = sub.add_parser("remap")
|
||||
m.add_argument("--arm", required=True)
|
||||
m.add_argument("map", help="JSON file {old: new}")
|
||||
m.add_argument("--write", action="store_true")
|
||||
|
||||
f = sub.add_parser("refs")
|
||||
f.add_argument("--arm", required=True)
|
||||
f.add_argument("--write", action="store_true")
|
||||
|
||||
a = ap.parse_args()
|
||||
site = load_site(a.site)
|
||||
{"run": cmd_run, "compare": cmd_compare, "report": cmd_report,
|
||||
"discover": cmd_discover, "refs": cmd_refs}[a.cmd](a, site)
|
||||
report.ALIASES = load_aliases(site)
|
||||
{"run": cmd_run, "list": cmd_list, "compare": cmd_compare, "report": cmd_report,
|
||||
"abort": cmd_abort, "discover": cmd_discover, "refs": cmd_refs, "remap": cmd_remap}[a.cmd](a, site)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+37
-37
@@ -1,38 +1,38 @@
|
||||
{"arm": "inhouse", "sets": [
|
||||
{"id": "E08_lyso_2_006", "input": "E08_lyso_2_006/lyso_2_006_master.h5", "ref": {"sgno": 92, "cell": [76.879, 76.879, 36.961, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.75, "completeness": 99.3, "r_meas": 0.182, "cc_half": 0.995, "multiplicity": 3.51, "dmin": 1.966, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "E08_lyso_2_007", "input": "E08_lyso_2_007/lyso_2_007_master.h5", "ref": {"sgno": 92, "cell": [76.904, 76.904, 36.976, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.14, "completeness": 99.3, "r_meas": 0.18, "cc_half": 0.994, "multiplicity": 3.51, "dmin": 1.973, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_2", "input": "lyso_2/lyso_2_002_master.h5", "ref": {"sgno": 92, "cell": [78.235, 78.235, 37.798, 90.0, 90.0, 90.0], "anomalous": true, "isa": 16.56, "completeness": 99.9, "r_meas": 0.239, "cc_half": 0.998, "multiplicity": 10.97, "dmin": 1.264, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_6_5keV", "input": "lyso_6_5keV/lyso_6_001_master.h5", "ref": {"sgno": 92, "cell": [78.225, 78.225, 37.897, 90.0, 90.0, 90.0], "anomalous": true, "isa": 19.36, "completeness": 86.4, "r_meas": 0.064, "cc_half": 0.998, "multiplicity": 8.85, "dmin": 2.45, "dmax": 50.0}, "tags": ["h5", "lysozyme", "long-wavelength"]},
|
||||
{"id": "lysoC_14_f46f0a", "input": "lysoC_14_f46f0a/lysoC_14_56b5b9_master.h5", "ref": {"sgno": 89, "cell": [78.387, 78.387, 37.427, 90.0, 90.0, 90.0], "anomalous": true, "isa": 23.33, "completeness": 100.0, "r_meas": 0.219, "cc_half": 0.998, "multiplicity": 11.86, "dmin": 1.431, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_ref", "input": "lyso_ref/hewl_35_001_master.h5", "ref": {"sgno": 89, "cell": [78.091, 78.091, 37.704, 90.0, 90.0, 90.0], "anomalous": true, "isa": 28.31, "completeness": 100.0, "r_meas": 0.045, "cc_half": 1.0, "multiplicity": 13.76, "dmin": 1.2, "dmax": 50.0}, "tags": ["h5", "lysozyme"], "tiers": {"smoke": "lysozyme reference set"}},
|
||||
{"id": "lyso_strong", "input": "lyso_strong/lyso_5_f6ac7e_master.h5", "ref": {"sgno": 92, "cell": [77.769, 77.769, 37.442, 90.0, 90.0, 90.0], "anomalous": false, "isa": 8.63, "completeness": 65.8, "r_meas": 0.112, "cc_half": 0.998, "multiplicity": 11.46, "dmin": 1.18, "dmax": 999.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_half_image", "input": "lyso_half_image/lyso3_7_b81d83_master.h5", "ref": {"sgno": 1, "cell": [37.267, 79.125, 81.145, 87.995, 90.636, 90.239], "anomalous": true, "isa": 1.87, "completeness": 43.2, "r_meas": 3.17, "cc_half": 0.179, "multiplicity": 1.44, "dmin": 0.8, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "thau_16keV", "input": "thau_16keV/lyso_4_d2bb9b_master.h5", "ref": {"sgno": 92, "cell": [57.958, 57.958, 150.547, 90.0, 90.0, 90.0], "anomalous": true, "isa": 44.79, "completeness": 94.3, "r_meas": 0.073, "cc_half": 1.0, "multiplicity": 9.76, "dmin": 1.3, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "Thau_9", "input": "Thau_9/Thau_9_002_master.h5", "ref": {"sgno": 92, "cell": [58.488, 58.488, 150.788, 90.0, 90.0, 90.0], "anomalous": true, "isa": 9.15, "completeness": 99.0, "r_meas": 0.23, "cc_half": 0.997, "multiplicity": 11.37, "dmin": 2.197, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "thau_BL1A_1p9A", "input": "thau_BL1A_1p9A/JTH14_1p9_50ds_90T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.031, 58.031, 150.577, 90.0, 90.0, 90.0], "anomalous": true, "isa": 34.46, "completeness": 91.8, "r_meas": 0.072, "cc_half": 0.999, "multiplicity": 7.95, "dmin": 1.78, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]},
|
||||
{"id": "thau_BL1A_2p7A", "input": "thau_BL1A_2p7A/JTH14_2p7_50ds_19T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.007, 58.007, 150.511, 90.0, 90.0, 90.0], "anomalous": true, "isa": 35.64, "completeness": 92.0, "r_meas": 0.062, "cc_half": 0.998, "multiplicity": 7.94, "dmin": 2.53, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]},
|
||||
{"id": "thau_BL1A_3p3A", "input": "thau_BL1A_3p3A/JTH14_3p3_50ds_12T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [57.895, 57.895, 150.213, 90.0, 90.0, 90.0], "anomalous": true, "isa": 30.8, "completeness": 92.2, "r_meas": 0.057, "cc_half": 0.997, "multiplicity": 7.92, "dmin": 3.1, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"], "tiers": {"smoke": "long wavelength (3.3 A), KEK BL-1A thaumatin"}},
|
||||
{"id": "thau_injection", "input": "thau_injection/lyso_13_0c7e7a_master.h5", "ref": {"sgno": 92, "cell": [57.922, 57.922, 150.678, 90.0, 90.0, 90.0], "anomalous": true, "isa": 36.23, "completeness": 86.2, "r_meas": 0.037, "cc_half": 1.0, "multiplicity": 9.96, "dmin": 1.28, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "thau_pink", "input": "thau_pink/Thau-Thaumatin_2_master.h5", "ref": {"sgno": 92, "cell": [58.784, 58.784, 152.226, 90.0, 90.0, 90.0], "anomalous": true, "isa": 21.18, "completeness": 98.5, "r_meas": 0.08, "cc_half": 0.999, "multiplicity": 9.38, "dmin": 1.4, "dmax": 50.0}, "tags": ["h5", "thaumatin", "pink-beam"], "tiers": {"smoke": "pink beam"}},
|
||||
{"id": "Ins_H_2_fa9841", "input": "Ins_H_2_fa9841/Ins_H_2_fa9841_master.h5", "ref": {"sgno": 146, "cell": [81.393, 81.393, 33.272, 90.0, 90.0, 120.0], "anomalous": true, "isa": 6.82, "completeness": 95.0, "r_meas": 0.118, "cc_half": 0.988, "multiplicity": 3.61, "dmin": 1.455, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "Ins_H_3_d05684", "input": "Ins_H_3_d05684/Ins_H_3_d05684_master.h5", "ref": {"sgno": 146, "cell": [81.378, 81.378, 33.269, 90.0, 90.0, 120.0], "anomalous": true, "isa": 17.69, "completeness": 97.4, "r_meas": 0.068, "cc_half": 0.998, "multiplicity": 3.52, "dmin": 1.544, "dmax": 50.0}, "tags": ["h5", "insulin"], "tiers": {"smoke": "rhombohedral insulin (H3)"}},
|
||||
{"id": "Ins_I_16", "input": "Ins_I_16/Ins_16_776a92_master.h5", "ref": {"sgno": 197, "cell": [77.085, 77.085, 77.085, 90.0, 90.0, 90.0], "anomalous": false, "isa": 18.91, "completeness": 92.4, "r_meas": 0.401, "cc_half": 0.999, "multiplicity": 29.04, "dmin": 1.08, "dmax": 999.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "Ins_I_2_1a530d", "input": "Ins_I_2_1a530d/Ins_I_2_1a530d_master.h5", "ref": {"sgno": 197, "cell": [77.582, 77.582, 77.582, 90.0, 90.0, 90.0], "anomalous": true, "isa": 4.15, "completeness": 90.8, "r_meas": 0.22, "cc_half": 0.998, "multiplicity": 13.33, "dmin": 1.3, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "Ins_I_3_5e90e3", "input": "Ins_I_3_5e90e3/Ins_I_3_5e90e3_master.h5", "ref": {"sgno": 197, "cell": [77.52, 77.52, 77.52, 90.0, 90.0, 90.0], "anomalous": true, "isa": 25.08, "completeness": 100.0, "r_meas": 0.349, "cc_half": 0.998, "multiplicity": 13.82, "dmin": 1.621, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "Ins_I_6_13keV", "input": "Ins_I_6_13keV/Ins_I_6_001_master.h5", "ref": {"sgno": 197, "cell": [77.468, 77.468, 77.468, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.83, "completeness": 100.0, "r_meas": 0.279, "cc_half": 0.998, "multiplicity": 17.21, "dmin": 1.635, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "Ins_I_6_5keV", "input": "Ins_I_6_5keV/Ins_I_6_006_master.h5", "ref": {"sgno": 197, "cell": [77.785, 77.785, 77.785, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.54, "completeness": 91.7, "r_meas": 0.073, "cc_half": 0.999, "multiplicity": 11.17, "dmin": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]},
|
||||
{"id": "Ins_I_6_6keV", "input": "Ins_I_6_6keV/Ins_I_6_004_master.h5", "ref": {"sgno": 197, "cell": [77.71, 77.71, 77.71, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.95, "completeness": 92.4, "r_meas": 0.065, "cc_half": 0.999, "multiplicity": 12.42, "dmin": 2.04, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]},
|
||||
{"id": "Ins_I_7_5keV", "input": "Ins_I_7_5keV/Ins_I_7_001_master.h5", "ref": {"sgno": 197, "cell": [77.694, 77.694, 77.694, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.01, "completeness": 91.8, "r_meas": 0.076, "cc_half": 0.999, "multiplicity": 12.73, "dmin": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"], "tiers": {"smoke": "cubic insulin at 5 keV (long wavelength), fast"}},
|
||||
{"id": "cytC_10", "input": "cytC_10/cytC_10_002_master.h5", "ref": {"sgno": 152, "cell": [83.659, 83.659, 86.011, 90.0, 90.0, 120.0], "anomalous": true, "isa": 31.75, "completeness": 99.8, "r_meas": 0.232, "cc_half": 0.999, "multiplicity": 10.65, "dmin": 2.039, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "cytC_2_0770b5", "input": "cytC_2_0770b5/cytC_2_0770b5_master.h5", "ref": {"sgno": 152, "cell": [83.836, 83.836, 86.733, 90.0, 90.0, 120.0], "anomalous": true, "isa": 24.49, "completeness": 99.9, "r_meas": 0.084, "cc_half": 0.999, "multiplicity": 7.67, "dmin": 1.875, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "cytC_3_874285", "input": "cytC_3_874285/cytC_3_874285_master.h5", "ref": {"sgno": 152, "cell": [83.719, 83.719, 88.617, 90.0, 90.0, 120.0], "anomalous": true, "isa": 26.98, "completeness": 99.9, "r_meas": 0.097, "cc_half": 0.999, "multiplicity": 7.82, "dmin": 1.69, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "MyoB_13", "input": "MyoB_13/MyoB_13_110ee0_master.h5", "ref": {"sgno": 4, "cell": [35.229, 28.461, 63.076, 90.0, 106.504, 90.0], "anomalous": true, "isa": 5.21, "completeness": 97.7, "r_meas": 0.247, "cc_half": 0.965, "multiplicity": 3.29, "dmin": 1.742, "dmax": 50.0}, "tags": ["h5", "myoglobin"]},
|
||||
{"id": "MyoB_9_446b70", "input": "MyoB_9_446b70/MyoB_9_446b70_master.h5", "ref": {"sgno": 3, "cell": [35.182, 28.544, 62.83, 90.0, 106.138, 90.0], "anomalous": true, "isa": 7.59, "completeness": 99.4, "r_meas": 0.232, "cc_half": 0.987, "multiplicity": 3.47, "dmin": 1.422, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "tiers": {"smoke": "monoclinic myoglobin, screw not arbitrable by XDS"}},
|
||||
{"id": "MyoB2-1", "input": "MyoB2-1/MyoB2-1_75979a_master.h5", "ref": {"sgno": 1, "cell": [28.705, 34.796, 64.373, 74.314, 89.879, 90.057], "anomalous": false, "isa": 5.54, "completeness": 74.2, "r_meas": 0.493, "cc_half": 0.966, "multiplicity": 2.87, "dmin": 1.495, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"], "tiers": {"smoke": "known hard: triclinic twin"}},
|
||||
{"id": "MyoB2-2", "input": "MyoB2-2/MyoB2-2_5ddd7d_master.h5", "ref": {"sgno": 3, "cell": [35.576, 28.811, 63.529, 90.0, 105.593, 90.0], "anomalous": false, "isa": 2.19, "completeness": 58.0, "r_meas": 1.106, "cc_half": 0.655, "multiplicity": 4.71, "dmin": 0.99, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "MyoB2-4", "input": "MyoB2-4/MyoB2-4_079740_master.h5", "ref": {"sgno": 3, "cell": [35.39, 28.783, 63.626, 90.0, 105.544, 90.0], "anomalous": false, "isa": 12.41, "completeness": 77.7, "r_meas": 0.577, "cc_half": 0.984, "multiplicity": 4.38, "dmin": 1.506, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "MyoB2-5", "input": "MyoB2-5/MyoB2-5_7cef9c_master.h5", "ref": {"sgno": 1, "cell": [43.226, 67.346, 121.692, 100.15, 89.855, 96.62], "anomalous": false, "isa": 1.24, "completeness": 11.8, "r_meas": 1.527, "cc_half": 0.125, "multiplicity": 1.41, "dmin": 1.11, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "onlyice", "input": "onlyice/onlyice_master.h5", "expect": "no_lattice", "tags": ["h5", "control"], "tiers": {"smoke": "no-crystal control (ice rings only): must refuse"}},
|
||||
{"id": "nothing_3400ac", "input": "nothing/test-28_3400ac_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]},
|
||||
{"id": "nothing_9bf604", "input": "nothing/test-28_9bf604_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]}
|
||||
{"arm": "inhouse", "aliases": {"E08_lyso_2_006": "lyso_90deg_1", "E08_lyso_2_007": "lyso_90deg_2", "lyso_2": "lyso_atten_wedge", "lyso_6_5keV": "lyso_5keV", "lysoC_14_f46f0a": "lyso_ice", "Thau_9": "thau_0p1deg", "thau_BL1A_1p9A": "thau_1p9A", "thau_BL1A_2p7A": "thau_2p7A", "thau_BL1A_3p3A": "thau_3p3A", "Ins_H_2_fa9841": "insu_H_twin", "Ins_H_3_d05684": "insu_H_notwin", "Ins_I_16": "insu_I_weak", "Ins_I_2_1a530d": "insu_I_low_isa", "Ins_I_3_5e90e3": "insu_I_ref", "Ins_I_6_13keV": "insu_I_13keV", "Ins_I_6_5keV": "insu_I_5keV", "Ins_I_6_6keV": "insu_I_6keV", "Ins_I_7_5keV": "insu_I_5keV_2", "cytC_10": "cytc_eiger", "cytC_2_0770b5": "cytc_pilatus_1", "cytC_3_874285": "cytc_pilatus_2", "MyoB_13": "myob_eiger", "MyoB_9_446b70": "myob_pilatus", "MyoB2-1": "myob_powder_1", "MyoB2-2": "myob_powder_2", "MyoB2-4": "myob_split", "MyoB2-5": "myob_aggregate", "onlyice": "nothing_ice", "nothing_3400ac": "nothing_2", "nothing_9bf604": "nothing_1"}, "sets": [
|
||||
{"id": "lyso_90deg_1", "input": "lyso_90deg_1/lyso_2_006_master.h5", "ref": {"sgno": 92, "cell": [76.879, 76.879, 36.961, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.75, "completeness": 99.3, "r_meas": 0.182, "cc_half": 0.995, "multiplicity": 3.51, "dmin": 1.966, "dmin_rule": "xds_range", "dmin_xds": 1.966, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_90deg_2", "input": "lyso_90deg_2/lyso_2_007_master.h5", "ref": {"sgno": 92, "cell": [76.904, 76.904, 36.976, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.14, "completeness": 99.3, "r_meas": 0.18, "cc_half": 0.994, "multiplicity": 3.51, "dmin": 1.973, "dmin_rule": "xds_range", "dmin_xds": 1.973, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_atten_wedge", "input": "lyso_atten_wedge/lyso_2_002_master.h5", "ref": {"sgno": 92, "cell": [78.235, 78.235, 37.798, 90.0, 90.0, 90.0], "anomalous": true, "isa": 16.56, "completeness": 99.9, "r_meas": 0.239, "cc_half": 0.998, "multiplicity": 10.97, "dmin": 1.264, "dmin_rule": "xds_range", "dmin_xds": 1.264, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_5keV", "input": "lyso_5keV/lyso_6_001_master.h5", "ref": {"sgno": 92, "cell": [78.225, 78.225, 37.897, 90.0, 90.0, 90.0], "anomalous": true, "isa": 19.36, "completeness": 86.4, "r_meas": 0.064, "cc_half": 0.998, "multiplicity": 8.85, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "lysozyme", "long-wavelength"]},
|
||||
{"id": "lyso_ice", "input": "lyso_ice/lysoC_14_56b5b9_master.h5", "ref": {"sgno": 89, "cell": [78.387, 78.387, 37.427, 90.0, 90.0, 90.0], "anomalous": true, "isa": 23.33, "completeness": 100.0, "r_meas": 0.219, "cc_half": 0.998, "multiplicity": 11.86, "dmin": 1.431, "dmin_rule": "xds_range", "dmin_xds": 1.431, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_ref", "input": "lyso_ref/hewl_35_001_master.h5", "ref": {"sgno": 89, "cell": [78.091, 78.091, 37.704, 90.0, 90.0, 90.0], "anomalous": true, "isa": 28.31, "completeness": 100.0, "r_meas": 0.045, "cc_half": 1.0, "multiplicity": 13.76, "dmin": 1.2, "dmin_rule": "xds_range", "dmin_xds": 1.2, "dmax": 50.0}, "tags": ["h5", "lysozyme"], "tiers": {"smoke": "lysozyme reference set"}},
|
||||
{"id": "lyso_strong", "input": "lyso_strong/lyso_5_f6ac7e_master.h5", "ref": {"sgno": 92, "cell": [77.769, 77.769, 37.442, 90.0, 90.0, 90.0], "anomalous": false, "isa": 8.63, "completeness": 65.8, "r_meas": 0.112, "cc_half": 0.998, "multiplicity": 11.46, "dmin": 1.24, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.18, "dmax": 999.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "lyso_half_image", "input": "lyso_half_image/lyso3_7_b81d83_master.h5", "ref": {"sgno": 1, "cell": [37.267, 79.125, 81.145, 87.995, 90.636, 90.239], "anomalous": true, "isa": 1.87, "completeness": 43.2, "r_meas": 3.17, "cc_half": 0.179, "multiplicity": 1.44, "dmin": 2.602, "dmin_rule": "cc_half_0.30", "dmin_xds": 0.8, "dmax": 50.0}, "tags": ["h5", "lysozyme"]},
|
||||
{"id": "thau_16keV", "input": "thau_16keV/lyso_4_d2bb9b_master.h5", "ref": {"sgno": 92, "cell": [57.958, 57.958, 150.547, 90.0, 90.0, 90.0], "anomalous": true, "isa": 44.79, "completeness": 94.3, "r_meas": 0.073, "cc_half": 1.0, "multiplicity": 9.76, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "thau_0p1deg", "input": "thau_0p1deg/Thau_9_002_master.h5", "ref": {"sgno": 92, "cell": [58.488, 58.488, 150.788, 90.0, 90.0, 90.0], "anomalous": true, "isa": 9.15, "completeness": 99.0, "r_meas": 0.23, "cc_half": 0.997, "multiplicity": 11.37, "dmin": 2.197, "dmin_rule": "xds_range", "dmin_xds": 2.197, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "thau_1p9A", "input": "thau_1p9A/JTH14_1p9_50ds_90T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.031, 58.031, 150.577, 90.0, 90.0, 90.0], "anomalous": true, "isa": 34.46, "completeness": 91.8, "r_meas": 0.072, "cc_half": 0.999, "multiplicity": 7.95, "dmin": 1.78, "dmin_rule": "xds_range", "dmin_xds": 1.78, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]},
|
||||
{"id": "thau_2p7A", "input": "thau_2p7A/JTH14_2p7_50ds_19T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [58.007, 58.007, 150.511, 90.0, 90.0, 90.0], "anomalous": true, "isa": 35.64, "completeness": 92.0, "r_meas": 0.062, "cc_half": 0.998, "multiplicity": 7.94, "dmin": 2.53, "dmin_rule": "xds_range", "dmin_xds": 2.53, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"]},
|
||||
{"id": "thau_3p3A", "input": "thau_3p3A/JTH14_3p3_50ds_12T_k0_JF4M_01_50001_master.h5", "ref": {"sgno": 89, "cell": [57.895, 57.895, 150.213, 90.0, 90.0, 90.0], "anomalous": true, "isa": 30.8, "completeness": 92.2, "r_meas": 0.057, "cc_half": 0.997, "multiplicity": 7.92, "dmin": 3.1, "dmin_rule": "xds_range", "dmin_xds": 3.1, "dmax": 100.0}, "tags": ["h5", "thaumatin", "long-wavelength"], "tiers": {"smoke": "long wavelength (3.3 A), KEK BL-1A thaumatin"}},
|
||||
{"id": "thau_injection", "input": "thau_injection/lyso_13_0c7e7a_master.h5", "ref": {"sgno": 92, "cell": [57.922, 57.922, 150.678, 90.0, 90.0, 90.0], "anomalous": true, "isa": 36.23, "completeness": 86.2, "r_meas": 0.037, "cc_half": 1.0, "multiplicity": 9.96, "dmin": 1.28, "dmin_rule": "xds_range", "dmin_xds": 1.28, "dmax": 50.0}, "tags": ["h5", "thaumatin"]},
|
||||
{"id": "thau_pink", "input": "thau_pink/Thau-Thaumatin_2_master.h5", "ref": {"sgno": 92, "cell": [58.784, 58.784, 152.226, 90.0, 90.0, 90.0], "anomalous": true, "isa": 21.18, "completeness": 98.5, "r_meas": 0.08, "cc_half": 0.999, "multiplicity": 9.38, "dmin": 1.4, "dmin_rule": "xds_range", "dmin_xds": 1.4, "dmax": 50.0}, "tags": ["h5", "thaumatin", "pink-beam"], "tiers": {"smoke": "pink beam"}},
|
||||
{"id": "insu_H_twin", "input": "insu_H_twin/Ins_H_2_fa9841_master.h5", "ref": {"sgno": 146, "cell": [81.393, 81.393, 33.272, 90.0, 90.0, 120.0], "anomalous": true, "isa": 6.82, "completeness": 95.0, "r_meas": 0.118, "cc_half": 0.988, "multiplicity": 3.61, "dmin": 1.455, "dmin_rule": "xds_range", "dmin_xds": 1.455, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "insu_H_notwin", "input": "insu_H_notwin/Ins_H_3_d05684_master.h5", "ref": {"sgno": 146, "cell": [81.378, 81.378, 33.269, 90.0, 90.0, 120.0], "anomalous": true, "isa": 17.69, "completeness": 97.4, "r_meas": 0.068, "cc_half": 0.998, "multiplicity": 3.52, "dmin": 1.544, "dmin_rule": "xds_range", "dmin_xds": 1.544, "dmax": 50.0}, "tags": ["h5", "insulin"], "tiers": {"smoke": "rhombohedral insulin (H3)"}},
|
||||
{"id": "insu_I_weak", "input": "insu_I_weak/Ins_16_776a92_master.h5", "ref": {"sgno": 197, "cell": [77.085, 77.085, 77.085, 90.0, 90.0, 90.0], "anomalous": false, "isa": 18.91, "completeness": 92.4, "r_meas": 0.401, "cc_half": 0.999, "multiplicity": 29.04, "dmin": 1.807, "dmin_rule": "cc_half_0.30", "dmin_xds": 1.08, "dmax": 999.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "insu_I_low_isa", "input": "insu_I_low_isa/Ins_I_2_1a530d_master.h5", "ref": {"sgno": 197, "cell": [77.582, 77.582, 77.582, 90.0, 90.0, 90.0], "anomalous": true, "isa": 4.15, "completeness": 90.8, "r_meas": 0.22, "cc_half": 0.998, "multiplicity": 13.33, "dmin": 1.3, "dmin_rule": "xds_range", "dmin_xds": 1.3, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "insu_I_ref", "input": "insu_I_ref/Ins_I_3_5e90e3_master.h5", "ref": {"sgno": 197, "cell": [77.52, 77.52, 77.52, 90.0, 90.0, 90.0], "anomalous": true, "isa": 25.08, "completeness": 100.0, "r_meas": 0.349, "cc_half": 0.998, "multiplicity": 13.82, "dmin": 1.621, "dmin_rule": "xds_range", "dmin_xds": 1.621, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "insu_I_13keV", "input": "insu_I_13keV/Ins_I_6_001_master.h5", "ref": {"sgno": 197, "cell": [77.468, 77.468, 77.468, 90.0, 90.0, 90.0], "anomalous": true, "isa": 18.83, "completeness": 100.0, "r_meas": 0.279, "cc_half": 0.998, "multiplicity": 17.21, "dmin": 1.635, "dmin_rule": "xds_range", "dmin_xds": 1.635, "dmax": 50.0}, "tags": ["h5", "insulin"]},
|
||||
{"id": "insu_I_5keV", "input": "insu_I_5keV/Ins_I_6_006_master.h5", "ref": {"sgno": 197, "cell": [77.785, 77.785, 77.785, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.54, "completeness": 91.7, "r_meas": 0.073, "cc_half": 0.999, "multiplicity": 11.17, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]},
|
||||
{"id": "insu_I_6keV", "input": "insu_I_6keV/Ins_I_6_004_master.h5", "ref": {"sgno": 197, "cell": [77.71, 77.71, 77.71, 90.0, 90.0, 90.0], "anomalous": true, "isa": 17.95, "completeness": 92.4, "r_meas": 0.065, "cc_half": 0.999, "multiplicity": 12.42, "dmin": 2.04, "dmin_rule": "xds_range", "dmin_xds": 2.04, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"]},
|
||||
{"id": "insu_I_5keV_2", "input": "insu_I_5keV_2/Ins_I_7_001_master.h5", "ref": {"sgno": 197, "cell": [77.694, 77.694, 77.694, 90.0, 90.0, 90.0], "anomalous": true, "isa": 20.01, "completeness": 91.8, "r_meas": 0.076, "cc_half": 0.999, "multiplicity": 12.73, "dmin": 2.45, "dmin_rule": "xds_range", "dmin_xds": 2.45, "dmax": 50.0}, "tags": ["h5", "insulin", "long-wavelength"], "tiers": {"smoke": "cubic insulin at 5 keV (long wavelength), fast"}},
|
||||
{"id": "cytc_eiger", "input": "cytc_eiger/cytC_10_002_master.h5", "ref": {"sgno": 152, "cell": [83.659, 83.659, 86.011, 90.0, 90.0, 120.0], "anomalous": true, "isa": 31.75, "completeness": 99.8, "r_meas": 0.232, "cc_half": 0.999, "multiplicity": 10.65, "dmin": 2.267, "dmin_rule": "cc_half_0.30", "dmin_xds": 2.039, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "cytc_pilatus_1", "input": "cytc_pilatus_1/cytC_2_0770b5_master.h5", "ref": {"sgno": 152, "cell": [83.836, 83.836, 86.733, 90.0, 90.0, 120.0], "anomalous": true, "isa": 24.49, "completeness": 99.9, "r_meas": 0.084, "cc_half": 0.999, "multiplicity": 7.67, "dmin": 1.875, "dmin_rule": "xds_range", "dmin_xds": 1.875, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "cytc_pilatus_2", "input": "cytc_pilatus_2/cytC_3_874285_master.h5", "ref": {"sgno": 152, "cell": [83.719, 83.719, 88.617, 90.0, 90.0, 120.0], "anomalous": true, "isa": 26.98, "completeness": 99.9, "r_meas": 0.097, "cc_half": 0.999, "multiplicity": 7.82, "dmin": 1.69, "dmin_rule": "xds_range", "dmin_xds": 1.69, "dmax": 50.0}, "tags": ["h5", "cytochrome-c"]},
|
||||
{"id": "myob_eiger", "input": "myob_eiger/MyoB_13_110ee0_master.h5", "ref": {"sgno": 4, "cell": [35.229, 28.461, 63.076, 90.0, 106.504, 90.0], "anomalous": true, "isa": 5.21, "completeness": 97.7, "r_meas": 0.247, "cc_half": 0.965, "multiplicity": 3.29, "dmin": 1.742, "dmin_rule": "xds_range", "dmin_xds": 1.742, "dmax": 50.0}, "tags": ["h5", "myoglobin"]},
|
||||
{"id": "myob_pilatus", "input": "myob_pilatus/MyoB_9_446b70_master.h5", "ref": {"sgno": 3, "cell": [35.182, 28.544, 62.83, 90.0, 106.138, 90.0], "anomalous": true, "isa": 7.59, "completeness": 99.4, "r_meas": 0.232, "cc_half": 0.987, "multiplicity": 3.47, "dmin": 1.422, "dmin_rule": "xds_range", "dmin_xds": 1.422, "dmax": 50.0}, "tags": ["h5", "myoglobin"], "tiers": {"smoke": "monoclinic myoglobin, screw not arbitrable by XDS"}},
|
||||
{"id": "myob_powder_1", "input": "myob_powder_1/MyoB2-1_75979a_master.h5", "ref": {"sgno": 1, "cell": [28.705, 34.796, 64.373, 74.314, 89.879, 90.057], "anomalous": false, "isa": 5.54, "completeness": 74.2, "r_meas": 0.493, "cc_half": 0.966, "multiplicity": 2.87, "dmin": 1.495, "dmin_rule": "xds_range", "dmin_xds": 1.495, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"], "tiers": {"smoke": "known hard: triclinic twin"}},
|
||||
{"id": "myob_powder_2", "input": "myob_powder_2/MyoB2-2_5ddd7d_master.h5", "ref": {"sgno": 3, "cell": [35.576, 28.811, 63.529, 90.0, 105.593, 90.0], "anomalous": false, "isa": 2.19, "completeness": 58.0, "r_meas": 1.106, "cc_half": 0.655, "multiplicity": 4.71, "dmin": 0.99, "dmin_rule": "xds_range", "dmin_xds": 0.99, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "myob_split", "input": "myob_split/MyoB2-4_079740_master.h5", "ref": {"sgno": 3, "cell": [35.39, 28.783, 63.626, 90.0, 105.544, 90.0], "anomalous": false, "isa": 12.41, "completeness": 77.7, "r_meas": 0.577, "cc_half": 0.984, "multiplicity": 4.38, "dmin": 1.506, "dmin_rule": "xds_range", "dmin_xds": 1.506, "dmax": 50.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "myob_aggregate", "input": "MyoB2-5/MyoB2-5_7cef9c_master.h5", "ref": {"sgno": 1, "cell": [43.226, 67.346, 121.692, 100.15, 89.855, 96.62], "anomalous": false, "isa": 1.24, "completeness": 11.8, "r_meas": 1.527, "cc_half": 0.125, "multiplicity": 1.41, "dmin": 1.11, "dmin_rule": "xds_range", "dmin_xds": 1.11, "dmax": 999.0}, "tags": ["h5", "myoglobin", "twin"]},
|
||||
{"id": "nothing_ice", "input": "nothing_ice/onlyice_master.h5", "expect": "no_lattice", "tags": ["h5", "control"], "tiers": {"smoke": "no-crystal control (ice rings only): must refuse"}},
|
||||
{"id": "nothing_2", "input": "nothing_2/test-28_3400ac_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]},
|
||||
{"id": "nothing_1", "input": "nothing_1/test-28_9bf604_master.h5", "expect": "no_lattice", "tags": ["h5", "control"]}
|
||||
]}
|
||||
+41
-5
@@ -106,7 +106,8 @@ def input_format(path):
|
||||
|
||||
def parse_correct_lp(path):
|
||||
"""Reference values from an XDS CORRECT.LP: the space group and cell CORRECT refined, the
|
||||
resolution range it merged, and its overall merging statistics.
|
||||
resolution range it merged (dmin_xds, dmax), the reference d_min read off its statistics
|
||||
(dmin, dmin_rule - see reference_dmin) and its overall merging statistics.
|
||||
|
||||
CORRECT.LP states the group and cell twice - first as the INPUT to INTEGRATE (routinely P1),
|
||||
then as CORRECT's own answer. The last SPACE_GROUP_NUMBER and the refined UNIT CELL PARAMETERS
|
||||
@@ -143,12 +144,47 @@ def parse_correct_lp(path):
|
||||
ref["cc_half"] = round(num(total[10]) / 100, 4)
|
||||
ref["multiplicity"] = round(int(total[1]) / int(total[2]), 2) if int(total[2]) else None
|
||||
|
||||
# The resolution range to hand to rugnux so both programs merge the same reflections: the
|
||||
# explicit INCLUDE_RESOLUTION_RANGE, or - when XDS.INP left the high limit at 0.0, "the whole
|
||||
# detector" - the finest shell actually tabulated.
|
||||
# The range XDS merged: the explicit INCLUDE_RESOLUTION_RANGE, or - when XDS.INP left the high
|
||||
# limit at 0.0, "the whole detector" - the finest shell actually tabulated. The pooled statistics
|
||||
# above are over this range.
|
||||
m = re.search(r"INCLUDE_RESOLUTION_RANGE=\s*([\d.]+)\s+([\d.]+)", txt)
|
||||
include_low, include_high = (float(m.group(1)), float(m.group(2))) if m else (0.0, 0.0)
|
||||
table_high = float(shells[-1][0]) if shells else 0.0
|
||||
ref["dmin"] = (include_high if include_high > 0 else table_high) or None
|
||||
dmin_xds = (include_high if include_high > 0 else table_high) or None
|
||||
ref["dmin"], ref["dmin_rule"] = reference_dmin(
|
||||
[(float(f[0]), num(f[10]) / 100, f[10].endswith("*")) for f in shells], include_low, dmin_xds)
|
||||
ref["dmin_xds"] = dmin_xds
|
||||
ref["dmax"] = include_low or None
|
||||
return ref
|
||||
|
||||
|
||||
# The CC1/2 target rugnux's own resolution cutoff uses (--resolution-cc-target, default 0.30)
|
||||
CC_TARGET = 0.30
|
||||
|
||||
|
||||
def reference_dmin(shells, dmax, dmin_xds):
|
||||
"""(reference d_min, rule) from XDS's shell table [(shell high-resolution limit, CC1/2,
|
||||
CC1/2 significant)], low to high resolution.
|
||||
|
||||
XDS's merged range is a resolution only when someone chose it. Left at "the whole detector"
|
||||
(INCLUDE_RESOLUTION_RANGE 0.0, or a limit at the detector edge) it is the detector's reach, not
|
||||
the crystal's: scored against, it is a meaningless number, and forced on rugnux it makes rugnux
|
||||
merge shells of pure noise. XDS itself says which case it is - it marks each shell's CC1/2 with
|
||||
'*' when it is significant at the 0.1% level:
|
||||
|
||||
xds_range the finest shell's CC1/2 is significant: XDS's limit stands.
|
||||
cc_half_0.30 it is not - XDS merged past its own signal. The reference is then where CC1/2
|
||||
falls through the target rugnux's own cutoff uses, interpolated in 1/d^2
|
||||
between the centres of the last shell above it and the first below it."""
|
||||
if not shells or shells[-1][2]:
|
||||
return dmin_xds, "xds_range"
|
||||
below = next((i for i, (_, cc, _) in enumerate(shells) if cc < CC_TARGET), None)
|
||||
if not below: # None, or already the first shell: no fall-off to read
|
||||
return dmin_xds, "xds_range"
|
||||
s = [1 / d ** 2 for d, _, _ in shells]
|
||||
edges = [1 / dmax ** 2 if dmax else 0.0] + s
|
||||
centre = [(edges[i] + edges[i + 1]) / 2 for i in range(len(s))]
|
||||
cc_hi, cc_lo = shells[below - 1][1], shells[below][1]
|
||||
frac = (cc_hi - CC_TARGET) / (cc_hi - cc_lo)
|
||||
s_cross = centre[below - 1] + frac * (centre[below] - centre[below - 1])
|
||||
return round(1 / s_cross ** 0.5, 3), f"cc_half_{CC_TARGET:.2f}"
|
||||
+82
-2
@@ -7,11 +7,19 @@ import html
|
||||
|
||||
CSS = """
|
||||
:root { --bg:#fff; --fg:#1d1d1f; --mute:#6e6e73; --line:#d9d9de; --head:#f3f3f5;
|
||||
--pass:#1a7f37; --fail:#c62828; --warn:#9a6700; --passbg:#e8f5ec; --failbg:#fdecea; }
|
||||
--pass:#1a7f37; --fail:#c62828; --warn:#9a6700; --passbg:#e8f5ec; --failbg:#fdecea;
|
||||
--series1:#2a78d6; --good:#0ca30c; --critical:#d03b3b; --warning:#fab219;
|
||||
--neutral:#898781; --grid:#e1e0d9; --axis:#c3c2b7; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root { --bg:#161618; --fg:#e8e8ea; --mute:#9a9aa0; --line:#3a3a3e; --head:#232326;
|
||||
--pass:#5cc97a; --fail:#ff7b72; --warn:#e3b341; --passbg:#16301f; --failbg:#3a1a1a; }
|
||||
--pass:#5cc97a; --fail:#ff7b72; --warn:#e3b341; --passbg:#16301f; --failbg:#3a1a1a;
|
||||
--series1:#3987e5; --grid:#2c2c2a; --axis:#383835; }
|
||||
}
|
||||
figure { margin:12px 0 20px; max-width:900px; } figcaption { color:var(--mute); font-size:13px; }
|
||||
svg.plot { width:100%; height:auto; display:block; }
|
||||
svg.plot text { fill:var(--mute); font:11px system-ui, sans-serif; }
|
||||
svg.plot .grid { stroke:var(--grid); stroke-width:1; } svg.plot .ref { stroke:var(--axis); stroke-width:2; }
|
||||
svg.plot circle:hover, svg.plot rect:hover { stroke:var(--fg); stroke-width:2; }
|
||||
body { background:var(--bg); color:var(--fg); margin:0 auto; max-width:1400px; padding:16px;
|
||||
font:14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif; }
|
||||
h1 { font-size:22px; margin:8px 0 4px; } h2 { font-size:18px; margin:28px 0 8px; }
|
||||
@@ -43,6 +51,10 @@ class Doc:
|
||||
def ul(self, items):
|
||||
self.blocks.append(("ul", list(items)))
|
||||
|
||||
def svg(self, markup, caption):
|
||||
"""An inline SVG plot (HTML only; the Markdown names it and points at the HTML)."""
|
||||
self.blocks.append(("svg", markup, caption))
|
||||
|
||||
def table(self, head, rows, num=(), row_class=None, cell_class=None):
|
||||
"""head: column titles; rows: lists of str; num: indices of right-aligned columns;
|
||||
row_class(row) and cell_class(i, value) give optional CSS classes (HTML only)."""
|
||||
@@ -57,6 +69,8 @@ class Doc:
|
||||
out += [b[1], ""]
|
||||
elif b[0] == "ul":
|
||||
out += ["- " + x for x in b[1]] + [""]
|
||||
elif b[0] == "svg":
|
||||
out += [f"_Plot in report.html: {b[2]}_", ""]
|
||||
else:
|
||||
_, head, rows, num, _, _ = b
|
||||
if not rows:
|
||||
@@ -82,6 +96,8 @@ class Doc:
|
||||
out.append(f"<p>{e(b[1])}</p>")
|
||||
elif b[0] == "ul":
|
||||
out.append("<ul>" + "".join(f"<li>{e(x)}</li>" for x in b[1]) + "</ul>")
|
||||
elif b[0] == "svg":
|
||||
out.append(f"<figure>{b[1]}<figcaption>{e(b[2])}</figcaption></figure>")
|
||||
else:
|
||||
_, head, rows, num, row_class, cell_class = b
|
||||
if not rows:
|
||||
@@ -105,3 +121,67 @@ class Doc:
|
||||
out.append("".join(t))
|
||||
out.append("</body></html>")
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ plots (inline SVG, no libraries)
|
||||
|
||||
STATUS = {"pass": "var(--good)", "fail": "var(--critical)", "unscored": "var(--neutral)",
|
||||
"not_run": "var(--warning)"}
|
||||
|
||||
|
||||
def verdict_bars(rows):
|
||||
"""One horizontal bar per arm, split pass / fail / unscored / not run.
|
||||
rows: [(arm, {verdict: count})]."""
|
||||
e = html.escape
|
||||
w, left, bar, gap = 900, 90, 22, 14
|
||||
h = 30 + len(rows) * (bar + gap)
|
||||
total = max(sum(c.values()) for _, c in rows) or 1
|
||||
out = [f'<svg class="plot" viewBox="0 0 {w} {h}" role="img" aria-label="verdicts per arm">']
|
||||
x = left
|
||||
for v, col in STATUS.items(): # legend, above the bars
|
||||
out.append(f'<rect x="{x}" y="4" width="10" height="10" rx="2" fill="{col}"/>'
|
||||
f'<text x="{x + 14}" y="13">{v.replace("_", " ")}</text>')
|
||||
x += 90
|
||||
for i, (arm, cnt) in enumerate(rows):
|
||||
y = 26 + i * (bar + gap)
|
||||
out.append(f'<text x="0" y="{y + bar / 2 + 4}">{e(arm)}</text>')
|
||||
x = left
|
||||
for v, col in STATUS.items():
|
||||
n = cnt.get(v, 0)
|
||||
if not n:
|
||||
continue
|
||||
bw = (w - left - 10) * n / total
|
||||
out.append(f'<rect x="{x:.1f}" y="{y}" width="{max(bw - 2, 1):.1f}" height="{bar}" rx="4" '
|
||||
f'fill="{col}"><title>{e(arm)}: {n} {v.replace("_", " ")}</title></rect>')
|
||||
if bw > 28:
|
||||
out.append(f'<text x="{x + 6:.1f}" y="{y + bar / 2 + 4}" style="fill:#fff">{n}</text>')
|
||||
x += bw
|
||||
out.append("</svg>")
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def ratio_dots(points, what):
|
||||
"""Sets sorted by a ratio (ours / reference), on a log axis clipped to 0.5..2, with the line
|
||||
at 1. points: [(set, ratio, verdict, tooltip)]. Failed sets are drawn in the fail colour."""
|
||||
e = html.escape
|
||||
import math
|
||||
w, h, left, top, bottom = 900, 240, 44, 22, 18
|
||||
lo, hi = math.log(0.5), math.log(2.0)
|
||||
y = lambda r: top + (hi - math.log(min(max(r, 0.5), 2.0))) / (hi - lo) * (h - top - bottom)
|
||||
pts = sorted(points, key=lambda p: p[1])
|
||||
step = (w - left - 10) / max(len(pts), 1)
|
||||
out = [f'<svg class="plot" viewBox="0 0 {w} {h}" role="img" aria-label="{e(what)}">']
|
||||
for t in (0.5, 0.75, 1.0, 1.5, 2.0):
|
||||
out.append(f'<line class="{"ref" if t == 1.0 else "grid"}" x1="{left}" x2="{w}" '
|
||||
f'y1="{y(t):.1f}" y2="{y(t):.1f}"/><text x="0" y="{y(t) + 4:.1f}">{t:g}</text>')
|
||||
out.append(f'<circle cx="{left + 6}" cy="10" r="4" fill="var(--series1)"/>'
|
||||
f'<text x="{left + 14}" y="14">pass or unscored</text>'
|
||||
f'<circle cx="{left + 136}" cy="10" r="4" fill="var(--critical)"/>'
|
||||
f'<text x="{left + 144}" y="14">fail</text>'
|
||||
f'<text x="{left + 190}" y="14">(clipped to 0.5-2; hover a point for the set)</text>')
|
||||
for i, (name, r, verdict, tip) in enumerate(pts):
|
||||
col = "var(--critical)" if verdict == "fail" else "var(--series1)"
|
||||
out.append(f'<circle cx="{left + step * (i + 0.5):.1f}" cy="{y(r):.1f}" r="4" fill="{col}">'
|
||||
f'<title>{e(name)}: {r:.3f} - {e(tip)}</title></circle>')
|
||||
out.append("</svg>")
|
||||
return "".join(out)
|
||||
+95
-10
@@ -1,9 +1,11 @@
|
||||
"""The per-run report and the run-to-run comparison."""
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import statistics
|
||||
import sys
|
||||
|
||||
from render import Doc
|
||||
from render import Doc, ratio_dots, verdict_bars
|
||||
|
||||
# A change smaller than these is treated as run-to-run noise. They are a starting point, to be
|
||||
# re-calibrated from `compare --rerun-changed`, which measures the noise directly.
|
||||
@@ -14,18 +16,59 @@ NOISE = {
|
||||
"cc_half": ("abs", 0.005),
|
||||
"completeness": ("abs", 1.0), # percentage points
|
||||
"cell_dev_pct": ("abs", 0.2), # percentage points
|
||||
"rfree": ("abs", 0.005),
|
||||
"wall_s": ("rel", 0.25), # and at least 10 s - see beyond_noise()
|
||||
}
|
||||
|
||||
|
||||
def load_run(run_dir):
|
||||
def run_state(man):
|
||||
"""complete, aborted (marked so), running (its runner is alive), or unfinished (killed
|
||||
without a trace - treat as aborted)."""
|
||||
if man.get("complete"):
|
||||
return "complete"
|
||||
if man.get("aborted"):
|
||||
return "aborted"
|
||||
if man.get("pid") and man.get("host") == socket.gethostname():
|
||||
try:
|
||||
os.kill(man["pid"], 0)
|
||||
return "running"
|
||||
except OSError:
|
||||
pass
|
||||
return "unfinished"
|
||||
|
||||
|
||||
def load_run(run_dir, allow_incomplete=False):
|
||||
"""A run's manifest and results. A run that did not finish covers an arbitrary part of its
|
||||
sets and is refused unless asked for."""
|
||||
man = json.load(open(os.path.join(run_dir, "manifest.json")))
|
||||
res = json.load(open(os.path.join(run_dir, "results.json")))
|
||||
state = run_state(man)
|
||||
if state != "complete" and not allow_incomplete:
|
||||
sys.exit(f"{run_dir} is {state}, not a finished run; pass --allow-incomplete to use it anyway")
|
||||
path = os.path.join(run_dir, "results.json")
|
||||
res = json.load(open(path)) if os.path.exists(path) else []
|
||||
return man, res
|
||||
|
||||
|
||||
def check_baseline(baseline, private):
|
||||
"""A baseline must have finished, and be as private as the run it is compared with: the delta
|
||||
table names its sets."""
|
||||
man, _ = load_run(baseline)
|
||||
if bool(man.get("private")) != bool(private):
|
||||
sys.exit(f"baseline {baseline} is {'' if man.get('private') else 'not '}a private run; "
|
||||
"a report may only be compared with a run of its own kind")
|
||||
|
||||
|
||||
# Version of the results.json row schema (README.md, "results.json"). Bump it when a field changes
|
||||
# meaning or is removed; adding a field does not need it.
|
||||
RESULTS_SCHEMA = 1
|
||||
|
||||
# {arm: {old set id: current id}} from the manifests (battery.py sets it), so a run made before a
|
||||
# dataset was renamed still pairs with one made after
|
||||
ALIASES = {}
|
||||
|
||||
|
||||
def key(r):
|
||||
return (r["arm"], r["set"])
|
||||
return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"]))
|
||||
|
||||
|
||||
def f(x, fmt="{:.2f}", dash="-"):
|
||||
@@ -71,7 +114,7 @@ def changes(ra, rb):
|
||||
if ra.get("volume_ratio") and rb.get("volume_ratio") and \
|
||||
abs(ra["volume_ratio"] - rb["volume_ratio"]) > 0.05:
|
||||
out.append("lattice")
|
||||
for name in ("d_min", "isa", "r_meas", "cc_half", "completeness", "cell_dev_pct"):
|
||||
for name in ("d_min", "isa", "r_meas", "cc_half", "completeness", "cell_dev_pct", "rfree"):
|
||||
if beyond_noise(name, ra.get(name), rb.get(name)):
|
||||
out.append(name)
|
||||
if beyond_noise("wall_s", ra.get("wall_s"), rb.get("wall_s")):
|
||||
@@ -97,7 +140,7 @@ def arrow(x, y, fmt=lambda v: str(v)):
|
||||
|
||||
def compare_table(doc, rows, only_changed=True):
|
||||
head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
|
||||
"cell dev %", "time s", "beyond noise"]
|
||||
"cell dev %", "R_free", "time s", "beyond noise"]
|
||||
out = []
|
||||
for (arm, s), ra, rb, ch in rows:
|
||||
if only_changed and not ch:
|
||||
@@ -110,6 +153,7 @@ def compare_table(doc, rows, only_changed=True):
|
||||
arrow(ra.get("r_meas"), rb.get("r_meas"), lambda v: f"{100 * v:.1f}%"),
|
||||
arrow(ra.get("cc_half"), rb.get("cc_half"), lambda v: f"{v:.3f}"),
|
||||
arrow(ra.get("cell_dev_pct"), rb.get("cell_dev_pct"), lambda v: f"{v:.2f}"),
|
||||
arrow(ra.get("rfree"), rb.get("rfree"), lambda v: f"{v:.3f}"),
|
||||
arrow(ra.get("wall_s"), rb.get("wall_s"), lambda v: f"{v:.0f}"),
|
||||
", ".join(ch)])
|
||||
doc.table(head, out, row_class=lambda r: "fail" if "verdict" in r[-1] else "")
|
||||
@@ -130,14 +174,21 @@ def rate(rs):
|
||||
return f"{n}/{len(s)} ({100 * n / len(s):.0f}%)" if s else "-"
|
||||
|
||||
|
||||
def build(run_dir, baseline=None):
|
||||
def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
"""Markdown and HTML report of one run, with the delta against a baseline run if given."""
|
||||
man, res = load_run(run_dir)
|
||||
man, res = load_run(run_dir, allow_incomplete)
|
||||
if baseline:
|
||||
check_baseline(baseline, man.get("private"))
|
||||
b = man["binary"]
|
||||
doc = Doc(f"Rugnux battery - {os.path.basename(os.path.normpath(run_dir))}")
|
||||
if man.get("private"):
|
||||
doc.p("PRIVATE ARM - confidential user data. This report stays in the run directory; it "
|
||||
"must not be copied into the repository, an issue or any public report.")
|
||||
state = run_state(man)
|
||||
if state != "complete":
|
||||
why = f" ({man['aborted']['reason']})" if man.get("aborted") else ""
|
||||
doc.p(f"RUN {state.upper()}{why}: {len(res)} of {len(man['sets'])} sets have a result. "
|
||||
"Not a reference and not comparable with a finished run.")
|
||||
doc.ul([
|
||||
f"rugnux {b.get('version', '?')}, sha256 {b['sha256'][:16]}, build flags: {b.get('flags') or 'unknown'}",
|
||||
f"runner {man['runner_git']}, host {man['host']}, {man['start']} -> {man.get('end') or 'NOT FINISHED'}",
|
||||
@@ -171,6 +222,9 @@ def build(run_dir, baseline=None):
|
||||
f(median(r.get("wall_s") for r in rs), "{:.0f}"),
|
||||
f(sum(r.get("wall_s") or 0 for r in rs) / 60, "{:.1f}")])
|
||||
doc.table(head, rows, num=range(1, 12))
|
||||
doc.svg(verdict_bars([(arm, {v: sum(r["verdict"] == v and r["arm"] == arm for r in res)
|
||||
for v in ("pass", "fail", "unscored", "not_run")})
|
||||
for arm in man["arms"]]), "verdicts per arm")
|
||||
doc.p("Pass rate is over the scored sets (pass + fail). Resolution gain is (reference d_min - "
|
||||
"our d_min) / reference d_min: positive = finer than the deposition, or than XDS; 'forced' "
|
||||
"where XDS's resolution range was imposed, which makes it zero by construction. Medians "
|
||||
@@ -201,6 +255,35 @@ def build(run_dir, baseline=None):
|
||||
rows.append([arm, lab, sum(r.get(name) is not None for r in rs)] + cells)
|
||||
doc.table(["arm", "metric", "n", "min", "q1", "median", "q3", "max"], rows, num=range(2, 8))
|
||||
|
||||
doc.h(3, "Per set, against the reference")
|
||||
for arm in man["arms"]:
|
||||
if arm in forced:
|
||||
doc.p(f"{arm}: d_min forced to the reference's, so its ratio is 1 by construction.")
|
||||
continue
|
||||
pts = [(r["set"], r["d_min"] / r["d_min_ref"], r["verdict"],
|
||||
f"d_min {r['d_min']:.2f} vs {r['d_min_ref']:.2f} A, {r['verdict']}")
|
||||
for r in res if r["arm"] == arm and r.get("d_min") and r.get("d_min_ref")]
|
||||
if pts:
|
||||
doc.svg(ratio_dots(pts, f"{arm}: d_min ratio"),
|
||||
f"{arm}: d_min(rugnux) / d_min(reference), one point per set, sorted; below 1 = "
|
||||
"finer than the reference")
|
||||
pts = [(r["set"], r["rfree_ratio"], r["verdict"],
|
||||
f"R_free {r['rfree']:.3f} vs deposited {r['rfree_deposited']:.3f}")
|
||||
for r in res if r.get("rfree_ratio")]
|
||||
if pts:
|
||||
doc.svg(ratio_dots(pts, "R_free ratio"),
|
||||
"open: R_free of the deposited model against our merge / the deposited R_free, one "
|
||||
"point per set, sorted; below 1 = our data fit the model better")
|
||||
|
||||
derived = [r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"]
|
||||
if derived:
|
||||
doc.p("XDS merged past its own signal (CC1/2 of its finest shell not significant), so the "
|
||||
"reference d_min is where XDS's CC1/2 falls through 0.30 (rule "
|
||||
f"{derived[0]['d_min_ref_rule']}), and that is also the range forced on rugnux: "
|
||||
+ "; ".join(f"{r['set']} (XDS range {f(r.get('d_min_xds'))} A, reference "
|
||||
f"{f(r.get('d_min_ref'))} A)" for r in derived)
|
||||
+ ". XDS's pooled R_meas, CC1/2 and completeness are over its whole range.")
|
||||
|
||||
doc.h(2, "Failures")
|
||||
fails = [r for r in res if r["verdict"] == "fail"]
|
||||
doc.table(["set", "arm", "cause", "reason"],
|
||||
@@ -227,7 +310,8 @@ def build(run_dir, baseline=None):
|
||||
|
||||
doc.h(2, "Per set")
|
||||
head = ["set", "arm", "verdict", "space group", "ref", "cell dev %", "V ratio", "d_min", "ref d_min",
|
||||
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx", "time s", "note"]
|
||||
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx", "R_free", "dep R_free",
|
||||
"time s", "note"]
|
||||
rows = []
|
||||
for r in sorted(res, key=lambda r: (order.get(r["arm"], 9), r["set"])):
|
||||
rows.append([r["set"], r["arm"], r["verdict"], r.get("sg") or "-", r.get("sg_ref") or "-",
|
||||
@@ -236,8 +320,9 @@ def build(run_dir, baseline=None):
|
||||
f(r.get("completeness"), "{:.1f}"), f(r.get("multiplicity"), "{:.1f}"),
|
||||
pct(r.get("r_meas")), f(r.get("cc_half"), "{:.3f}"), f(r.get("isa"), "{:.1f}"),
|
||||
f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0),
|
||||
f(r.get("rfree"), "{:.3f}"), f(r.get("rfree_deposited"), "{:.3f}"),
|
||||
f(r.get("wall_s"), "{:.0f}"), r.get("reason") or ""])
|
||||
doc.table(head, rows, num=range(5, 18),
|
||||
doc.table(head, rows, num=range(5, 20),
|
||||
row_class=lambda row: "fail" if row[2] == "fail" else "",
|
||||
cell_class=lambda i, v: {"pass": "pass", "fail": "fail"}.get(v) if i == 2 else None)
|
||||
return doc
|
||||
@@ -87,14 +87,15 @@ def judge(entry, rep, run_note):
|
||||
"sgno_ref": ref.get("sgno"), "sg_ref": ref.get("sg") or sg_name(ref.get("sgno")),
|
||||
"pg_ref": point_group(ref["sgno"]) if ref.get("sgno") else None,
|
||||
"cell": None, "cell_ref": ref.get("cell"), "cell_dev_pct": None, "volume_ratio": None,
|
||||
"d_min": None, "d_min_ref": ref.get("dmin"), "res_gain_pct": None,
|
||||
"d_min": None, "d_min_ref": ref.get("dmin"), "d_min_ref_rule": ref.get("dmin_rule"),
|
||||
"d_min_xds": ref.get("dmin_xds"), "res_gain_pct": None,
|
||||
"r_meas": fnum(rep.get("R_MEAS")), "cc_half": fnum(rep.get("CC_HALF")),
|
||||
"isa": fnum(rep.get("ISA")), "completeness": fnum(rep.get("COMPLETENESS")),
|
||||
"multiplicity": fnum(rep.get("MULTIPLICITY")), "i_over_sigma": fnum(rep.get("I_OVER_SIGMA")),
|
||||
"indexing_rate": fnum(rep.get("INDEXING_RATE")), "images": fnum(rep.get("IMAGES_PROCESSED")),
|
||||
"rugnux_wall_s": fnum(rep.get("WALL_TIME")), "rugnux_verdict": rep.get("VERDICT_TEXT"),
|
||||
"r_meas_ref": ref.get("r_meas"), "cc_half_ref": ref.get("cc_half"), "isa_ref": ref.get("isa"),
|
||||
"completeness_ref": ref.get("completeness"),
|
||||
"completeness_ref": ref.get("completeness"), "sg_relation": None,
|
||||
}
|
||||
if rep.get("SPACE_GROUP_NUMBER"):
|
||||
r["sgno"] = int(fnum(rep["SPACE_GROUP_NUMBER"]))
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"_comment": "Copy to site.json beside battery.py (not committed), or anywhere and point JFJOCH_BATTERY_SITE at it. Paths are absolute; a relative manifest resolves against tools/battery/.",
|
||||
"runs_root": "/path/to/battery/runs",
|
||||
"lock_file": "/path/to/battery/full_run.lock",
|
||||
"gpulock": null,
|
||||
"baseline": null,
|
||||
"baseline_private": null,
|
||||
"arms": {
|
||||
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/path/to/battery/open"},
|
||||
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/path/to/battery/inhouse"}
|
||||
}
|
||||
}
|
||||
@@ -1,11 +0,0 @@
|
||||
{
|
||||
"runs_root": "/data/battery/runs",
|
||||
"lock_file": "/data/battery/full_run.lock",
|
||||
"gpulock": "/data/tmp/gpulock",
|
||||
"arms": {
|
||||
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/nonsls/raw"},
|
||||
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/data/rotation_test"},
|
||||
"private": {"manifest": "/data/battery/private.json", "reference": "xds", "data_root": "/data/rotation_test",
|
||||
"private": true}
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user