tools/battery: run every variant of a set back to back
The battery is bound by reading images from disk, so instead of one battery run per setting each set now runs in all its variants in a row while its images are still in the page cache: open arm bare, then model (rugnux --model <deposited coordinates>) XDS arms bare, then xds (XDS's resolution range forced, -A where XDS was anomalous) bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk read whichever variants are selected; each row records first_read and the report's timing table says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant; compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is read as one variant. The model variant takes the coordinates and published R-free from model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio; the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2. model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50 were rugnux's defaults, so the command is the same). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
162aaafef7
commit
bb0648ae87
8 files changed
+397
-450
No files matched your search
@@ -106,9 +106,16 @@ scripts.
|
||||
```
|
||||
tools/battery/battery.py run --rugnux build/rugnux/rugnux --only 5reo,lyso_ref # named sets
|
||||
tools/battery/battery.py run --rugnux build/rugnux/rugnux --tier smoke # smoke tier
|
||||
tools/battery/battery.py run --rugnux build/rugnux/rugnux --variants bare # one variant only
|
||||
tools/battery/battery.py compare RUN_A RUN_B [--rerun-changed]
|
||||
```
|
||||
|
||||
Each set runs in all its arm's **variants** back to back, so its images are read from disk once:
|
||||
`bare` (plain `rugnux <input>`, what a user gets) first everywhere, then `model` (`--model` with the
|
||||
deposited coordinates, R-free vs the published one) on the open arm, or `xds` (XDS's resolution
|
||||
range forced, `-A` where XDS was anomalous) on the XDS arms. Only the first variant's time includes
|
||||
the disk read; compare times within one variant. The REFMAC check is opt-in (`--model-check`).
|
||||
|
||||
- **Never start a full battery run (no `--only`/`--tier`) without the owner's go-ahead.**
|
||||
- Judge a change against a **persisted baseline run** (the site config's `baseline`) with
|
||||
`compare`, never against numbers remembered from an earlier session.
|
||||
|
||||
+3
-3
@@ -40,9 +40,9 @@ pipeline is exercised on, and the DOI to cite for each, are listed in
|
||||
* `tools/battery/battery.py` - the rotation battery, the one canonical way to judge a rotation
|
||||
change: public PDB depositions scored against the deposition, in-house standard crystals and
|
||||
no-crystal controls scored against XDS, and a local-only private arm. Its manifests, the run
|
||||
and compare protocol and the report are described in `tools/battery/README.md`.
|
||||
* `tools/battery/model_sweep.py` - the battery's open arm run with `--model`: each merge validated
|
||||
against the depositor's own model and compared with the published resolution and R-work/R-free.
|
||||
and compare protocol and the report are described in `tools/battery/README.md`. Each set runs
|
||||
in several variants back to back: plain, with the deposited model (`--model`, R-free against the
|
||||
published one) on the open arm, and with XDS's settings on the XDS arms.
|
||||
* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial
|
||||
dataset by what it does to the merge. Takes its dataset list from outside the repository.
|
||||
* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.
|
||||
|
||||
+98
-45
@@ -23,6 +23,45 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X
|
||||
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
|
||||
`score.py` has the details.
|
||||
|
||||
## Variants
|
||||
|
||||
Every set is run more than once, with different settings, and each run is a **variant**. The
|
||||
variants of one set run straight after each other, before the next set starts. The battery is
|
||||
limited by reading images from disk, and this way only the first variant reads them; the others
|
||||
find them in the page cache.
|
||||
|
||||
| arm (reference) | variants, in the order they run |
|
||||
|---|---|
|
||||
| open (deposition) | `bare`, then `model` |
|
||||
| inhouse, private (XDS) | `bare`, then `xds` |
|
||||
|
||||
- **`bare`**: plain `rugnux <input>` and nothing else (no `-A`, no resolution range). This is
|
||||
what a user gets. It is scored against the reference like the others, and its d_min ratio
|
||||
against the reference is plotted.
|
||||
- **`model`** (open arm): `rugnux --model <deposited coordinates>`. The coordinates (`<entry>.cif.gz`)
|
||||
and the published R-free come from the RCSB and are cached in the site's `pdb_cache` (default
|
||||
`/data/battery/pdb_cache`), so the first run of a new entry needs network access. The entry is the
|
||||
set id up to the first `_`, so `6h2p_native` uses `6h2p`. A set with no PDB entry (small
|
||||
molecules) or whose coordinates cannot be downloaded gets a `not_run` row with cause `no_model`.
|
||||
The row records the R-free and R-work that rugnux reports for the model, the published R-free
|
||||
and their ratio, which the report plots. rugnux's R-free is that of a rigid-body placement against
|
||||
its own merge, and the published one comes from full refinement against the depositor's own
|
||||
data, so the ratio is not a like-for-like score; watch how it moves between runs.
|
||||
- **`xds`** (XDS arms): XDS's reference settings. XDS's resolution range is forced (see "The XDS
|
||||
reference d_min" below), and where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true`
|
||||
in the manifest) rugnux gets `-A`. Both programs then merge the same reflections the same way.
|
||||
|
||||
The variants are defined in one place, `VARIANTS` in `battery.py`. `--variants` runs a subset,
|
||||
for example `--variants bare`, and they always run in the table's order. `--unforced` is gone:
|
||||
`--variants bare` gives rugnux its own resolution cut (and, unlike `--unforced`, no `-A`).
|
||||
|
||||
**Order and timing.** `bare` goes first on every arm. So its time always includes reading the
|
||||
data from disk (unless the images were still cached from before the run), whichever variants a run
|
||||
selects, and it is the time a user sees. The time of a later variant is taken on cached images.
|
||||
Each row records whether it was its set's `first_read`, and the report's timing table says which
|
||||
variant carries the disk reads. Compare times of one variant between runs only, never across
|
||||
variants.
|
||||
|
||||
The private arm is never mixed with the others. A private run gets its own run directory
|
||||
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
|
||||
public report or be written into the repository. Its names, paths and values never go into a
|
||||
@@ -86,6 +125,7 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
|
||||
"gpulock": "/data/tmp/gpulock",
|
||||
"baseline": "/data/battery/runs/<a blessed full run>",
|
||||
"baseline_private": null,
|
||||
"pdb_cache": "/data/battery/pdb_cache",
|
||||
"arms": {
|
||||
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"},
|
||||
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"},
|
||||
@@ -97,7 +137,8 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
|
||||
|
||||
A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
|
||||
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
|
||||
the shared GPU slots. Set it to `null` if you have no such wrapper.
|
||||
the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
|
||||
the deposited models and their metadata are cached for the `model` variant and the REFMAC check.
|
||||
|
||||
## Running
|
||||
|
||||
@@ -109,7 +150,7 @@ $B run --rugnux $R # full battery: open + inhou
|
||||
$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases
|
||||
$B run --rugnux $R --only 5reo,lyso_ref # named sets
|
||||
$B run --rugnux $R --arm private # the private arm, on its own
|
||||
$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced
|
||||
$B run --rugnux $R --variants bare # only what a user gets (no model, no XDS settings)
|
||||
```
|
||||
|
||||
Useful options:
|
||||
@@ -119,7 +160,8 @@ Useful options:
|
||||
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
|
||||
- `--extra "<args>"` adds rugnux arguments to every set.
|
||||
- `--baseline RUN|none` sets the run the report is compared against.
|
||||
- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below).
|
||||
- `--variants bare,model,xds` runs a subset of the variants (default: all of them).
|
||||
- `--model-check` adds the REFMAC check to the open arm's `bare` rows (off by default, see below).
|
||||
|
||||
A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
|
||||
manifests (`"tiers": {"smoke": "<why this set>"}`).
|
||||
@@ -128,23 +170,23 @@ manifests (`"tiers": {"smoke": "<why this set>"}`).
|
||||
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
|
||||
everything else on the machine competes with it.
|
||||
|
||||
### The model check (open arm)
|
||||
### The REFMAC model check (open arm, opt-in)
|
||||
|
||||
`tools/battery/model_check.py` scores each open-arm merge (`p.mtz`) against the deposited model,
|
||||
The `model` variant is the battery's R-free check. `--model-check` adds a second, independent one
|
||||
that needs CCP4: `tools/battery/model_check.py` scores each open-arm `bare` merge (`p.mtz`) against
|
||||
the deposited model,
|
||||
unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before
|
||||
any shift, the data reindexed into the model's setting and reduced to the deposited space group.
|
||||
Our own free set was mostly work reflections for the depositor, so `rfree` reads low. The fair
|
||||
comparison is `rfree_depflags` (our data on the depositor's free set) against `rfree_depdata` (the
|
||||
depositor's structure factors, same free set, same protocol); `rfree_ratio` is that ratio, and the
|
||||
report plots it. It is null when the entry deposited no structure factors with a free set.
|
||||
`rfree_deposited` is the value in the entry, often from twin or full refinement. The runner records
|
||||
`rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`,
|
||||
`model_status` and `model_reason`. The check is on by default when the module exists. The PDB entry
|
||||
is the set id up to the first `_`, so `6h2p_native` is checked as `6h2p`. Small-molecule sets are skipped.
|
||||
Our own free set was mostly work reflections for the depositor, so `refmac_rfree` reads low. The
|
||||
fair comparison is `refmac_rfree_depflags` (our data on the depositor's free set) against
|
||||
`refmac_rfree_depdata` (the depositor's structure factors, same free set, same protocol);
|
||||
`refmac_rfree_ratio` is that ratio, and the report plots it. It is null when the entry deposited no
|
||||
structure factors with a free set. The runner records `refmac_rfree`, `refmac_rwork`,
|
||||
`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and
|
||||
`refmac_reason`. Small-molecule sets are skipped.
|
||||
|
||||
`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same
|
||||
site config) and compares the R-free that rugnux itself reports with the published value. It needs
|
||||
network access to the RCSB and runs separately from `battery.py`.
|
||||
`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
|
||||
`model` variant does the same inside the battery.
|
||||
|
||||
## A run directory
|
||||
|
||||
@@ -155,9 +197,9 @@ read-only when the run completes.
|
||||
|---|---|
|
||||
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
|
||||
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
|
||||
| `results.json` | one row per set (the schema is below) |
|
||||
| `results.json` | one row per set and variant (the schema is below) |
|
||||
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
|
||||
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
|
||||
| `work/<arm>/<set>/<variant>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
|
||||
|
||||
**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
|
||||
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
|
||||
@@ -165,16 +207,24 @@ read-only when the run completes.
|
||||
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
|
||||
`--allow-incomplete`, and the report then says so at the top.
|
||||
|
||||
### results.json (schema 1)
|
||||
### results.json (schema 2)
|
||||
|
||||
This is a flat list with one object per set, and every arm uses the same keys. Missing values are
|
||||
`null`. `manifest.json` records the schema version as `results_schema`.
|
||||
This is a flat list with one object per set and variant, and every arm uses the same keys. Missing
|
||||
values are `null`. `manifest.json` records the schema version as `results_schema`, and the variants
|
||||
each arm ran as `variants`.
|
||||
|
||||
Schema 2 added `variant` and `first_read`. It also changed the meaning of `rfree`, `rwork` and
|
||||
`rfree_ratio`: they are now rugnux's own `--model` values (model variant), and the REFMAC check's
|
||||
moved to `refmac_*`. A schema-1 run is still read, and compared, as if each set had one variant:
|
||||
`bare` on the open arm; on the XDS arms `xds`, or `bare` if it was run `--unforced` (which still
|
||||
passed `-A`).
|
||||
|
||||
| key | meaning |
|
||||
|---|---|
|
||||
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
|
||||
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) |
|
||||
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` |
|
||||
| `set`, `arm`, `variant`, `tags`, `input`, `cmd` | the set's id, arm, variant (`bare`, `model`, `xds`), population tags, input file, and the exact command that ran |
|
||||
| `first_read` | true for the first variant of its set that ran: the one that read the images from disk |
|
||||
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing, or no model for the `model` variant) |
|
||||
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` / `no_model` |
|
||||
| `reason` | one line for a human |
|
||||
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours / the reference's |
|
||||
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
|
||||
@@ -183,7 +233,8 @@ This is a flat list with one object per set, and every arm uses the same keys. M
|
||||
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
|
||||
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt` |
|
||||
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref` | XDS's overall values, over XDS's own range |
|
||||
| `rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`, `model_status`, `model_reason` | the model check (open arm) |
|
||||
| `rfree`, `rwork`, `model_fit`, `rfree_deposited`, `rfree_ratio` | `model` variant: R-free, R-work and MODEL_FIT as rugnux reports them for the deposited model, the published R-free, and rfree / rfree_deposited |
|
||||
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm, `bare` rows) |
|
||||
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |
|
||||
|
||||
## The report
|
||||
@@ -191,16 +242,19 @@ This is a flat list with one object per set, and every arm uses the same keys. M
|
||||
Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
|
||||
SVG, no external files):
|
||||
|
||||
- a header with the binary, build flags, host, subset, resolution mode and whether timing counts;
|
||||
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
|
||||
ISa, R_meas, time) with a verdict bar chart;
|
||||
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
|
||||
the main metrics;
|
||||
- plots per set: d_min(rugnux) / d_min(reference) for each arm whose resolution is not forced, and
|
||||
R_free / deposited R_free for the open arm (HTML only; hovering a point shows the set);
|
||||
- the **failures**, and, if there is a baseline, the **delta against it**: pass rates on the
|
||||
common sets and every set that moved beyond noise;
|
||||
- **one row per set** with all the numbers.
|
||||
- a header with the binary, build flags, host, subset, variants and whether timing counts;
|
||||
- a **timing** table per variant, saying which variant's times include reading the data;
|
||||
- **a section per variant**, each with:
|
||||
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
|
||||
ISa, R_meas, time) with a verdict bar chart;
|
||||
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
|
||||
the main metrics;
|
||||
- plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
|
||||
for each arm (not for `xds`, whose resolution is forced), and, for `model`, R_free / published
|
||||
R_free;
|
||||
- the **failures**, and **one row per set** with all the numbers;
|
||||
- if there is a baseline, the **delta against it**: pass rates per arm and variant on the common
|
||||
rows, and every row that moved beyond noise.
|
||||
|
||||
Re-render a report, for example against another baseline (run directories are read-only, so the
|
||||
output goes elsewhere and is named after the run):
|
||||
@@ -221,8 +275,9 @@ $B compare RUN_A RUN_B [--all] # per-set table of what move
|
||||
$B compare RUN_A RUN_B --rerun-changed # rerun the changed sets with A's binary
|
||||
```
|
||||
|
||||
`compare` pairs sets by arm and id, following the manifests' `aliases` across renames, and lists
|
||||
every set whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
|
||||
`compare` pairs rows by arm, set id and variant, following the manifests' `aliases` across
|
||||
renames. Only the variants both runs ran on an arm are compared, so a `--variants bare` run against a full
|
||||
one compares `bare`. It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
|
||||
R-free or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds
|
||||
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
|
||||
A's saved binary and options. A set that moves again under the same binary is noise. A set that
|
||||
@@ -240,13 +295,11 @@ reproduces A was really changed by B.
|
||||
fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
|
||||
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
|
||||
reference". Only a run with the machine to itself can be used to judge speed.
|
||||
5. **Resolution on the XDS arms.** By default XDS's resolution range is forced on rugnux, so both
|
||||
programs merge the same reflections and the statistics are like for like. This answers "is our
|
||||
processing better than XDS's, reflection for reflection". `--unforced` leaves rugnux its own
|
||||
cut, which answers "what does a user get". A change can win one and lose the other, so at
|
||||
important moments run both. Never quote the forced run as what rugnux delivers.
|
||||
Where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the manifest), rugnux gets
|
||||
`-A` in both modes, so Friedel mates are merged separately on both sides.
|
||||
5. **Resolution on the XDS arms.** The `xds` variant forces XDS's resolution range (and `-A`
|
||||
where XDS kept Friedel mates apart), so both programs merge the same reflections and the
|
||||
statistics are like for like. This answers "is our processing better than XDS's, reflection for
|
||||
reflection". The `bare` variant answers "what does a user get". A change can win one and lose
|
||||
the other, and every run has both. Never quote the `xds` variant as what rugnux delivers.
|
||||
|
||||
### The XDS reference d_min
|
||||
|
||||
@@ -262,7 +315,7 @@ is significant, and that decides which case applies (`inputs.reference_dmin`):
|
||||
point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
|
||||
1/d² between shell centres.
|
||||
|
||||
The derived value is both the score reference and the forced range. The manifest keeps XDS's own
|
||||
The derived value is both the score reference and the range the `xds` variant forces. The manifest keeps XDS's own
|
||||
limit as `dmin_xds` and the rule used as `dmin_rule`. XDS's pooled R_meas, CC1/2 and completeness
|
||||
still cover its whole range.
|
||||
|
||||
|
||||
+121
-62
@@ -12,7 +12,8 @@ Arms and where their truth comes from:
|
||||
|
||||
Usage (README.md in this directory is the full guide):
|
||||
battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke]
|
||||
[--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN|none]
|
||||
[--variants bare,model,xds] [--build-dir DIR] [--gpulock] [--threads N]
|
||||
[--model-check] [--baseline RUN|none]
|
||||
battery.py list runs under the runs root, with their state
|
||||
battery.py compare RUN_A RUN_B [--all] [--rerun-changed]
|
||||
battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/<run>.{md,html}
|
||||
@@ -24,9 +25,13 @@ Usage (README.md in this directory is the full guide):
|
||||
The site config (data roots, runs root, lock, gpulock, private manifest, baselines) is --site, else
|
||||
$JFJOCH_BATTERY_SITE, else site.json beside this script (not committed; site.example.json is).
|
||||
|
||||
Every set runs in each of its arm's VARIANTS (bare + model on the open arm, bare + xds on the XDS
|
||||
arms), one straight after the other, so only the first reads the images from disk.
|
||||
|
||||
A run writes <runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>/ holding a copy of the binary,
|
||||
manifest.json (what was run, on what, with which references), results.json (one row per set),
|
||||
report.md + report.html and work/<arm>/<set>/ (log, report, cif). It is made read-only when done.
|
||||
manifest.json (what was run, on what, with which references), results.json (one row per set and
|
||||
variant), report.md + report.html and work/<arm>/<set>/<variant>/ (log, report, cif). It is made
|
||||
read-only when done.
|
||||
A full-arm run (no --only / --tier) takes the site lock and refuses to start if another holds it.
|
||||
A run that did not finish is never compared or reported on unless asked (--allow-incomplete).
|
||||
"""
|
||||
@@ -52,13 +57,24 @@ import inputs # noqa: E402
|
||||
import report # noqa: E402
|
||||
import score # noqa: E402
|
||||
|
||||
try:
|
||||
import model_check # optional: R-free of the deposited model against our merge (open arm)
|
||||
except ImportError:
|
||||
model_check = None
|
||||
import model_check # noqa: E402 the deposited model (model variant); REFMAC check (--model-check)
|
||||
|
||||
PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$")
|
||||
|
||||
# The variants every set of an arm runs in, by the arm's reference, in the order they run. All the
|
||||
# variants of one set run back to back, so only the first reads the images from disk and the later
|
||||
# ones find them in the page cache. `bare` goes first: it is in every arm, so its time is taken
|
||||
# under the same condition (the set's first read) whichever variants a run selects, and it is the
|
||||
# number a user sees.
|
||||
VARIANTS = {"deposition": ["bare", "model"], "xds": ["bare", "xds"]}
|
||||
VARIANT_DOC = {
|
||||
"bare": "plain `rugnux <input>`, nothing else: what a user gets",
|
||||
"model": "`rugnux --model <deposited coordinates>`: R-free of the deposited model against our merge, "
|
||||
"as rugnux reports it",
|
||||
"xds": "XDS's reference settings: its resolution range forced, and -A where XDS kept Friedel mates "
|
||||
"apart, so both programs merge the same reflections the same way",
|
||||
}
|
||||
|
||||
LOCAL_SITE = os.path.join(HERE, "site.json")
|
||||
|
||||
|
||||
@@ -155,22 +171,23 @@ def gpu_others():
|
||||
|
||||
# ------------------------------------------------------------------ running
|
||||
|
||||
def command(binary, e, opts):
|
||||
def command(binary, e, variant, opts, model=None):
|
||||
cmd = [binary, "-o", "p", "--no-export-unmerged"]
|
||||
if opts["threads"]:
|
||||
cmd += ["-N", str(opts["threads"])]
|
||||
ref = e.get("ref") or {}
|
||||
# On an XDS-referenced arm the only thing fixed is XDS's resolution range, so the two programs
|
||||
# merge the same reflections. --unforced leaves rugnux its own cut.
|
||||
if opts["xds_range"] and e.get("reference") == "xds":
|
||||
if variant == "xds":
|
||||
# XDS's resolution range, so the two programs merge the same reflections
|
||||
if ref.get("dmin"):
|
||||
cmd += ["--scaling-high-resolution", f"{ref['dmin']:.3f}"]
|
||||
if ref.get("dmax"):
|
||||
cmd += ["--scaling-low-resolution", f"{ref['dmax']:.3f}"]
|
||||
# XDS merged with FRIEDEL'S_LAW=FALSE: merge Friedel mates separately too, or the merging
|
||||
# statistics are not comparable
|
||||
if e.get("reference") == "xds" and ref.get("anomalous"):
|
||||
cmd.append("-A")
|
||||
# XDS merged with FRIEDEL'S_LAW=FALSE: merge Friedel mates separately too, or the merging
|
||||
# statistics are not comparable
|
||||
if ref.get("anomalous"):
|
||||
cmd.append("-A")
|
||||
if variant == "model":
|
||||
cmd += ["--model", model]
|
||||
cmd += opts["extra_args"]
|
||||
cmd.append(e["input_path"])
|
||||
if opts["gpulock"]:
|
||||
@@ -190,16 +207,37 @@ def failure_note(log):
|
||||
return lines[-1].split("] ")[-1][:300] if lines else ""
|
||||
|
||||
|
||||
def run_one(run_dir, binary, e, opts):
|
||||
wd = os.path.join(run_dir, "work", e["arm"], e["id"])
|
||||
def deposited_model(e, opts):
|
||||
"""(deposition record from model_check's RCSB cache, None) for the entry a set belongs to, or
|
||||
(None, why not). A set id may carry a suffix (6h2p_native); the entry is the part before it."""
|
||||
pdb = e["id"].split("_")[0].lower()
|
||||
if not PDB_ID.match(pdb):
|
||||
return None, "no model: not a PDB entry (small molecule or unpublished set)"
|
||||
meta = model_check.deposition(pdb, opts["pdb_cache"])
|
||||
if meta is None:
|
||||
return None, f"no model: coordinates of {pdb} could not be downloaded"
|
||||
return meta, None
|
||||
|
||||
|
||||
def run_one(run_dir, binary, e, variant, first_read, opts):
|
||||
wd = os.path.join(run_dir, "work", e["arm"], e["id"], variant)
|
||||
os.makedirs(wd)
|
||||
row = {"set": e["id"], "arm": e["arm"], "tags": e.get("tags", []), "input": e["input_path"],
|
||||
"gpu_others": gpu_others(), "rfree": None, "rwork": None, "rfree_deposited": None,
|
||||
"rfree_depflags": None, "rfree_depdata": None, "rfree_ratio": None, "model_status": None, "model_reason": None}
|
||||
row = {"set": e["id"], "arm": e["arm"], "variant": variant, "first_read": first_read,
|
||||
"tags": e.get("tags", []), "input": e["input_path"], "gpu_others": gpu_others(),
|
||||
"rfree_deposited": None, "rfree_ratio": None, "cmd": None, "exit_code": None,
|
||||
"elapsed_s": None, "wall_s": None}
|
||||
row.update({k: None for k in REFMAC_KEYS})
|
||||
if not os.path.exists(e["input_path"]):
|
||||
row.update(score.judge(e, {}, "no input"), exit_code=None, elapsed_s=None, wall_s=None, cmd=None)
|
||||
row.update(score.judge(e, {}, "no input"))
|
||||
return row
|
||||
cmd = command(binary, e, opts)
|
||||
meta = None
|
||||
if variant == "model":
|
||||
meta, why = deposited_model(e, opts)
|
||||
if meta is None:
|
||||
row.update(score.judge(e, {}, why))
|
||||
return row
|
||||
row["rfree_deposited"] = meta["rfree"]
|
||||
cmd = command(binary, e, variant, opts, meta and meta["xyz"])
|
||||
row["cmd"] = " ".join(cmd)
|
||||
t0 = time.time()
|
||||
note = ""
|
||||
@@ -217,34 +255,38 @@ def run_one(run_dir, binary, e, opts):
|
||||
# rugnux's own WALL_TIME where it wrote one: the runner's clock also counts the wait for a
|
||||
# GPU slot under --gpulock
|
||||
row["wall_s"] = row["rugnux_wall_s"] if row["rugnux_wall_s"] is not None else row["elapsed_s"]
|
||||
if opts.get("model_check") and e["arm"] == "open":
|
||||
if row["rfree"] and row["rfree_deposited"]:
|
||||
row["rfree_ratio"] = round(row["rfree"] / row["rfree_deposited"], 4)
|
||||
if opts["model_check"] and e["arm"] == "open" and variant == "bare":
|
||||
row.update(check_model(e, wd))
|
||||
return row
|
||||
|
||||
|
||||
# what the REFMAC model check (model_check.py, --model-check) adds to an open-arm bare row
|
||||
REFMAC_KEYS = ("refmac_rfree", "refmac_rwork", "refmac_rfree_depflags", "refmac_rfree_depdata",
|
||||
"refmac_rfree_ratio", "refmac_status", "refmac_reason")
|
||||
|
||||
|
||||
def check_model(e, wd):
|
||||
"""R-free of the deposited model against this run's merge (model_check.py). A set id may
|
||||
carry a suffix (6h2p_native); the PDB entry is the part before it."""
|
||||
out = {"rfree": None, "rwork": None, "rfree_deposited": None, "rfree_depflags": None,
|
||||
"rfree_depdata": None, "rfree_ratio": None}
|
||||
"""R-free of the deposited model against this run's merge, by REFMAC (model_check.py)."""
|
||||
out = {k: None for k in REFMAC_KEYS}
|
||||
pdb = e["id"].split("_")[0]
|
||||
mtz = os.path.join(wd, "p.mtz")
|
||||
if not PDB_ID.match(pdb):
|
||||
return dict(out, model_status="skipped", model_reason="not a PDB entry")
|
||||
return dict(out, refmac_status="skipped", refmac_reason="not a PDB entry")
|
||||
if not os.path.exists(mtz):
|
||||
return dict(out, model_status="skipped", model_reason="no merged MTZ")
|
||||
return dict(out, refmac_status="skipped", refmac_reason="no merged MTZ")
|
||||
try:
|
||||
res = model_check.check(mtz, pdb, os.path.join(wd, "model_check"))
|
||||
except Exception as ex: # a failing check must not lose the set's processing result
|
||||
return dict(out, model_status="error", model_reason=f"{type(ex).__name__}: {ex}"[:300])
|
||||
out.update({k: res.get(k) for k in ("rfree", "rwork", "rfree_deposited", "rfree_depflags",
|
||||
"rfree_depdata")},
|
||||
model_status=res.get("status"), model_reason=res.get("reason"))
|
||||
return dict(out, refmac_status="error", refmac_reason=f"{type(ex).__name__}: {ex}"[:300])
|
||||
out.update({"refmac_" + k: res.get(k) for k in ("rfree", "rwork", "rfree_depflags", "rfree_depdata")},
|
||||
refmac_status=res.get("status"), refmac_reason=res.get("reason"))
|
||||
# The fair ratio: our merge and the depositor's structure factors, both scored on the
|
||||
# depositor's free set by the same protocol. Our own free set was mostly work reflections
|
||||
# for the depositor, so R_free on it reads low.
|
||||
if out["rfree_depflags"] and out["rfree_depdata"]:
|
||||
out["rfree_ratio"] = round(out["rfree_depflags"] / out["rfree_depdata"], 4)
|
||||
if out["refmac_rfree_depflags"] and out["refmac_rfree_depdata"]:
|
||||
out["refmac_rfree_ratio"] = round(out["refmac_rfree_depflags"] / out["refmac_rfree_depdata"], 4)
|
||||
return out
|
||||
|
||||
|
||||
@@ -314,7 +356,10 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
for e in sets:
|
||||
e["reference"] = site["arms"][e["arm"]]["reference"]
|
||||
e.setdefault("input_path", os.path.join(site["arms"][e["arm"]]["data_root"], e["input"]))
|
||||
variants = {arm: [v for v in VARIANTS[site["arms"][arm]["reference"]] if v in opts["variants"]]
|
||||
for arm in arms}
|
||||
man = {"label": label, "arms": arms, "private": private, "subset": subset, "baseline": baseline,
|
||||
"variants": variants, "variant_doc": {v: VARIANT_DOC[v] for v in opts["variants"]},
|
||||
"binary": info,
|
||||
"runner_git": git(HERE, "rev-parse", "--short", "HEAD") or "unknown",
|
||||
"runner_dirty": bool(git(HERE, "status", "--porcelain", "--", ".")),
|
||||
@@ -325,20 +370,25 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
|
||||
print(f"run directory {run_dir}", flush=True)
|
||||
|
||||
results = []
|
||||
todo = sum(len(variants[e["arm"]]) for e in sets)
|
||||
# a kill by SIGTERM is recorded like Ctrl-C: the run directory then says it was aborted
|
||||
signal.signal(signal.SIGTERM, interrupted)
|
||||
try:
|
||||
for n, e in enumerate(sets, 1):
|
||||
row = run_one(run_dir, binary, e, opts)
|
||||
results.append(row)
|
||||
if row.get("gpu_others"):
|
||||
man["gpu_shared"] = True
|
||||
write_json(os.path.join(run_dir, "results.json"), results)
|
||||
print(f"[{n}/{len(sets)}] {e['arm']:8s} {e['id']:24s} {row['verdict']:8s} "
|
||||
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
|
||||
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
|
||||
for e in sets:
|
||||
# the variants of one set back to back: only the first rugnux reads the images from disk
|
||||
first_read = True
|
||||
for v in variants[e["arm"]]:
|
||||
row = run_one(run_dir, binary, e, v, first_read, opts)
|
||||
first_read = first_read and row["cmd"] is None
|
||||
results.append(row)
|
||||
if row.get("gpu_others"):
|
||||
man["gpu_shared"] = True
|
||||
write_json(os.path.join(run_dir, "results.json"), results)
|
||||
print(f"[{len(results)}/{todo}] {e['arm']:8s} {e['id']:24s} {v:6s} {row['verdict']:8s} "
|
||||
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
|
||||
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
|
||||
except KeyboardInterrupt:
|
||||
mark_aborted(run_dir, f"interrupted after {len(results)} of {len(sets)} sets")
|
||||
mark_aborted(run_dir, f"interrupted after {len(results)} of {todo} rows")
|
||||
sys.exit(f"aborted; {run_dir} is marked as such")
|
||||
|
||||
man.update(end=datetime.datetime.now().isoformat(timespec="seconds"), complete=True)
|
||||
@@ -380,11 +430,16 @@ def cmd_run(a, site):
|
||||
if not sets:
|
||||
sys.exit("nothing selected")
|
||||
subset = a.only or (f"tier {a.tier}" if a.tier else None)
|
||||
if a.model_check and not model_check:
|
||||
sys.exit("--model-check: tools/battery/model_check.py is not there")
|
||||
use_model = a.model_check if a.model_check is not None else bool(model_check)
|
||||
opts = {"threads": a.threads, "xds_range": not a.unforced, "timeout": a.timeout,
|
||||
"model_check": use_model and "open" in arms,
|
||||
known = [v for vs in VARIANTS.values() for v in vs]
|
||||
want = [v.strip() for v in a.variants.split(",")] if a.variants else known
|
||||
if set(want) - set(known):
|
||||
sys.exit(f"unknown variant(s) {', '.join(sorted(set(want) - set(known)))}; "
|
||||
f"there are {', '.join(dict.fromkeys(known))}")
|
||||
if not any(v in want for arm in arms for v in VARIANTS[site["arms"][arm]["reference"]]):
|
||||
sys.exit(f"none of the variants {a.variants} runs on the {'/'.join(arms)} arm(s)")
|
||||
opts = {"threads": a.threads, "variants": [v for v in dict.fromkeys(known) if v in want],
|
||||
"timeout": a.timeout, "model_check": a.model_check and "open" in arms,
|
||||
"pdb_cache": site.get("pdb_cache") or model_check.DEFAULT_CACHE,
|
||||
"gpulock": site["gpulock"] if a.gpulock else None, "extra_args": a.extra.split(),
|
||||
"runs_root": a.runs_root or site["runs_root"]}
|
||||
if a.gpulock and not site.get("gpulock"):
|
||||
@@ -404,7 +459,7 @@ def cmd_list(a, site):
|
||||
n = len(json.load(open(os.path.join(run, "results.json")))) \
|
||||
if os.path.exists(os.path.join(run, "results.json")) else 0
|
||||
mark = " <- baseline" if run in (site.get("baseline"), site.get("baseline_private")) else ""
|
||||
print(f"{d:48s} {state:10s} {n:3d}/{len(man['sets']):3d} sets "
|
||||
print(f"{d:48s} {state:10s} {n:3d}/{report.expected_rows(man):3d} rows "
|
||||
f"{man.get('subset') or 'full arm':20.20s} {','.join(man['arms'])}{mark}")
|
||||
|
||||
|
||||
@@ -426,7 +481,9 @@ def cmd_compare(a, site):
|
||||
f"{os.path.basename(os.path.normpath(a.run_b))}")
|
||||
doc.p(f"A: rugnux {man_a['binary'].get('version')} ({man_a['binary']['sha256'][:12]}); "
|
||||
f"B: rugnux {man_b['binary'].get('version')} ({man_b['binary']['sha256'][:12]}). "
|
||||
f"{len(rows)} sets, {sum(1 for r in rows if r[3])} changed beyond noise.")
|
||||
f"{len(rows)} set/variant rows, {sum(1 for r in rows if r[3])} changed beyond noise. "
|
||||
f"Variants compared: {report.pairs_text(report.common_variants(res_a, res_b))} "
|
||||
"(the ones both runs have).")
|
||||
report.compare_table(doc, rows, only_changed=not a.all)
|
||||
print(doc.markdown())
|
||||
if not a.rerun_changed:
|
||||
@@ -437,9 +494,11 @@ def cmd_compare(a, site):
|
||||
if not changed:
|
||||
print("nothing changed beyond noise; no rerun")
|
||||
return
|
||||
sets = [e for e in man_a["sets"] if (e["arm"], e["id"]) in changed]
|
||||
opts = dict(man_a["options"], runs_root=a.runs_root or site["runs_root"])
|
||||
sets = [e for e in man_a["sets"] if (e["arm"], e["id"]) in {k[:2] for k in changed}]
|
||||
opts = dict(man_a["options"], runs_root=a.runs_root or site["runs_root"],
|
||||
variants=[v for v in VARIANT_DOC if v in {k[2] for k in changed}])
|
||||
opts["gpulock"] = opts["gpulock"] and site.get("gpulock")
|
||||
opts.setdefault("pdb_cache", site.get("pdb_cache") or model_check.DEFAULT_CACHE)
|
||||
label = "rerunA-" + man_a["label"]
|
||||
rerun = run_sets(site, sets, man_a["arms"], os.path.join(a.run_a, "bin", "rugnux"), None, label,
|
||||
opts, ",".join(e["id"] for e in sets))
|
||||
@@ -451,10 +510,10 @@ def cmd_compare(a, site):
|
||||
if k not in changed:
|
||||
continue
|
||||
noise = [c for c in again.get(k, []) if c != "time"]
|
||||
rows_out.append([k[1], k[0], ", ".join(c for c in ch if c != "time"),
|
||||
rows_out.append([k[1], k[0], k[2], ", ".join(c for c in ch if c != "time"),
|
||||
", ".join(noise) or "-",
|
||||
"NOISE - A itself moved" if noise else "REAL - A reproduces"])
|
||||
doc.table(["set", "arm", "A -> B", "A -> A again", "reading"], rows_out)
|
||||
doc.table(["set", "arm", "variant", "A -> B", "A -> A again", "reading"], rows_out)
|
||||
doc.p(f"rerun directory: {rerun}")
|
||||
print(doc.markdown())
|
||||
|
||||
@@ -572,12 +631,12 @@ def main():
|
||||
r.add_argument("--gpulock", action="store_true",
|
||||
help="queue each rugnux behind the site's GPU lock (timing is then not a reference)")
|
||||
r.add_argument("--threads", type=int, default=0, help="-N for rugnux (default: its own choice)")
|
||||
r.add_argument("--unforced", action="store_true",
|
||||
help="do not force XDS's resolution range on the XDS-referenced arms")
|
||||
r.add_argument("--timeout", type=int, default=10800)
|
||||
r.add_argument("--model-check", action=argparse.BooleanOptionalAction, default=None,
|
||||
help="open arm: R-free of the deposited model against each merge "
|
||||
"(default: on when model_check.py is present)")
|
||||
r.add_argument("--variants", help="comma-separated subset of bare,model,xds (default: all of "
|
||||
"each arm's variants; e.g. 'bare' for what a user gets)")
|
||||
r.add_argument("--timeout", type=int, default=10800, help="per rugnux run, in s")
|
||||
r.add_argument("--model-check", action="store_true",
|
||||
help="open arm: also score each bare merge against the deposited model with "
|
||||
"REFMAC (model_check.py; needs CCP4)")
|
||||
r.add_argument("--extra", default="", help="extra rugnux arguments, applied to every set")
|
||||
r.add_argument("--baseline", help="earlier run directory to report the delta against (default: "
|
||||
"the site's persisted baseline; 'none' for no delta)")
|
||||
|
||||
@@ -1,262 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
model_sweep.py -- process a set of crystals whose data are deposited in the PDB, validate
|
||||
each merge against the depositor's own model, and compare the result with what they published.
|
||||
|
||||
It runs the open arm of the battery (the same inputs, from open.json and the site config), but
|
||||
separately from battery.py: --model places the deposited coordinates, which needs network access
|
||||
to the RCSB and adds R-free, maps and model placement to every run.
|
||||
|
||||
For every dataset given it:
|
||||
* looks the entry up in the RCSB data API and records the deposited high-resolution limit,
|
||||
R-work and R-free;
|
||||
* downloads the deposited coordinates (gzipped mmCIF -- rugnux recognises the format by
|
||||
content, so no conversion);
|
||||
* runs rugnux de novo with --model and NOTHING else: no space group, no cell, no resolution
|
||||
limit. That is the point. The question is what a bare invocation gets to, so anything told
|
||||
to the program is a thumb on the scale;
|
||||
* reads R_FREE / R_WORK and the resolution range back out of the results report, which is a
|
||||
stable interface, and not out of the console log, which is not;
|
||||
* tabulates deposited against measured, and plots the two scatters that answer the question:
|
||||
resolution, and R-free.
|
||||
|
||||
R-free here is not a refinement result. rugnux places the deposited model rigid-body and reports
|
||||
the free residual of THAT placement against its own merge, so it is a validation number: how well
|
||||
the depositor's model explains data this pipeline produced from their images. It is compared with
|
||||
the depositor's published R-free, which came from full refinement against their own reduction --
|
||||
a harder target, and deliberately so.
|
||||
|
||||
Datasets are the open-arm sets whose id is a four-character PDB entry ID (optionally with a
|
||||
suffix, as in 6h2p_native); small-molecule sets and anything without a deposited model are
|
||||
skipped with a reason.
|
||||
|
||||
tools/battery/model_sweep.py --rugnux build/rugnux/rugnux --work /tmp/modelrun [--only 5reo,9jzo]
|
||||
tools/battery/model_sweep.py --work /tmp/modelrun --score-only # re-score without re-running
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import battery # noqa: E402
|
||||
|
||||
RCSB_ENTRY = "https://data.rcsb.org/rest/v1/core/entry/{}"
|
||||
RCSB_MODEL = "https://files.rcsb.org/download/{}.cif.gz"
|
||||
PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$")
|
||||
|
||||
|
||||
def curl(url, dest=None, timeout=180):
|
||||
cmd = ["curl", "-sfL", "--max-time", str(timeout), url]
|
||||
if dest:
|
||||
cmd += ["-o", dest]
|
||||
r = subprocess.run(cmd, capture_output=True)
|
||||
return r.returncode == 0, r.stdout
|
||||
|
||||
|
||||
def deposited(pdb, models_dir):
|
||||
"""Published resolution and R-factors, plus the coordinate file, for one entry."""
|
||||
rec = {"id": pdb}
|
||||
ok, body = curl(RCSB_ENTRY.format(pdb))
|
||||
if ok:
|
||||
try:
|
||||
d = json.loads(body)
|
||||
# The refine category is a list; a multi-stage refinement has more than one block and
|
||||
# the last is the one the depositor stopped at.
|
||||
refine = d.get("refine") or []
|
||||
r = refine[-1] if refine else {}
|
||||
rec["dep_dmin"] = r.get("ls_d_res_high")
|
||||
rec["dep_rfree"] = r.get("ls_R_factor_R_free")
|
||||
rec["dep_rwork"] = r.get("ls_R_factor_R_work")
|
||||
rec["method"] = (d.get("exptl") or [{}])[0].get("method")
|
||||
except Exception as e:
|
||||
rec["meta_error"] = str(e)
|
||||
else:
|
||||
rec["meta_error"] = "entry not found"
|
||||
dest = os.path.join(models_dir, f"{pdb}.cif.gz")
|
||||
if not os.path.exists(dest) or os.path.getsize(dest) == 0:
|
||||
ok, _ = curl(RCSB_MODEL.format(pdb), dest)
|
||||
if not ok and os.path.exists(dest):
|
||||
os.remove(dest)
|
||||
if os.path.exists(dest) and os.path.getsize(dest) > 0:
|
||||
rec["model"] = dest
|
||||
else:
|
||||
rec["model_error"] = "no deposited coordinates"
|
||||
return rec
|
||||
|
||||
|
||||
def parse_report(path):
|
||||
rec = {}
|
||||
with open(path, errors="replace") as fh:
|
||||
for line in fh:
|
||||
if "=" in line:
|
||||
k, _, v = line.partition("=")
|
||||
rec[k.strip()] = v.strip()
|
||||
return rec
|
||||
|
||||
|
||||
def num(rec, key):
|
||||
try:
|
||||
return float(rec[key])
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def run_one(rugnux, model, image, outdir, threads, gpulock):
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
cmd = [gpulock] if gpulock else []
|
||||
# threads=0 -> no -N: rugnux picks its own thread count
|
||||
cmd += [rugnux, "-o", "p"] + (["-N", str(threads)] if threads else []) + ["--no-export-unmerged",
|
||||
"--spot-low-resolution", "50", "--scaling-low-resolution", "50",
|
||||
"--model", model, image]
|
||||
with open(os.path.join(outdir, "run.log"), "w") as log:
|
||||
return subprocess.run(cmd, cwd=outdir, stdout=log, stderr=subprocess.STDOUT).returncode
|
||||
|
||||
|
||||
def collect(work, meta):
|
||||
rows = []
|
||||
for pdb in sorted(meta):
|
||||
report = os.path.join(work, "runs", pdb, "p_report.txt")
|
||||
if not os.path.exists(report):
|
||||
continue
|
||||
rep = parse_report(report)
|
||||
rng = rep.get("INCLUDE_RESOLUTION_RANGE", "").split()
|
||||
rows.append(dict(
|
||||
id=pdb,
|
||||
dep_dmin=meta[pdb].get("dep_dmin"),
|
||||
dep_rfree=meta[pdb].get("dep_rfree"),
|
||||
dep_rwork=meta[pdb].get("dep_rwork"),
|
||||
our_dmin=float(rng[1]) if len(rng) == 2 else None,
|
||||
our_rfree=num(rep, "R_FREE"),
|
||||
our_rwork=num(rep, "R_WORK"),
|
||||
model_fit=rep.get("MODEL_FIT"),
|
||||
space_group=rep.get("SPACE_GROUP_NAME"),
|
||||
cc_half=num(rep, "CC_HALF"),
|
||||
))
|
||||
return rows
|
||||
|
||||
|
||||
def scatter(pts, xlabel, ylabel, title, path, floor_at_zero=False):
|
||||
try:
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
except ImportError:
|
||||
print(f" (matplotlib absent, skipping {os.path.basename(path)})")
|
||||
return
|
||||
if not pts:
|
||||
return
|
||||
xs, ys = [p[0] for p in pts], [p[1] for p in pts]
|
||||
lo = 0.0 if floor_at_zero else min(min(xs), min(ys)) * 0.95
|
||||
hi = max(max(xs), max(ys)) * 1.05
|
||||
fig, ax = plt.subplots(figsize=(6, 6))
|
||||
ax.plot([lo, hi], [lo, hi], color="0.6", lw=1, zorder=1)
|
||||
ax.scatter(xs, ys, s=26, alpha=0.8, edgecolor="none", zorder=2)
|
||||
ax.set_xlim(lo, hi)
|
||||
ax.set_ylim(lo, hi)
|
||||
ax.set_aspect("equal")
|
||||
ax.set_xlabel(xlabel)
|
||||
ax.set_ylabel(ylabel)
|
||||
ax.set_title(title)
|
||||
ax.grid(alpha=0.25, lw=0.5)
|
||||
fig.tight_layout()
|
||||
fig.savefig(path, dpi=140)
|
||||
plt.close(fig)
|
||||
print(f" wrote {path}")
|
||||
|
||||
|
||||
def report(work, rows):
|
||||
both = [r for r in rows if r["our_rfree"] is not None and r["dep_rfree"] is not None]
|
||||
res = [r for r in rows if r["our_dmin"] is not None and r["dep_dmin"] is not None]
|
||||
print(f"\n{len(rows)} processed; R-free on both sides for {len(both)}; resolution for {len(res)}\n")
|
||||
if both:
|
||||
n = len(both)
|
||||
below = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"])
|
||||
w02 = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"] + 0.02)
|
||||
w05 = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"] + 0.05)
|
||||
d = sorted(r["our_rfree"] - r["dep_rfree"] for r in both)
|
||||
print(f" R-free at or below deposited {below:3d}/{n} ({100 * below / n:.0f}%)")
|
||||
print(f" within +0.02 {w02:3d}/{n} ({100 * w02 / n:.0f}%)")
|
||||
print(f" within +0.05 {w05:3d}/{n} ({100 * w05 / n:.0f}%)")
|
||||
print(f" median delta R-free {d[len(d) // 2]:+.4f} ({d[0]:+.4f} .. {d[-1]:+.4f})")
|
||||
if res:
|
||||
deeper = sum(1 for r in res if r["our_dmin"] < r["dep_dmin"])
|
||||
d = sorted(r["our_dmin"] - r["dep_dmin"] for r in res)
|
||||
print(f" d_min deeper than deposited {deeper:3d}/{len(res)} ({100 * deeper / len(res):.0f}%)")
|
||||
print(f" median delta d_min {d[len(d) // 2]:+.3f} A ({d[0]:+.3f} .. {d[-1]:+.3f})")
|
||||
print()
|
||||
scatter([(r["dep_dmin"], r["our_dmin"]) for r in res],
|
||||
"deposited $d_{min}$ (A)", "rugnux $d_{min}$ (A)",
|
||||
f"Resolution, n={len(res)} (below the line: rugnux cuts deeper)",
|
||||
os.path.join(work, "dmin.png"))
|
||||
scatter([(r["dep_rfree"], r["our_rfree"]) for r in both],
|
||||
"deposited $R_{free}$", "rugnux $R_{free}$",
|
||||
f"R-free against the deposited model, n={len(both)} (below the line: better)",
|
||||
os.path.join(work, "rfree.png"), floor_at_zero=True)
|
||||
no_rfree = [r for r in rows if r["our_rfree"] is None]
|
||||
if no_rfree:
|
||||
print(f"\nran but reported no R-free ({len(no_rfree)}):")
|
||||
for r in no_rfree:
|
||||
print(f" {r['id']:<8}MODEL_FIT={r['model_fit']}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Process deposited datasets with --model and compare with the deposition.")
|
||||
ap.add_argument("--work", required=True, help="working directory (models, runs, results)")
|
||||
ap.add_argument("--rugnux", help="rugnux binary (required unless --score-only)")
|
||||
ap.add_argument("--site", help="battery site config (default as for battery.py)")
|
||||
ap.add_argument("--only", help="comma-separated open-arm set ids")
|
||||
ap.add_argument("--datasets", help="JSON file mapping entry ID to image file, instead of the "
|
||||
"open arm")
|
||||
ap.add_argument("--threads", type=int, default=0)
|
||||
ap.add_argument("--gpulock", action="store_true",
|
||||
help="queue each rugnux behind the site's GPU lock")
|
||||
ap.add_argument("--score-only", action="store_true", help="re-score existing runs, run nothing")
|
||||
args = ap.parse_args()
|
||||
|
||||
work = args.work
|
||||
os.makedirs(os.path.join(work, "models"), exist_ok=True)
|
||||
os.makedirs(os.path.join(work, "runs"), exist_ok=True)
|
||||
meta_path = os.path.join(work, "deposited.json")
|
||||
meta = json.load(open(meta_path)) if os.path.exists(meta_path) else {}
|
||||
|
||||
if not args.score_only:
|
||||
if not args.rugnux:
|
||||
ap.error("--rugnux is required unless --score-only")
|
||||
site = battery.load_site(args.site)
|
||||
gpulock = site.get("gpulock") if args.gpulock else None
|
||||
if args.datasets:
|
||||
inputs = json.load(open(args.datasets))
|
||||
else:
|
||||
root = site["arms"]["open"]["data_root"]
|
||||
inputs = {e["id"].lower(): os.path.join(root, e["input"])
|
||||
for e in battery.select(site, ["open"], args.only, None) if PDB_ID.match(e["id"].split("_")[0])}
|
||||
|
||||
print(f"{len(inputs)} datasets with a PDB-shaped name and an image file")
|
||||
for i, (pdb, image) in enumerate(sorted(inputs.items()), 1):
|
||||
if pdb not in meta or "model" not in meta[pdb]:
|
||||
# a set id may carry a suffix (6h2p_native): the entry is the part before it
|
||||
meta[pdb] = deposited(pdb.split("_")[0], os.path.join(work, "models"))
|
||||
json.dump(meta, open(meta_path, "w"), indent=1)
|
||||
if "model" not in meta[pdb]:
|
||||
print(f"[{i}/{len(inputs)}] {pdb}: {meta[pdb].get('model_error', 'no model')}")
|
||||
continue
|
||||
outdir = os.path.join(work, "runs", pdb)
|
||||
if os.path.exists(os.path.join(outdir, "p_report.txt")):
|
||||
continue # resumable: an interrupted sweep picks up here
|
||||
print(f"[{i}/{len(inputs)}] {pdb}", flush=True)
|
||||
rc = run_one(args.rugnux, meta[pdb]["model"], image, outdir, args.threads, gpulock)
|
||||
if rc != 0:
|
||||
print(f" exited {rc}")
|
||||
|
||||
rows = collect(work, meta)
|
||||
json.dump(rows, open(os.path.join(work, "results.json"), "w"), indent=1)
|
||||
report(work, rows)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+161
-77
@@ -46,9 +46,33 @@ def load_run(run_dir, allow_incomplete=False):
|
||||
sys.exit(f"{run_dir} is {state}, not a finished run; pass --allow-incomplete to use it anyway")
|
||||
path = os.path.join(run_dir, "results.json")
|
||||
res = json.load(open(path)) if os.path.exists(path) else []
|
||||
if man.get("results_schema", 1) < 2:
|
||||
schema1(man, res)
|
||||
return man, res
|
||||
|
||||
|
||||
def schema1(man, res):
|
||||
"""Read a run from before the variants (results schema 1) as one: it ran each set once, plain
|
||||
on the open arm and with XDS's settings on the XDS arms - unless --unforced, which left the
|
||||
range free but still passed -A, and is read as `bare`. Its R-factors came from the REFMAC check."""
|
||||
for r in res:
|
||||
r.setdefault("variant", "xds" if r["arm"] != "open" and man["options"].get("xds_range") else "bare")
|
||||
r.setdefault("first_read", True)
|
||||
if "model_status" in r:
|
||||
for k in ("rfree", "rwork", "rfree_depflags", "rfree_depdata", "rfree_ratio"):
|
||||
r["refmac_" + k] = r.pop(k, None)
|
||||
r["refmac_status"], r["refmac_reason"] = r.pop("model_status"), r.pop("model_reason", None)
|
||||
r["rfree_deposited"] = None
|
||||
man.setdefault("variants", {arm: sorted({r["variant"] for r in res if r["arm"] == arm})
|
||||
for arm in man["arms"]})
|
||||
man.setdefault("variant_doc", {})
|
||||
|
||||
|
||||
def expected_rows(man):
|
||||
"""Rows a finished run has: one per set and variant."""
|
||||
return sum(len(man.get("variants", {}).get(e["arm"], [None])) for e in man["sets"])
|
||||
|
||||
|
||||
def check_baseline(baseline, private):
|
||||
"""A baseline must have finished, and be as private as the run it is compared with: the delta
|
||||
table names its sets."""
|
||||
@@ -60,7 +84,9 @@ def check_baseline(baseline, private):
|
||||
|
||||
# Version of the results.json row schema (README.md, "results.json"). Bump it when a field changes
|
||||
# meaning or is removed; adding a field does not need it.
|
||||
RESULTS_SCHEMA = 1
|
||||
# 2: one row per set AND variant (`variant`, `first_read`); `rfree`/`rwork`/`rfree_ratio` are rugnux's
|
||||
# own --model R-factors, the REFMAC check's moved to `refmac_*`.
|
||||
RESULTS_SCHEMA = 2
|
||||
|
||||
# {arm: {old set id: current id}} from the manifests (battery.py sets it), so a run made before a
|
||||
# dataset was renamed still pairs with one made after
|
||||
@@ -68,7 +94,25 @@ ALIASES = {}
|
||||
|
||||
|
||||
def key(r):
|
||||
return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"]))
|
||||
return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"]), r["variant"])
|
||||
|
||||
|
||||
# the order variants are listed in (battery.VARIANTS runs them in this order within an arm)
|
||||
VARIANT_ORDER = ("bare", "model", "xds")
|
||||
|
||||
|
||||
def variants_of(res):
|
||||
return [v for v in VARIANT_ORDER if any(r["variant"] == v for r in res)]
|
||||
|
||||
|
||||
def common_variants(res_a, res_b):
|
||||
"""The (arm, variant) pairs both runs ran - what a comparison of the two covers."""
|
||||
pairs = lambda res: {(r["arm"], r["variant"]) for r in res}
|
||||
return sorted(pairs(res_a) & pairs(res_b), key=lambda p: (p[0], VARIANT_ORDER.index(p[1])))
|
||||
|
||||
|
||||
def pairs_text(pairs):
|
||||
return ", ".join(f"{arm} {v}" for arm, v in pairs) or "none"
|
||||
|
||||
|
||||
def f(x, fmt="{:.2f}", dash="-"):
|
||||
@@ -123,9 +167,12 @@ def changes(ra, rb):
|
||||
|
||||
|
||||
def compare_rows(res_a, res_b):
|
||||
"""[(key, ra, rb, changes)] over the sets both runs have, then those only one has."""
|
||||
a = {key(r): r for r in res_a}
|
||||
b = {key(r): r for r in res_b}
|
||||
"""[(key, ra, rb, changes)] over the (set, variant) rows both runs have, then those only one
|
||||
has. Only the variants both runs ran on an arm are compared: a run of just `bare` against a
|
||||
full one is a comparison of `bare`."""
|
||||
common = common_variants(res_a, res_b)
|
||||
a = {key(r): r for r in res_a if (r["arm"], r["variant"]) in common}
|
||||
b = {key(r): r for r in res_b if (r["arm"], r["variant"]) in common}
|
||||
rows = [(k, a[k], b[k], changes(a[k], b[k])) for k in sorted(a) if k in b]
|
||||
rows += [(k, a[k], None, ["only in A"]) for k in sorted(a) if k not in b]
|
||||
rows += [(k, None, b[k], ["only in B"]) for k in sorted(b) if k not in a]
|
||||
@@ -139,14 +186,14 @@ def arrow(x, y, fmt=lambda v: str(v)):
|
||||
|
||||
|
||||
def compare_table(doc, rows, only_changed=True):
|
||||
head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
|
||||
head = ["set", "arm", "variant", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
|
||||
"cell dev %", "R_free", "time s", "beyond noise"]
|
||||
out = []
|
||||
for (arm, s), ra, rb, ch in rows:
|
||||
for (arm, s, v), ra, rb, ch in rows:
|
||||
if only_changed and not ch:
|
||||
continue
|
||||
ra, rb = ra or {}, rb or {}
|
||||
out.append([s, arm, arrow(ra.get("verdict"), rb.get("verdict")),
|
||||
out.append([s, arm, v, arrow(ra.get("verdict"), rb.get("verdict")),
|
||||
arrow(ra.get("sg"), rb.get("sg")),
|
||||
arrow(ra.get("d_min"), rb.get("d_min"), lambda v: f"{v:.2f}"),
|
||||
arrow(ra.get("isa"), rb.get("isa"), lambda v: f"{v:.1f}"),
|
||||
@@ -187,67 +234,118 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
state = run_state(man)
|
||||
if state != "complete":
|
||||
why = f" ({man['aborted']['reason']})" if man.get("aborted") else ""
|
||||
doc.p(f"RUN {state.upper()}{why}: {len(res)} of {len(man['sets'])} sets have a result. "
|
||||
"Not a reference and not comparable with a finished run.")
|
||||
doc.p(f"RUN {state.upper()}{why}: {len(res)} of {expected_rows(man)} set/variant rows have a "
|
||||
"result. Not a reference and not comparable with a finished run.")
|
||||
variants = variants_of(res)
|
||||
doc.ul([
|
||||
f"rugnux {b.get('version', '?')}, sha256 {b['sha256'][:16]}, build flags: {b.get('flags') or 'unknown'}",
|
||||
f"runner {man['runner_git']}, host {man['host']}, {man['start']} -> {man.get('end') or 'NOT FINISHED'}",
|
||||
f"arms: {', '.join(man['arms'])}; {len(man['sets'])} sets"
|
||||
+ (f" (subset: {man['subset']})" if man.get("subset") else " (full arm)"),
|
||||
"resolution: XDS's range forced on the XDS-referenced arms" if man["options"]["xds_range"]
|
||||
else "resolution: rugnux's own choice everywhere (unforced)",
|
||||
"variants: " + "; ".join(f"{arm}: {', '.join(man['variants'][arm])}" for arm in man["arms"]
|
||||
if man["variants"].get(arm)),
|
||||
"timing: NOT a reference - the GPU was shared" if man.get("gpu_shared")
|
||||
else "timing: GPU was not shared during the run",
|
||||
f"command options: {' '.join(man['options']['extra_args']) or '(none beyond output selection)'}",
|
||||
])
|
||||
] + [f"variant {v}: {man['variant_doc'][v]}" for v in variants if man["variant_doc"].get(v)])
|
||||
|
||||
forced = {e["arm"] for e in man["sets"]
|
||||
if man["options"]["xds_range"] and e.get("reference") == "xds"}
|
||||
order = {arm: i for i, arm in enumerate(man["arms"])}
|
||||
doc.h(2, "Timing")
|
||||
rows = []
|
||||
for v in variants:
|
||||
rs = [r for r in res if r["variant"] == v and r.get("wall_s") is not None]
|
||||
rows.append([v, len(rs), f"{sum(bool(r.get('first_read')) for r in rs)}/{len(rs)}",
|
||||
f(median(r["wall_s"] for r in rs), "{:.0f}"),
|
||||
f(sum(r["wall_s"] for r in rs) / 60, "{:.1f}")])
|
||||
doc.table(["variant", "rows run", "first read of the set", "median time s", "total time min"],
|
||||
rows, num=range(1, 5))
|
||||
doc.p("The variants of a set run back to back, in the order listed. The first one to run reads "
|
||||
"the images from disk (unless they were still in the page cache from before the run); the "
|
||||
"later ones find them in the page cache. So only the first variant's times include reading "
|
||||
"the data - normally `bare`, which runs first on every arm - and times are comparable only "
|
||||
"between runs of the same variant.")
|
||||
|
||||
def gain(rs, arm):
|
||||
return "forced" if arm in forced else f(median(r.get("res_gain_pct") for r in rs), "{:+.1f}%")
|
||||
derived = {r["set"]: r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"}
|
||||
if derived:
|
||||
doc.p("XDS merged past its own signal (CC1/2 of its finest shell not significant), so the "
|
||||
"reference d_min is where XDS's CC1/2 falls through 0.30, and that is also the range "
|
||||
"the xds variant forces on rugnux: "
|
||||
+ "; ".join(f"{s} (XDS range {f(r.get('d_min_xds'))} A, reference "
|
||||
f"{f(r.get('d_min_ref'))} A)" for s, r in sorted(derived.items()))
|
||||
+ ". XDS's pooled R_meas, CC1/2 and completeness are over its whole range.")
|
||||
|
||||
doc.h(2, "Summary")
|
||||
for v in variants:
|
||||
variant_section(doc, man, [r for r in res if r["variant"] == v], v)
|
||||
|
||||
if baseline:
|
||||
bman, bres = load_run(baseline)
|
||||
doc.h(2, f"Delta vs baseline {os.path.basename(os.path.normpath(baseline))}")
|
||||
common = common_variants(bres, res)
|
||||
doc.p(f"Baseline rugnux {bman['binary'].get('version', '?')}. Pass rates on the set/variant "
|
||||
f"rows both runs have (variants compared: {pairs_text(common)}):")
|
||||
both = {key(r) for r in res} & {key(r) for r in bres}
|
||||
rows = []
|
||||
for arm, v in common:
|
||||
ra = [r for r in bres if key(r) in both and r["arm"] == arm and r["variant"] == v]
|
||||
rb = [r for r in res if key(r) in both and r["arm"] == arm and r["variant"] == v]
|
||||
rows.append([arm, v, len(rb), rate(ra), rate(rb)])
|
||||
doc.table(["arm", "variant", "common sets", "baseline", "this run"], rows, num=(2,))
|
||||
doc.p("Rows whose result moved beyond noise (thresholds in report.py NOISE; confirm with "
|
||||
"`compare --rerun-changed` before believing a single-set change):")
|
||||
compare_table(doc, compare_rows(bres, res))
|
||||
return doc
|
||||
|
||||
|
||||
def variant_section(doc, man, res, v):
|
||||
"""Summary, populations, distributions, plots, failures and the per-set table of one variant."""
|
||||
forced = v == "xds" # XDS's resolution range imposed: d_min ratio 1 by construction
|
||||
arms = [arm for arm in man["arms"] if any(r["arm"] == arm for r in res)]
|
||||
order = {arm: i for i, arm in enumerate(arms)}
|
||||
|
||||
def gain(rs):
|
||||
return "forced" if forced else f(median(r.get("res_gain_pct") for r in rs), "{:+.1f}%")
|
||||
|
||||
doc.h(2, f"Variant {v}")
|
||||
if man["variant_doc"].get(v):
|
||||
doc.p(man["variant_doc"][v][0].upper() + man["variant_doc"][v][1:] + ".")
|
||||
head = ["arm", "sets", "pass", "fail", "unscored", "not run", "pass rate",
|
||||
"median res. gain", "median ISa", "median R_meas", "median time s", "total time min"]
|
||||
rows = []
|
||||
for arm in man["arms"]:
|
||||
for arm in arms:
|
||||
rs = [r for r in res if r["arm"] == arm]
|
||||
cnt = {v: sum(r["verdict"] == v for r in rs) for v in ("pass", "fail", "unscored", "not_run")}
|
||||
cnt = {x: sum(r["verdict"] == x for r in rs) for x in ("pass", "fail", "unscored", "not_run")}
|
||||
xs = crystals(rs)
|
||||
rows.append([arm, len(rs), cnt["pass"], cnt["fail"], cnt["unscored"], cnt["not_run"], rate(rs),
|
||||
gain(xs, arm), f(median(r.get("isa") for r in xs), "{:.1f}"),
|
||||
gain(xs), f(median(r.get("isa") for r in xs), "{:.1f}"),
|
||||
pct(median(r.get("r_meas") for r in xs)),
|
||||
f(median(r.get("wall_s") for r in rs), "{:.0f}"),
|
||||
f(sum(r.get("wall_s") or 0 for r in rs) / 60, "{:.1f}")])
|
||||
doc.table(head, rows, num=range(1, 12))
|
||||
doc.svg(verdict_bars([(arm, {v: sum(r["verdict"] == v and r["arm"] == arm for r in res)
|
||||
for v in ("pass", "fail", "unscored", "not_run")})
|
||||
for arm in man["arms"]]), "verdicts per arm")
|
||||
doc.svg(verdict_bars([(arm, {x: sum(r["verdict"] == x and r["arm"] == arm for r in res)
|
||||
for x in ("pass", "fail", "unscored", "not_run")})
|
||||
for arm in arms]), f"{v}: verdicts per arm")
|
||||
doc.p("Pass rate is over the scored sets (pass + fail). Resolution gain is (reference d_min - "
|
||||
"our d_min) / reference d_min: positive = finer than the deposition, or than XDS; 'forced' "
|
||||
"where XDS's resolution range was imposed, which makes it zero by construction. Medians "
|
||||
"leave out the no-crystal controls. R_meas and CC1/2 are pooled over each run's own "
|
||||
"resolution range, so they describe a run and do not rank two.")
|
||||
|
||||
doc.h(3, "By population")
|
||||
doc.h(3, f"{v}: by population")
|
||||
rows = []
|
||||
for arm in man["arms"]:
|
||||
for arm in arms:
|
||||
rs = [r for r in res if r["arm"] == arm]
|
||||
for tag in sorted({t for r in rs for t in r.get("tags", [])}):
|
||||
sub = [r for r in rs if tag in r.get("tags", [])]
|
||||
rows.append([arm, tag, len(sub), rate(sub), gain(sub, arm)])
|
||||
rows.append([arm, tag, len(sub), rate(sub), gain(sub)])
|
||||
doc.table(["arm", "population", "sets", "pass rate", "median res. gain"], rows, num=(2, 4))
|
||||
|
||||
doc.h(3, "Distributions (sets that merged, controls left out)")
|
||||
doc.h(3, f"{v}: distributions (sets that merged, controls left out)")
|
||||
rows = []
|
||||
for arm in man["arms"]:
|
||||
for arm in arms:
|
||||
rs = crystals(r for r in res if r["arm"] == arm)
|
||||
for name, lab, fm in (("isa", "ISa", "{:.1f}"), ("r_meas", "R_meas", None),
|
||||
("cc_half", "CC1/2", "{:.3f}"), ("res_gain_pct", "res. gain %", "{:+.1f}"),
|
||||
("wall_s", "time s", "{:.0f}")):
|
||||
if name == "res_gain_pct" and arm in forced:
|
||||
("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")):
|
||||
if name == "res_gain_pct" and forced:
|
||||
continue
|
||||
q = quartiles(r.get(name) for r in rs)
|
||||
if q:
|
||||
@@ -255,37 +353,36 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
rows.append([arm, lab, sum(r.get(name) is not None for r in rs)] + cells)
|
||||
doc.table(["arm", "metric", "n", "min", "q1", "median", "q3", "max"], rows, num=range(2, 8))
|
||||
|
||||
doc.h(3, "Per set, against the reference")
|
||||
for arm in man["arms"]:
|
||||
if arm in forced:
|
||||
doc.h(3, f"{v}: per set, against the reference")
|
||||
for arm in arms:
|
||||
if forced:
|
||||
doc.p(f"{arm}: d_min forced to the reference's, so its ratio is 1 by construction.")
|
||||
continue
|
||||
pts = [(r["set"], r["d_min"] / r["d_min_ref"], r["verdict"],
|
||||
f"d_min {r['d_min']:.2f} vs {r['d_min_ref']:.2f} A, {r['verdict']}")
|
||||
for r in res if r["arm"] == arm and r.get("d_min") and r.get("d_min_ref")]
|
||||
if pts:
|
||||
doc.svg(ratio_dots(pts, f"{arm}: d_min ratio"),
|
||||
f"{arm}: d_min(rugnux) / d_min(reference), one point per set, sorted; below 1 = "
|
||||
doc.svg(ratio_dots(pts, f"{arm} {v}: d_min ratio"),
|
||||
f"{arm}, {v}: d_min(rugnux) / d_min(reference), one point per set, sorted; below 1 = "
|
||||
"finer than the reference")
|
||||
pts = [(r["set"], r["rfree_ratio"], r["verdict"],
|
||||
f"R_free {r['rfree_depflags']:.3f} vs {r['rfree_depdata']:.3f} on the deposited data")
|
||||
f"R_free {r['rfree']:.3f} vs deposited {r['rfree_deposited']:.3f}")
|
||||
for r in res if r.get("rfree_ratio")]
|
||||
if pts:
|
||||
doc.svg(ratio_dots(pts, "R_free ratio"),
|
||||
"open: R_free of the deposited model against our merge / against the deposited structure "
|
||||
"factors, both on the depositor's free set with the same protocol, one point per set, "
|
||||
"sorted; below 1 = our data fit the model better")
|
||||
doc.svg(ratio_dots(pts, f"{v}: R_free ratio"),
|
||||
"R_free of the deposited model against our merge, as rugnux reports it with --model / "
|
||||
"the R_free the depositor published (from full refinement against their own data), one "
|
||||
"point per set, sorted; below 1 = lower than published")
|
||||
pts = [(r["set"], r["refmac_rfree_ratio"], r["verdict"],
|
||||
f"R_free {r['refmac_rfree_depflags']:.3f} vs {r['refmac_rfree_depdata']:.3f} on the deposited data")
|
||||
for r in res if r.get("refmac_rfree_ratio")]
|
||||
if pts:
|
||||
doc.svg(ratio_dots(pts, f"{v}: REFMAC R_free ratio"),
|
||||
"REFMAC check (--model-check): R_free of the deposited model against our merge / against "
|
||||
"the deposited structure factors, both on the depositor's free set with the same protocol, "
|
||||
"one point per set, sorted; below 1 = our data fit the model better")
|
||||
|
||||
derived = [r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"]
|
||||
if derived:
|
||||
doc.p("XDS merged past its own signal (CC1/2 of its finest shell not significant), so the "
|
||||
"reference d_min is where XDS's CC1/2 falls through 0.30 (rule "
|
||||
f"{derived[0]['d_min_ref_rule']}), and that is also the range forced on rugnux: "
|
||||
+ "; ".join(f"{r['set']} (XDS range {f(r.get('d_min_xds'))} A, reference "
|
||||
f"{f(r.get('d_min_ref'))} A)" for r in derived)
|
||||
+ ". XDS's pooled R_meas, CC1/2 and completeness are over its whole range.")
|
||||
|
||||
doc.h(2, "Failures")
|
||||
doc.h(3, f"{v}: failures")
|
||||
fails = [r for r in res if r["verdict"] == "fail"]
|
||||
doc.table(["set", "arm", "cause", "reason"],
|
||||
[[r["set"], r["arm"], r["cause"], r["reason"]] for r in fails])
|
||||
@@ -293,26 +390,14 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
if other:
|
||||
doc.p("Not scored: " + "; ".join(f"{r['set']} ({r['reason']})" for r in other))
|
||||
|
||||
if baseline:
|
||||
bman, bres = load_run(baseline)
|
||||
doc.h(2, f"Delta vs baseline {os.path.basename(os.path.normpath(baseline))}")
|
||||
doc.p(f"Baseline rugnux {bman['binary'].get('version', '?')}. Pass rates on the sets both "
|
||||
"runs have:")
|
||||
both = {key(r) for r in res} & {key(r) for r in bres}
|
||||
rows = []
|
||||
for arm in man["arms"]:
|
||||
ra = [r for r in bres if key(r) in both and r["arm"] == arm]
|
||||
rb = [r for r in res if key(r) in both and r["arm"] == arm]
|
||||
rows.append([arm, len(rb), rate(ra), rate(rb)])
|
||||
doc.table(["arm", "common sets", "baseline", "this run"], rows, num=(1,))
|
||||
doc.p("Sets whose result moved beyond noise (thresholds in report.py NOISE; confirm with "
|
||||
"`compare --rerun-changed` before believing a single-set change):")
|
||||
compare_table(doc, compare_rows(bres, res))
|
||||
|
||||
doc.h(2, "Per set")
|
||||
doc.h(3, f"{v}: per set")
|
||||
model = [("R_free", "rfree"), ("R_work", "rwork"), ("dep R_free", "rfree_deposited"),
|
||||
("R_free ratio", "rfree_ratio")] if any(r.get("rfree") for r in res) else []
|
||||
model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"),
|
||||
("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else []
|
||||
head = ["set", "arm", "verdict", "space group", "ref", "cell dev %", "V ratio", "d_min", "ref d_min",
|
||||
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx", "R_free", "dep R_free",
|
||||
"time s", "note"]
|
||||
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx"] + \
|
||||
[h for h, _ in model] + ["time s", "note"]
|
||||
rows = []
|
||||
for r in sorted(res, key=lambda r: (order.get(r["arm"], 9), r["set"])):
|
||||
rows.append([r["set"], r["arm"], r["verdict"], r.get("sg") or "-", r.get("sg_ref") or "-",
|
||||
@@ -320,10 +405,9 @@ def build(run_dir, baseline=None, allow_incomplete=False):
|
||||
f(r.get("d_min")), f(r.get("d_min_ref")), f(r.get("res_gain_pct"), "{:+.1f}%"),
|
||||
f(r.get("completeness"), "{:.1f}"), f(r.get("multiplicity"), "{:.1f}"),
|
||||
pct(r.get("r_meas")), f(r.get("cc_half"), "{:.3f}"), f(r.get("isa"), "{:.1f}"),
|
||||
f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0),
|
||||
f(r.get("rfree"), "{:.3f}"), f(r.get("rfree_deposited"), "{:.3f}"),
|
||||
f(r.get("wall_s"), "{:.0f}"), r.get("reason") or ""])
|
||||
doc.table(head, rows, num=range(5, 20),
|
||||
f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0)]
|
||||
+ [f(r.get(k), "{:.3f}") for _, k in model]
|
||||
+ [f(r.get("wall_s"), "{:.0f}"), r.get("reason") or ""])
|
||||
doc.table(head, rows, num=range(5, len(head) - 1),
|
||||
row_class=lambda row: "fail" if row[2] == "fail" else "",
|
||||
cell_class=lambda i, v: {"pass": "pass", "fail": "fail"}.get(v) if i == 2 else None)
|
||||
return doc
|
||||
cell_class=lambda i, x: {"pass": "pass", "fail": "fail"}.get(x) if i == 2 else None)
|
||||
@@ -78,7 +78,7 @@ def sg_name(sgno):
|
||||
|
||||
def judge(entry, rep, run_note):
|
||||
"""Score one set. entry is its manifest row (with 'ref'), rep its parsed report, run_note
|
||||
what the runner saw ('', 'timeout', 'exit N: message', 'no input')."""
|
||||
what the runner saw ('', 'timeout', 'exit N: message', 'no input', 'no model: why')."""
|
||||
ref = dict(entry.get("ref") or {})
|
||||
if entry.get("ref_override"): # a reference measured by hand, replacing the file's
|
||||
ref.update(entry["ref_override"])
|
||||
@@ -96,6 +96,9 @@ def judge(entry, rep, run_note):
|
||||
"rugnux_wall_s": fnum(rep.get("WALL_TIME")), "rugnux_verdict": rep.get("VERDICT_TEXT"),
|
||||
"r_meas_ref": ref.get("r_meas"), "cc_half_ref": ref.get("cc_half"), "isa_ref": ref.get("isa"),
|
||||
"completeness_ref": ref.get("completeness"), "sg_relation": None,
|
||||
# rugnux's own R-factors of the deposited model, when it was given one (--model)
|
||||
"rfree": fnum(rep.get("R_FREE")), "rwork": fnum(rep.get("R_WORK")),
|
||||
"model_fit": rep.get("MODEL_FIT"),
|
||||
}
|
||||
if rep.get("SPACE_GROUP_NUMBER"):
|
||||
r["sgno"] = int(fnum(rep["SPACE_GROUP_NUMBER"]))
|
||||
@@ -125,6 +128,8 @@ def judge(entry, rep, run_note):
|
||||
if not have_lattice:
|
||||
if run_note == "no input":
|
||||
return verdict("not_run", "no_input", "input file not found")
|
||||
if run_note.startswith("no model"):
|
||||
return verdict("not_run", "no_model", run_note)
|
||||
if run_note == "timeout":
|
||||
return verdict("fail", "timeout", "timed out")
|
||||
if re.search(r"Error reading input|Cannot open", run_note):
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
"gpulock": null,
|
||||
"baseline": null,
|
||||
"baseline_private": null,
|
||||
"pdb_cache": "/path/to/battery/pdb_cache",
|
||||
"arms": {
|
||||
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/path/to/battery/open"},
|
||||
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/path/to/battery/inhouse"}
|
||||
|
||||
Reference in new issue
Block a user