tools/battery: run every variant of a set back to back

The battery is bound by reading images from disk, so instead of one battery run per setting each
set now runs in all its variants in a row while its images are still in the page cache:

  open arm        bare, then model  (rugnux --model <deposited coordinates>)
  XDS arms        bare, then xds    (XDS's resolution range forced, -A where XDS was anomalous)

bare (plain rugnux <input>) runs first on every arm, so its time always carries the set's disk
read whichever variants are selected; each row records first_read and the report's timing table
says which variant's times are cold. --variants runs a subset; --unforced is gone (--variants
bare). Rows, work dirs (work/<arm>/<set>/<variant>/), compare and the report are per variant;
compare pairs (arm, set, variant) on the variants both runs have on an arm, and a schema-1 run is
read as one variant. The model variant takes the coordinates and published R-free from
model_check's RCSB cache (site key pdb_cache) and records rugnux's R-free/R-work and the ratio;
the REFMAC check is now opt-in (--model-check) and its keys moved to refmac_*. results_schema 2.

model_sweep.py is retired: the model variant replaces it (its --spot/--scaling-low-resolution 50
were rugnux's defaults, so the command is the same).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5 committed 2026-09-20 18:45:03 +02:00
1 parent 162aaafef7
commit bb0648ae87
8 files changed
+397 -450

No files matched your search

+7
View File
@@ -106,9 +106,16 @@ scripts.
```
tools/battery/battery.py run --rugnux build/rugnux/rugnux --only 5reo,lyso_ref # named sets
tools/battery/battery.py run --rugnux build/rugnux/rugnux --tier smoke # smoke tier
tools/battery/battery.py run --rugnux build/rugnux/rugnux --variants bare # one variant only
tools/battery/battery.py compare RUN_A RUN_B [--rerun-changed]
```
Each set runs in all its arm's **variants** back to back, so its images are read from disk once:
`bare` (plain `rugnux <input>`, what a user gets) first everywhere, then `model` (`--model` with the
deposited coordinates, R-free vs the published one) on the open arm, or `xds` (XDS's resolution
range forced, `-A` where XDS was anomalous) on the XDS arms. Only the first variant's time includes
the disk read; compare times within one variant. The REFMAC check is opt-in (`--model-check`).
- **Never start a full battery run (no `--only`/`--tier`) without the owner's go-ahead.**
- Judge a change against a **persisted baseline run** (the site config's `baseline`) with
`compare`, never against numbers remembered from an earlier session.
+3 -3
View File
@@ -40,9 +40,9 @@ pipeline is exercised on, and the DOI to cite for each, are listed in
* `tools/battery/battery.py` - the rotation battery, the one canonical way to judge a rotation
change: public PDB depositions scored against the deposition, in-house standard crystals and
no-crystal controls scored against XDS, and a local-only private arm. Its manifests, the run
and compare protocol and the report are described in `tools/battery/README.md`.
* `tools/battery/model_sweep.py` - the battery's open arm run with `--model`: each merge validated
against the depositor's own model and compared with the published resolution and R-work/R-free.
and compare protocol and the report are described in `tools/battery/README.md`. Each set runs
in several variants back to back: plain, with the deposited model (`--model`, R-free against the
published one) on the open arm, and with XDS's settings on the XDS arms.
* `rugnux_stills_ab.py` - the stills analogue of the battery: scores a change on a serial
dataset by what it does to the merge. Takes its dataset list from outside the repository.
* `rugnux_anomalous.py` - the anomalous-peak-height arbiter, below.
+98 -45
View File
@@ -23,6 +23,45 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X
screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported.
`score.py` has the details.
## Variants
Every set is run more than once, with different settings, and each run is a **variant**. The
variants of one set run straight after each other, before the next set starts. The battery is
limited by reading images from disk, and this way only the first variant reads them; the others
find them in the page cache.
| arm (reference) | variants, in the order they run |
|---|---|
| open (deposition) | `bare`, then `model` |
| inhouse, private (XDS) | `bare`, then `xds` |
- **`bare`**: plain `rugnux <input>` and nothing else (no `-A`, no resolution range). This is
what a user gets. It is scored against the reference like the others, and its d_min ratio
against the reference is plotted.
- **`model`** (open arm): `rugnux --model <deposited coordinates>`. The coordinates (`<entry>.cif.gz`)
and the published R-free come from the RCSB and are cached in the site's `pdb_cache` (default
`/data/battery/pdb_cache`), so the first run of a new entry needs network access. The entry is the
set id up to the first `_`, so `6h2p_native` uses `6h2p`. A set with no PDB entry (small
molecules) or whose coordinates cannot be downloaded gets a `not_run` row with cause `no_model`.
The row records the R-free and R-work that rugnux reports for the model, the published R-free
and their ratio, which the report plots. rugnux's R-free is that of a rigid-body placement against
its own merge, and the published one comes from full refinement against the depositor's own
data, so the ratio is not a like-for-like score; watch how it moves between runs.
- **`xds`** (XDS arms): XDS's reference settings. XDS's resolution range is forced (see "The XDS
reference d_min" below), and where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true`
in the manifest) rugnux gets `-A`. Both programs then merge the same reflections the same way.
The variants are defined in one place, `VARIANTS` in `battery.py`. `--variants` runs a subset,
for example `--variants bare`, and they always run in the table's order. `--unforced` is gone:
`--variants bare` gives rugnux its own resolution cut (and, unlike `--unforced`, no `-A`).
**Order and timing.** `bare` goes first on every arm. So its time always includes reading the
data from disk (unless the images were still cached from before the run), whichever variants a run
selects, and it is the time a user sees. The time of a later variant is taken on cached images.
Each row records whether it was its set's `first_read`, and the report's timing table says which
variant carries the disk reads. Compare times of one variant between runs only, never across
variants.
The private arm is never mixed with the others. A private run gets its own run directory
(suffix `_private`) and its own report (`report_PRIVATE.*`), and it cannot be the baseline of a
public report or be written into the repository. Its names, paths and values never go into a
@@ -86,6 +125,7 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
"gpulock": "/data/tmp/gpulock",
"baseline": "/data/battery/runs/<a blessed full run>",
"baseline_private": null,
"pdb_cache": "/data/battery/pdb_cache",
"arms": {
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/home/data/open"},
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/home/data/inhouse"},
@@ -97,7 +137,8 @@ export `JFJOCH_BATTERY_SITE=/data/battery/site.json`.
A relative `manifest` path is resolved against `tools/battery/`. `gpulock` is a command that
`rugnux` is prefixed with under `--gpulock`. At PSI it is a wrapper script that queues for one of
the shared GPU slots. Set it to `null` if you have no such wrapper.
the shared GPU slots. Set it to `null` if you have no such wrapper. `pdb_cache` (optional) is where
the deposited models and their metadata are cached for the `model` variant and the REFMAC check.
## Running
@@ -109,7 +150,7 @@ $B run --rugnux $R # full battery: open + inhou
$B run --rugnux $R --tier smoke # 22 sets covering readers, symmetry, known hard cases
$B run --rugnux $R --only 5reo,lyso_ref # named sets
$B run --rugnux $R --arm private # the private arm, on its own
$B run --rugnux $R --arm inhouse --unforced # XDS arms without XDS's resolution forced
$B run --rugnux $R --variants bare # only what a user gets (no model, no XDS settings)
```
Useful options:
@@ -119,7 +160,8 @@ Useful options:
- `--gpulock` queues every rugnux behind the site's GPU wrapper.
- `--extra "<args>"` adds rugnux arguments to every set.
- `--baseline RUN|none` sets the run the report is compared against.
- `--model-check` / `--no-model-check` turns the open-arm R-free check on or off (see below).
- `--variants bare,model,xds` runs a subset of the variants (default: all of them).
- `--model-check` adds the REFMAC check to the open arm's `bare` rows (off by default, see below).
A run prints one line per set and ends by writing its report. The **smoke** tier is marked in the
manifests (`"tiers": {"smoke": "<why this set>"}`).
@@ -128,23 +170,23 @@ manifests (`"tiers": {"smoke": "<why this set>"}`).
two full runs cannot overlap. Do not start one without the owner's go-ahead: it takes hours, and
everything else on the machine competes with it.
### The model check (open arm)
### The REFMAC model check (open arm, opt-in)
`tools/battery/model_check.py` scores each open-arm merge (`p.mtz`) against the deposited model,
The `model` variant is the battery's R-free check. `--model-check` adds a second, independent one
that needs CCP4: `tools/battery/model_check.py` scores each open-arm `bare` merge (`p.mtz`) against
the deposited model,
unmodified (ligands, waters and hydrogens kept): REFMAC's rigid-body mode, R-factors taken before
any shift, the data reindexed into the model's setting and reduced to the deposited space group.
Our own free set was mostly work reflections for the depositor, so `rfree` reads low. The fair
comparison is `rfree_depflags` (our data on the depositor's free set) against `rfree_depdata` (the
depositor's structure factors, same free set, same protocol); `rfree_ratio` is that ratio, and the
report plots it. It is null when the entry deposited no structure factors with a free set.
`rfree_deposited` is the value in the entry, often from twin or full refinement. The runner records
`rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`,
`model_status` and `model_reason`. The check is on by default when the module exists. The PDB entry
is the set id up to the first `_`, so `6h2p_native` is checked as `6h2p`. Small-molecule sets are skipped.
Our own free set was mostly work reflections for the depositor, so `refmac_rfree` reads low. The
fair comparison is `refmac_rfree_depflags` (our data on the depositor's free set) against
`refmac_rfree_depdata` (the depositor's structure factors, same free set, same protocol);
`refmac_rfree_ratio` is that ratio, and the report plots it. It is null when the entry deposited no
structure factors with a free set. The runner records `refmac_rfree`, `refmac_rwork`,
`refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status` and
`refmac_reason`. Small-molecule sets are skipped.
`model_sweep.py` is a different tool: it runs `rugnux --model` over the open arm (same inputs, same
site config) and compares the R-free that rugnux itself reports with the published value. It needs
network access to the RCSB and runs separately from `battery.py`.
`model_sweep.py`, which ran `rugnux --model` over the open arm as a separate sweep, is retired. The
`model` variant does the same inside the battery.
## A run directory
@@ -155,9 +197,9 @@ read-only when the run completes.
|---|---|
| `bin/rugnux` | the exact binary that ran (sha256 in the manifest) |
| `manifest.json` | the label, arms, subset, options, binary version, build flags and source HEAD, runner git and dirty flag, host, pid, start and end, `complete` / `aborted`, `gpu_shared`, and every set with its resolved input, reference and tags |
| `results.json` | one row per set (the schema is below) |
| `results.json` | one row per set and variant (the schema is below) |
| `report.md`, `report.html` | the report (`report_PRIVATE.*` for a private run) |
| `work/<arm>/<set>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
| `work/<arm>/<set>/<variant>/` | `run.log`, `p_report.txt`, `p.mtz`, `p.cif`, ... and `model_check/` |
**Run states** (`battery.py list`): `complete`; `running` (its runner process is alive);
`aborted` (interrupted by Ctrl-C or SIGTERM, or marked with `battery.py abort RUN --reason ...`);
@@ -165,16 +207,24 @@ read-only when the run completes.
covers an arbitrary part of its sets. `compare` and `report` refuse it unless given
`--allow-incomplete`, and the report then says so at the top.
### results.json (schema 1)
### results.json (schema 2)
This is a flat list with one object per set, and every arm uses the same keys. Missing values are
`null`. `manifest.json` records the schema version as `results_schema`.
This is a flat list with one object per set and variant, and every arm uses the same keys. Missing
values are `null`. `manifest.json` records the schema version as `results_schema`, and the variants
each arm ran as `variants`.
Schema 2 added `variant` and `first_read`. It also changed the meaning of `rfree`, `rwork` and
`rfree_ratio`: they are now rugnux's own `--model` values (model variant), and the REFMAC check's
moved to `refmac_*`. A schema-1 run is still read, and compared, as if each set had one variant:
`bare` on the open arm; on the XDS arms `xds`, or `bare` if it was run `--unforced` (which still
passed `-A`).
| key | meaning |
|---|---|
| `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran |
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) |
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` |
| `set`, `arm`, `variant`, `tags`, `input`, `cmd` | the set's id, arm, variant (`bare`, `model`, `xds`), population tags, input file, and the exact command that ran |
| `first_read` | true for the first variant of its set that ran: the one that read the images from disk |
| `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing, or no model for the `model` variant) |
| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge`, `false_lattice`; or `no_reference` / `no_input` / `no_model` |
| `reason` | one line for a human |
| `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours / the reference's |
| `sg_relation` | open arm: how the two groups relate (`sgequiv`) |
@@ -183,7 +233,8 @@ This is a flat list with one object per set, and every arm uses the same keys. M
| `d_min_ref_rule`, `d_min_xds` | XDS arms: how `d_min_ref` was derived (below), and the range XDS merged |
| `r_meas`, `cc_half`, `isa`, `completeness`, `multiplicity`, `i_over_sigma`, `indexing_rate`, `images` | from rugnux's `_report.txt` |
| `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref` | XDS's overall values, over XDS's own range |
| `rfree`, `rwork`, `rfree_deposited`, `rfree_depflags`, `rfree_depdata`, `rfree_ratio`, `model_status`, `model_reason` | the model check (open arm) |
| `rfree`, `rwork`, `model_fit`, `rfree_deposited`, `rfree_ratio` | `model` variant: R-free, R-work and MODEL_FIT as rugnux reports them for the deposited model, the published R-free, and rfree / rfree_deposited |
| `refmac_rfree`, `refmac_rwork`, `refmac_rfree_depflags`, `refmac_rfree_depdata`, `refmac_rfree_ratio`, `refmac_status`, `refmac_reason` | the REFMAC check (`--model-check`, open arm, `bare` rows) |
| `wall_s`, `rugnux_wall_s`, `elapsed_s`, `exit_code`, `gpu_others`, `rugnux_verdict` | timing (`wall_s` is rugnux's own WALL_TIME, not the time spent waiting for the GPU queue), the exit code, other GPU processes seen before the set, and rugnux's own verdict line |
## The report
@@ -191,16 +242,19 @@ This is a flat list with one object per set, and every arm uses the same keys. M
Every run writes the same report as Markdown and as a self-contained HTML page (inline CSS and
SVG, no external files):
- a header with the binary, build flags, host, subset, resolution mode and whether timing counts;
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
ISa, R_meas, time) with a verdict bar chart;
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
the main metrics;
- plots per set: d_min(rugnux) / d_min(reference) for each arm whose resolution is not forced, and
R_free / deposited R_free for the open arm (HTML only; hovering a point shows the set);
- the **failures**, and, if there is a baseline, the **delta against it**: pass rates on the
common sets and every set that moved beyond noise;
- **one row per set** with all the numbers.
- a header with the binary, build flags, host, subset, variants and whether timing counts;
- a **timing** table per variant, saying which variant's times include reading the data;
- **a section per variant**, each with:
- a **summary per arm** (pass / fail / unscored / not run, pass rate, median resolution gain,
ISa, R_meas, time) with a verdict bar chart;
- a table **per population** (tags such as `cubic`, `cbf`, `lysozyme`) and the distributions of
the main metrics;
- plots per set (HTML only; hovering a point shows the set): d_min(rugnux) / d_min(reference)
for each arm (not for `xds`, whose resolution is forced), and, for `model`, R_free / published
R_free;
- the **failures**, and **one row per set** with all the numbers;
- if there is a baseline, the **delta against it**: pass rates per arm and variant on the common
rows, and every row that moved beyond noise.
Re-render a report, for example against another baseline (run directories are read-only, so the
output goes elsewhere and is named after the run):
@@ -221,8 +275,9 @@ $B compare RUN_A RUN_B [--all] # per-set table of what move
$B compare RUN_A RUN_B --rerun-changed # rerun the changed sets with A's binary
```
`compare` pairs sets by arm and id, following the manifests' `aliases` across renames, and lists
every set whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
`compare` pairs rows by arm, set id and variant, following the manifests' `aliases` across
renames. Only the variants both runs ran on an arm are compared, so a `--variants bare` run against a full
one compares `bare`. It lists every row whose verdict, space group, lattice, d_min, ISa, R_meas, CC1/2, completeness, cell,
R-free or time moved by more than the noise thresholds in `report.py` (`NOISE`). Those thresholds
are a first guess. `--rerun-changed` measures the noise directly: it reruns the changed sets with
A's saved binary and options. A set that moves again under the same binary is noise. A set that
@@ -240,13 +295,11 @@ reproduces A was really changed by B.
fix. For one fix, run the sets it touches with `--only`, plus the smoke tier.
4. **Timing.** `--gpulock`, or another process seen on the GPU, marks the run "timing NOT a
reference". Only a run with the machine to itself can be used to judge speed.
5. **Resolution on the XDS arms.** By default XDS's resolution range is forced on rugnux, so both
programs merge the same reflections and the statistics are like for like. This answers "is our
processing better than XDS's, reflection for reflection". `--unforced` leaves rugnux its own
cut, which answers "what does a user get". A change can win one and lose the other, so at
important moments run both. Never quote the forced run as what rugnux delivers.
Where XDS merged with `FRIEDEL'S_LAW=FALSE` (`"anomalous": true` in the manifest), rugnux gets
`-A` in both modes, so Friedel mates are merged separately on both sides.
5. **Resolution on the XDS arms.** The `xds` variant forces XDS's resolution range (and `-A`
where XDS kept Friedel mates apart), so both programs merge the same reflections and the
statistics are like for like. This answers "is our processing better than XDS's, reflection for
reflection". The `bare` variant answers "what does a user get". A change can win one and lose
the other, and every run has both. Never quote the `xds` variant as what rugnux delivers.
### The XDS reference d_min
@@ -262,7 +315,7 @@ is significant, and that decides which case applies (`inputs.reference_dmin`):
point where XDS's CC1/2 falls through 0.30 (the target of rugnux's own cutoff), interpolated in
1/d² between shell centres.
The derived value is both the score reference and the forced range. The manifest keeps XDS's own
The derived value is both the score reference and the range the `xds` variant forces. The manifest keeps XDS's own
limit as `dmin_xds` and the rule used as `dmin_rule`. XDS's pooled R_meas, CC1/2 and completeness
still cover its whole range.
+121 -62
View File
@@ -12,7 +12,8 @@ Arms and where their truth comes from:
Usage (README.md in this directory is the full guide):
battery.py run --rugnux BIN [--label L] [--arm open,inhouse|private] [--only a,b | --tier smoke]
[--build-dir DIR] [--gpulock] [--threads N] [--unforced] [--baseline RUN|none]
[--variants bare,model,xds] [--build-dir DIR] [--gpulock] [--threads N]
[--model-check] [--baseline RUN|none]
battery.py list runs under the runs root, with their state
battery.py compare RUN_A RUN_B [--all] [--rerun-changed]
battery.py report RUN --out DIR [--baseline RUN] re-render a run's report into DIR/<run>.{md,html}
@@ -24,9 +25,13 @@ Usage (README.md in this directory is the full guide):
The site config (data roots, runs root, lock, gpulock, private manifest, baselines) is --site, else
$JFJOCH_BATTERY_SITE, else site.json beside this script (not committed; site.example.json is).
Every set runs in each of its arm's VARIANTS (bare + model on the open arm, bare + xds on the XDS
arms), one straight after the other, so only the first reads the images from disk.
A run writes <runs_root>/<YYYYMMDD-HHMM>_<rugnux git>_<label>/ holding a copy of the binary,
manifest.json (what was run, on what, with which references), results.json (one row per set),
report.md + report.html and work/<arm>/<set>/ (log, report, cif). It is made read-only when done.
manifest.json (what was run, on what, with which references), results.json (one row per set and
variant), report.md + report.html and work/<arm>/<set>/<variant>/ (log, report, cif). It is made
read-only when done.
A full-arm run (no --only / --tier) takes the site lock and refuses to start if another holds it.
A run that did not finish is never compared or reported on unless asked (--allow-incomplete).
"""
@@ -52,13 +57,24 @@ import inputs # noqa: E402
import report # noqa: E402
import score # noqa: E402
try:
import model_check # optional: R-free of the deposited model against our merge (open arm)
except ImportError:
model_check = None
import model_check # noqa: E402 the deposited model (model variant); REFMAC check (--model-check)
PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$")
# The variants every set of an arm runs in, by the arm's reference, in the order they run. All the
# variants of one set run back to back, so only the first reads the images from disk and the later
# ones find them in the page cache. `bare` goes first: it is in every arm, so its time is taken
# under the same condition (the set's first read) whichever variants a run selects, and it is the
# number a user sees.
VARIANTS = {"deposition": ["bare", "model"], "xds": ["bare", "xds"]}
VARIANT_DOC = {
"bare": "plain `rugnux <input>`, nothing else: what a user gets",
"model": "`rugnux --model <deposited coordinates>`: R-free of the deposited model against our merge, "
"as rugnux reports it",
"xds": "XDS's reference settings: its resolution range forced, and -A where XDS kept Friedel mates "
"apart, so both programs merge the same reflections the same way",
}
LOCAL_SITE = os.path.join(HERE, "site.json")
@@ -155,22 +171,23 @@ def gpu_others():
# ------------------------------------------------------------------ running
def command(binary, e, opts):
def command(binary, e, variant, opts, model=None):
cmd = [binary, "-o", "p", "--no-export-unmerged"]
if opts["threads"]:
cmd += ["-N", str(opts["threads"])]
ref = e.get("ref") or {}
# On an XDS-referenced arm the only thing fixed is XDS's resolution range, so the two programs
# merge the same reflections. --unforced leaves rugnux its own cut.
if opts["xds_range"] and e.get("reference") == "xds":
if variant == "xds":
# XDS's resolution range, so the two programs merge the same reflections
if ref.get("dmin"):
cmd += ["--scaling-high-resolution", f"{ref['dmin']:.3f}"]
if ref.get("dmax"):
cmd += ["--scaling-low-resolution", f"{ref['dmax']:.3f}"]
# XDS merged with FRIEDEL'S_LAW=FALSE: merge Friedel mates separately too, or the merging
# statistics are not comparable
if e.get("reference") == "xds" and ref.get("anomalous"):
cmd.append("-A")
# XDS merged with FRIEDEL'S_LAW=FALSE: merge Friedel mates separately too, or the merging
# statistics are not comparable
if ref.get("anomalous"):
cmd.append("-A")
if variant == "model":
cmd += ["--model", model]
cmd += opts["extra_args"]
cmd.append(e["input_path"])
if opts["gpulock"]:
@@ -190,16 +207,37 @@ def failure_note(log):
return lines[-1].split("] ")[-1][:300] if lines else ""
def run_one(run_dir, binary, e, opts):
wd = os.path.join(run_dir, "work", e["arm"], e["id"])
def deposited_model(e, opts):
"""(deposition record from model_check's RCSB cache, None) for the entry a set belongs to, or
(None, why not). A set id may carry a suffix (6h2p_native); the entry is the part before it."""
pdb = e["id"].split("_")[0].lower()
if not PDB_ID.match(pdb):
return None, "no model: not a PDB entry (small molecule or unpublished set)"
meta = model_check.deposition(pdb, opts["pdb_cache"])
if meta is None:
return None, f"no model: coordinates of {pdb} could not be downloaded"
return meta, None
def run_one(run_dir, binary, e, variant, first_read, opts):
wd = os.path.join(run_dir, "work", e["arm"], e["id"], variant)
os.makedirs(wd)
row = {"set": e["id"], "arm": e["arm"], "tags": e.get("tags", []), "input": e["input_path"],
"gpu_others": gpu_others(), "rfree": None, "rwork": None, "rfree_deposited": None,
"rfree_depflags": None, "rfree_depdata": None, "rfree_ratio": None, "model_status": None, "model_reason": None}
row = {"set": e["id"], "arm": e["arm"], "variant": variant, "first_read": first_read,
"tags": e.get("tags", []), "input": e["input_path"], "gpu_others": gpu_others(),
"rfree_deposited": None, "rfree_ratio": None, "cmd": None, "exit_code": None,
"elapsed_s": None, "wall_s": None}
row.update({k: None for k in REFMAC_KEYS})
if not os.path.exists(e["input_path"]):
row.update(score.judge(e, {}, "no input"), exit_code=None, elapsed_s=None, wall_s=None, cmd=None)
row.update(score.judge(e, {}, "no input"))
return row
cmd = command(binary, e, opts)
meta = None
if variant == "model":
meta, why = deposited_model(e, opts)
if meta is None:
row.update(score.judge(e, {}, why))
return row
row["rfree_deposited"] = meta["rfree"]
cmd = command(binary, e, variant, opts, meta and meta["xyz"])
row["cmd"] = " ".join(cmd)
t0 = time.time()
note = ""
@@ -217,34 +255,38 @@ def run_one(run_dir, binary, e, opts):
# rugnux's own WALL_TIME where it wrote one: the runner's clock also counts the wait for a
# GPU slot under --gpulock
row["wall_s"] = row["rugnux_wall_s"] if row["rugnux_wall_s"] is not None else row["elapsed_s"]
if opts.get("model_check") and e["arm"] == "open":
if row["rfree"] and row["rfree_deposited"]:
row["rfree_ratio"] = round(row["rfree"] / row["rfree_deposited"], 4)
if opts["model_check"] and e["arm"] == "open" and variant == "bare":
row.update(check_model(e, wd))
return row
# what the REFMAC model check (model_check.py, --model-check) adds to an open-arm bare row
REFMAC_KEYS = ("refmac_rfree", "refmac_rwork", "refmac_rfree_depflags", "refmac_rfree_depdata",
"refmac_rfree_ratio", "refmac_status", "refmac_reason")
def check_model(e, wd):
"""R-free of the deposited model against this run's merge (model_check.py). A set id may
carry a suffix (6h2p_native); the PDB entry is the part before it."""
out = {"rfree": None, "rwork": None, "rfree_deposited": None, "rfree_depflags": None,
"rfree_depdata": None, "rfree_ratio": None}
"""R-free of the deposited model against this run's merge, by REFMAC (model_check.py)."""
out = {k: None for k in REFMAC_KEYS}
pdb = e["id"].split("_")[0]
mtz = os.path.join(wd, "p.mtz")
if not PDB_ID.match(pdb):
return dict(out, model_status="skipped", model_reason="not a PDB entry")
return dict(out, refmac_status="skipped", refmac_reason="not a PDB entry")
if not os.path.exists(mtz):
return dict(out, model_status="skipped", model_reason="no merged MTZ")
return dict(out, refmac_status="skipped", refmac_reason="no merged MTZ")
try:
res = model_check.check(mtz, pdb, os.path.join(wd, "model_check"))
except Exception as ex: # a failing check must not lose the set's processing result
return dict(out, model_status="error", model_reason=f"{type(ex).__name__}: {ex}"[:300])
out.update({k: res.get(k) for k in ("rfree", "rwork", "rfree_deposited", "rfree_depflags",
"rfree_depdata")},
model_status=res.get("status"), model_reason=res.get("reason"))
return dict(out, refmac_status="error", refmac_reason=f"{type(ex).__name__}: {ex}"[:300])
out.update({"refmac_" + k: res.get(k) for k in ("rfree", "rwork", "rfree_depflags", "rfree_depdata")},
refmac_status=res.get("status"), refmac_reason=res.get("reason"))
# The fair ratio: our merge and the depositor's structure factors, both scored on the
# depositor's free set by the same protocol. Our own free set was mostly work reflections
# for the depositor, so R_free on it reads low.
if out["rfree_depflags"] and out["rfree_depdata"]:
out["rfree_ratio"] = round(out["rfree_depflags"] / out["rfree_depdata"], 4)
if out["refmac_rfree_depflags"] and out["refmac_rfree_depdata"]:
out["refmac_rfree_ratio"] = round(out["refmac_rfree_depflags"] / out["refmac_rfree_depdata"], 4)
return out
@@ -314,7 +356,10 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
for e in sets:
e["reference"] = site["arms"][e["arm"]]["reference"]
e.setdefault("input_path", os.path.join(site["arms"][e["arm"]]["data_root"], e["input"]))
variants = {arm: [v for v in VARIANTS[site["arms"][arm]["reference"]] if v in opts["variants"]]
for arm in arms}
man = {"label": label, "arms": arms, "private": private, "subset": subset, "baseline": baseline,
"variants": variants, "variant_doc": {v: VARIANT_DOC[v] for v in opts["variants"]},
"binary": info,
"runner_git": git(HERE, "rev-parse", "--short", "HEAD") or "unknown",
"runner_dirty": bool(git(HERE, "status", "--porcelain", "--", ".")),
@@ -325,20 +370,25 @@ def run_sets(site, sets, arms, binary_src, build_dir, label, opts, subset, basel
print(f"run directory {run_dir}", flush=True)
results = []
todo = sum(len(variants[e["arm"]]) for e in sets)
# a kill by SIGTERM is recorded like Ctrl-C: the run directory then says it was aborted
signal.signal(signal.SIGTERM, interrupted)
try:
for n, e in enumerate(sets, 1):
row = run_one(run_dir, binary, e, opts)
results.append(row)
if row.get("gpu_others"):
man["gpu_shared"] = True
write_json(os.path.join(run_dir, "results.json"), results)
print(f"[{n}/{len(sets)}] {e['arm']:8s} {e['id']:24s} {row['verdict']:8s} "
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
for e in sets:
# the variants of one set back to back: only the first rugnux reads the images from disk
first_read = True
for v in variants[e["arm"]]:
row = run_one(run_dir, binary, e, v, first_read, opts)
first_read = first_read and row["cmd"] is None
results.append(row)
if row.get("gpu_others"):
man["gpu_shared"] = True
write_json(os.path.join(run_dir, "results.json"), results)
print(f"[{len(results)}/{todo}] {e['arm']:8s} {e['id']:24s} {v:6s} {row['verdict']:8s} "
f"{row.get('sg') or '-':12s} d={report.f(row.get('d_min'))} "
f"t={report.f(row.get('wall_s'), '{:.0f}')}s {row.get('reason') or ''}", flush=True)
except KeyboardInterrupt:
mark_aborted(run_dir, f"interrupted after {len(results)} of {len(sets)} sets")
mark_aborted(run_dir, f"interrupted after {len(results)} of {todo} rows")
sys.exit(f"aborted; {run_dir} is marked as such")
man.update(end=datetime.datetime.now().isoformat(timespec="seconds"), complete=True)
@@ -380,11 +430,16 @@ def cmd_run(a, site):
if not sets:
sys.exit("nothing selected")
subset = a.only or (f"tier {a.tier}" if a.tier else None)
if a.model_check and not model_check:
sys.exit("--model-check: tools/battery/model_check.py is not there")
use_model = a.model_check if a.model_check is not None else bool(model_check)
opts = {"threads": a.threads, "xds_range": not a.unforced, "timeout": a.timeout,
"model_check": use_model and "open" in arms,
known = [v for vs in VARIANTS.values() for v in vs]
want = [v.strip() for v in a.variants.split(",")] if a.variants else known
if set(want) - set(known):
sys.exit(f"unknown variant(s) {', '.join(sorted(set(want) - set(known)))}; "
f"there are {', '.join(dict.fromkeys(known))}")
if not any(v in want for arm in arms for v in VARIANTS[site["arms"][arm]["reference"]]):
sys.exit(f"none of the variants {a.variants} runs on the {'/'.join(arms)} arm(s)")
opts = {"threads": a.threads, "variants": [v for v in dict.fromkeys(known) if v in want],
"timeout": a.timeout, "model_check": a.model_check and "open" in arms,
"pdb_cache": site.get("pdb_cache") or model_check.DEFAULT_CACHE,
"gpulock": site["gpulock"] if a.gpulock else None, "extra_args": a.extra.split(),
"runs_root": a.runs_root or site["runs_root"]}
if a.gpulock and not site.get("gpulock"):
@@ -404,7 +459,7 @@ def cmd_list(a, site):
n = len(json.load(open(os.path.join(run, "results.json")))) \
if os.path.exists(os.path.join(run, "results.json")) else 0
mark = " <- baseline" if run in (site.get("baseline"), site.get("baseline_private")) else ""
print(f"{d:48s} {state:10s} {n:3d}/{len(man['sets']):3d} sets "
print(f"{d:48s} {state:10s} {n:3d}/{report.expected_rows(man):3d} rows "
f"{man.get('subset') or 'full arm':20.20s} {','.join(man['arms'])}{mark}")
@@ -426,7 +481,9 @@ def cmd_compare(a, site):
f"{os.path.basename(os.path.normpath(a.run_b))}")
doc.p(f"A: rugnux {man_a['binary'].get('version')} ({man_a['binary']['sha256'][:12]}); "
f"B: rugnux {man_b['binary'].get('version')} ({man_b['binary']['sha256'][:12]}). "
f"{len(rows)} sets, {sum(1 for r in rows if r[3])} changed beyond noise.")
f"{len(rows)} set/variant rows, {sum(1 for r in rows if r[3])} changed beyond noise. "
f"Variants compared: {report.pairs_text(report.common_variants(res_a, res_b))} "
"(the ones both runs have).")
report.compare_table(doc, rows, only_changed=not a.all)
print(doc.markdown())
if not a.rerun_changed:
@@ -437,9 +494,11 @@ def cmd_compare(a, site):
if not changed:
print("nothing changed beyond noise; no rerun")
return
sets = [e for e in man_a["sets"] if (e["arm"], e["id"]) in changed]
opts = dict(man_a["options"], runs_root=a.runs_root or site["runs_root"])
sets = [e for e in man_a["sets"] if (e["arm"], e["id"]) in {k[:2] for k in changed}]
opts = dict(man_a["options"], runs_root=a.runs_root or site["runs_root"],
variants=[v for v in VARIANT_DOC if v in {k[2] for k in changed}])
opts["gpulock"] = opts["gpulock"] and site.get("gpulock")
opts.setdefault("pdb_cache", site.get("pdb_cache") or model_check.DEFAULT_CACHE)
label = "rerunA-" + man_a["label"]
rerun = run_sets(site, sets, man_a["arms"], os.path.join(a.run_a, "bin", "rugnux"), None, label,
opts, ",".join(e["id"] for e in sets))
@@ -451,10 +510,10 @@ def cmd_compare(a, site):
if k not in changed:
continue
noise = [c for c in again.get(k, []) if c != "time"]
rows_out.append([k[1], k[0], ", ".join(c for c in ch if c != "time"),
rows_out.append([k[1], k[0], k[2], ", ".join(c for c in ch if c != "time"),
", ".join(noise) or "-",
"NOISE - A itself moved" if noise else "REAL - A reproduces"])
doc.table(["set", "arm", "A -> B", "A -> A again", "reading"], rows_out)
doc.table(["set", "arm", "variant", "A -> B", "A -> A again", "reading"], rows_out)
doc.p(f"rerun directory: {rerun}")
print(doc.markdown())
@@ -572,12 +631,12 @@ def main():
r.add_argument("--gpulock", action="store_true",
help="queue each rugnux behind the site's GPU lock (timing is then not a reference)")
r.add_argument("--threads", type=int, default=0, help="-N for rugnux (default: its own choice)")
r.add_argument("--unforced", action="store_true",
help="do not force XDS's resolution range on the XDS-referenced arms")
r.add_argument("--timeout", type=int, default=10800)
r.add_argument("--model-check", action=argparse.BooleanOptionalAction, default=None,
help="open arm: R-free of the deposited model against each merge "
"(default: on when model_check.py is present)")
r.add_argument("--variants", help="comma-separated subset of bare,model,xds (default: all of "
"each arm's variants; e.g. 'bare' for what a user gets)")
r.add_argument("--timeout", type=int, default=10800, help="per rugnux run, in s")
r.add_argument("--model-check", action="store_true",
help="open arm: also score each bare merge against the deposited model with "
"REFMAC (model_check.py; needs CCP4)")
r.add_argument("--extra", default="", help="extra rugnux arguments, applied to every set")
r.add_argument("--baseline", help="earlier run directory to report the delta against (default: "
"the site's persisted baseline; 'none' for no delta)")
-262
View File
@@ -1,262 +0,0 @@
#!/usr/bin/env python3
"""
model_sweep.py -- process a set of crystals whose data are deposited in the PDB, validate
each merge against the depositor's own model, and compare the result with what they published.
It runs the open arm of the battery (the same inputs, from open.json and the site config), but
separately from battery.py: --model places the deposited coordinates, which needs network access
to the RCSB and adds R-free, maps and model placement to every run.
For every dataset given it:
* looks the entry up in the RCSB data API and records the deposited high-resolution limit,
R-work and R-free;
* downloads the deposited coordinates (gzipped mmCIF -- rugnux recognises the format by
content, so no conversion);
* runs rugnux de novo with --model and NOTHING else: no space group, no cell, no resolution
limit. That is the point. The question is what a bare invocation gets to, so anything told
to the program is a thumb on the scale;
* reads R_FREE / R_WORK and the resolution range back out of the results report, which is a
stable interface, and not out of the console log, which is not;
* tabulates deposited against measured, and plots the two scatters that answer the question:
resolution, and R-free.
R-free here is not a refinement result. rugnux places the deposited model rigid-body and reports
the free residual of THAT placement against its own merge, so it is a validation number: how well
the depositor's model explains data this pipeline produced from their images. It is compared with
the depositor's published R-free, which came from full refinement against their own reduction --
a harder target, and deliberately so.
Datasets are the open-arm sets whose id is a four-character PDB entry ID (optionally with a
suffix, as in 6h2p_native); small-molecule sets and anything without a deposited model are
skipped with a reason.
tools/battery/model_sweep.py --rugnux build/rugnux/rugnux --work /tmp/modelrun [--only 5reo,9jzo]
tools/battery/model_sweep.py --work /tmp/modelrun --score-only # re-score without re-running
"""
import argparse
import json
import os
import re
import subprocess
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import battery # noqa: E402
RCSB_ENTRY = "https://data.rcsb.org/rest/v1/core/entry/{}"
RCSB_MODEL = "https://files.rcsb.org/download/{}.cif.gz"
PDB_ID = re.compile(r"^[0-9][A-Za-z0-9]{3}$")
def curl(url, dest=None, timeout=180):
cmd = ["curl", "-sfL", "--max-time", str(timeout), url]
if dest:
cmd += ["-o", dest]
r = subprocess.run(cmd, capture_output=True)
return r.returncode == 0, r.stdout
def deposited(pdb, models_dir):
"""Published resolution and R-factors, plus the coordinate file, for one entry."""
rec = {"id": pdb}
ok, body = curl(RCSB_ENTRY.format(pdb))
if ok:
try:
d = json.loads(body)
# The refine category is a list; a multi-stage refinement has more than one block and
# the last is the one the depositor stopped at.
refine = d.get("refine") or []
r = refine[-1] if refine else {}
rec["dep_dmin"] = r.get("ls_d_res_high")
rec["dep_rfree"] = r.get("ls_R_factor_R_free")
rec["dep_rwork"] = r.get("ls_R_factor_R_work")
rec["method"] = (d.get("exptl") or [{}])[0].get("method")
except Exception as e:
rec["meta_error"] = str(e)
else:
rec["meta_error"] = "entry not found"
dest = os.path.join(models_dir, f"{pdb}.cif.gz")
if not os.path.exists(dest) or os.path.getsize(dest) == 0:
ok, _ = curl(RCSB_MODEL.format(pdb), dest)
if not ok and os.path.exists(dest):
os.remove(dest)
if os.path.exists(dest) and os.path.getsize(dest) > 0:
rec["model"] = dest
else:
rec["model_error"] = "no deposited coordinates"
return rec
def parse_report(path):
rec = {}
with open(path, errors="replace") as fh:
for line in fh:
if "=" in line:
k, _, v = line.partition("=")
rec[k.strip()] = v.strip()
return rec
def num(rec, key):
try:
return float(rec[key])
except (KeyError, TypeError, ValueError):
return None
def run_one(rugnux, model, image, outdir, threads, gpulock):
os.makedirs(outdir, exist_ok=True)
cmd = [gpulock] if gpulock else []
# threads=0 -> no -N: rugnux picks its own thread count
cmd += [rugnux, "-o", "p"] + (["-N", str(threads)] if threads else []) + ["--no-export-unmerged",
"--spot-low-resolution", "50", "--scaling-low-resolution", "50",
"--model", model, image]
with open(os.path.join(outdir, "run.log"), "w") as log:
return subprocess.run(cmd, cwd=outdir, stdout=log, stderr=subprocess.STDOUT).returncode
def collect(work, meta):
rows = []
for pdb in sorted(meta):
report = os.path.join(work, "runs", pdb, "p_report.txt")
if not os.path.exists(report):
continue
rep = parse_report(report)
rng = rep.get("INCLUDE_RESOLUTION_RANGE", "").split()
rows.append(dict(
id=pdb,
dep_dmin=meta[pdb].get("dep_dmin"),
dep_rfree=meta[pdb].get("dep_rfree"),
dep_rwork=meta[pdb].get("dep_rwork"),
our_dmin=float(rng[1]) if len(rng) == 2 else None,
our_rfree=num(rep, "R_FREE"),
our_rwork=num(rep, "R_WORK"),
model_fit=rep.get("MODEL_FIT"),
space_group=rep.get("SPACE_GROUP_NAME"),
cc_half=num(rep, "CC_HALF"),
))
return rows
def scatter(pts, xlabel, ylabel, title, path, floor_at_zero=False):
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
except ImportError:
print(f" (matplotlib absent, skipping {os.path.basename(path)})")
return
if not pts:
return
xs, ys = [p[0] for p in pts], [p[1] for p in pts]
lo = 0.0 if floor_at_zero else min(min(xs), min(ys)) * 0.95
hi = max(max(xs), max(ys)) * 1.05
fig, ax = plt.subplots(figsize=(6, 6))
ax.plot([lo, hi], [lo, hi], color="0.6", lw=1, zorder=1)
ax.scatter(xs, ys, s=26, alpha=0.8, edgecolor="none", zorder=2)
ax.set_xlim(lo, hi)
ax.set_ylim(lo, hi)
ax.set_aspect("equal")
ax.set_xlabel(xlabel)
ax.set_ylabel(ylabel)
ax.set_title(title)
ax.grid(alpha=0.25, lw=0.5)
fig.tight_layout()
fig.savefig(path, dpi=140)
plt.close(fig)
print(f" wrote {path}")
def report(work, rows):
both = [r for r in rows if r["our_rfree"] is not None and r["dep_rfree"] is not None]
res = [r for r in rows if r["our_dmin"] is not None and r["dep_dmin"] is not None]
print(f"\n{len(rows)} processed; R-free on both sides for {len(both)}; resolution for {len(res)}\n")
if both:
n = len(both)
below = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"])
w02 = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"] + 0.02)
w05 = sum(1 for r in both if r["our_rfree"] <= r["dep_rfree"] + 0.05)
d = sorted(r["our_rfree"] - r["dep_rfree"] for r in both)
print(f" R-free at or below deposited {below:3d}/{n} ({100 * below / n:.0f}%)")
print(f" within +0.02 {w02:3d}/{n} ({100 * w02 / n:.0f}%)")
print(f" within +0.05 {w05:3d}/{n} ({100 * w05 / n:.0f}%)")
print(f" median delta R-free {d[len(d) // 2]:+.4f} ({d[0]:+.4f} .. {d[-1]:+.4f})")
if res:
deeper = sum(1 for r in res if r["our_dmin"] < r["dep_dmin"])
d = sorted(r["our_dmin"] - r["dep_dmin"] for r in res)
print(f" d_min deeper than deposited {deeper:3d}/{len(res)} ({100 * deeper / len(res):.0f}%)")
print(f" median delta d_min {d[len(d) // 2]:+.3f} A ({d[0]:+.3f} .. {d[-1]:+.3f})")
print()
scatter([(r["dep_dmin"], r["our_dmin"]) for r in res],
"deposited $d_{min}$ (A)", "rugnux $d_{min}$ (A)",
f"Resolution, n={len(res)} (below the line: rugnux cuts deeper)",
os.path.join(work, "dmin.png"))
scatter([(r["dep_rfree"], r["our_rfree"]) for r in both],
"deposited $R_{free}$", "rugnux $R_{free}$",
f"R-free against the deposited model, n={len(both)} (below the line: better)",
os.path.join(work, "rfree.png"), floor_at_zero=True)
no_rfree = [r for r in rows if r["our_rfree"] is None]
if no_rfree:
print(f"\nran but reported no R-free ({len(no_rfree)}):")
for r in no_rfree:
print(f" {r['id']:<8}MODEL_FIT={r['model_fit']}")
def main():
ap = argparse.ArgumentParser(
description="Process deposited datasets with --model and compare with the deposition.")
ap.add_argument("--work", required=True, help="working directory (models, runs, results)")
ap.add_argument("--rugnux", help="rugnux binary (required unless --score-only)")
ap.add_argument("--site", help="battery site config (default as for battery.py)")
ap.add_argument("--only", help="comma-separated open-arm set ids")
ap.add_argument("--datasets", help="JSON file mapping entry ID to image file, instead of the "
"open arm")
ap.add_argument("--threads", type=int, default=0)
ap.add_argument("--gpulock", action="store_true",
help="queue each rugnux behind the site's GPU lock")
ap.add_argument("--score-only", action="store_true", help="re-score existing runs, run nothing")
args = ap.parse_args()
work = args.work
os.makedirs(os.path.join(work, "models"), exist_ok=True)
os.makedirs(os.path.join(work, "runs"), exist_ok=True)
meta_path = os.path.join(work, "deposited.json")
meta = json.load(open(meta_path)) if os.path.exists(meta_path) else {}
if not args.score_only:
if not args.rugnux:
ap.error("--rugnux is required unless --score-only")
site = battery.load_site(args.site)
gpulock = site.get("gpulock") if args.gpulock else None
if args.datasets:
inputs = json.load(open(args.datasets))
else:
root = site["arms"]["open"]["data_root"]
inputs = {e["id"].lower(): os.path.join(root, e["input"])
for e in battery.select(site, ["open"], args.only, None) if PDB_ID.match(e["id"].split("_")[0])}
print(f"{len(inputs)} datasets with a PDB-shaped name and an image file")
for i, (pdb, image) in enumerate(sorted(inputs.items()), 1):
if pdb not in meta or "model" not in meta[pdb]:
# a set id may carry a suffix (6h2p_native): the entry is the part before it
meta[pdb] = deposited(pdb.split("_")[0], os.path.join(work, "models"))
json.dump(meta, open(meta_path, "w"), indent=1)
if "model" not in meta[pdb]:
print(f"[{i}/{len(inputs)}] {pdb}: {meta[pdb].get('model_error', 'no model')}")
continue
outdir = os.path.join(work, "runs", pdb)
if os.path.exists(os.path.join(outdir, "p_report.txt")):
continue # resumable: an interrupted sweep picks up here
print(f"[{i}/{len(inputs)}] {pdb}", flush=True)
rc = run_one(args.rugnux, meta[pdb]["model"], image, outdir, args.threads, gpulock)
if rc != 0:
print(f" exited {rc}")
rows = collect(work, meta)
json.dump(rows, open(os.path.join(work, "results.json"), "w"), indent=1)
report(work, rows)
if __name__ == "__main__":
main()
+161 -77
View File
@@ -46,9 +46,33 @@ def load_run(run_dir, allow_incomplete=False):
sys.exit(f"{run_dir} is {state}, not a finished run; pass --allow-incomplete to use it anyway")
path = os.path.join(run_dir, "results.json")
res = json.load(open(path)) if os.path.exists(path) else []
if man.get("results_schema", 1) < 2:
schema1(man, res)
return man, res
def schema1(man, res):
"""Read a run from before the variants (results schema 1) as one: it ran each set once, plain
on the open arm and with XDS's settings on the XDS arms - unless --unforced, which left the
range free but still passed -A, and is read as `bare`. Its R-factors came from the REFMAC check."""
for r in res:
r.setdefault("variant", "xds" if r["arm"] != "open" and man["options"].get("xds_range") else "bare")
r.setdefault("first_read", True)
if "model_status" in r:
for k in ("rfree", "rwork", "rfree_depflags", "rfree_depdata", "rfree_ratio"):
r["refmac_" + k] = r.pop(k, None)
r["refmac_status"], r["refmac_reason"] = r.pop("model_status"), r.pop("model_reason", None)
r["rfree_deposited"] = None
man.setdefault("variants", {arm: sorted({r["variant"] for r in res if r["arm"] == arm})
for arm in man["arms"]})
man.setdefault("variant_doc", {})
def expected_rows(man):
"""Rows a finished run has: one per set and variant."""
return sum(len(man.get("variants", {}).get(e["arm"], [None])) for e in man["sets"])
def check_baseline(baseline, private):
"""A baseline must have finished, and be as private as the run it is compared with: the delta
table names its sets."""
@@ -60,7 +84,9 @@ def check_baseline(baseline, private):
# Version of the results.json row schema (README.md, "results.json"). Bump it when a field changes
# meaning or is removed; adding a field does not need it.
RESULTS_SCHEMA = 1
# 2: one row per set AND variant (`variant`, `first_read`); `rfree`/`rwork`/`rfree_ratio` are rugnux's
# own --model R-factors, the REFMAC check's moved to `refmac_*`.
RESULTS_SCHEMA = 2
# {arm: {old set id: current id}} from the manifests (battery.py sets it), so a run made before a
# dataset was renamed still pairs with one made after
@@ -68,7 +94,25 @@ ALIASES = {}
def key(r):
return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"]))
return (r["arm"], ALIASES.get(r["arm"], {}).get(r["set"], r["set"]), r["variant"])
# the order variants are listed in (battery.VARIANTS runs them in this order within an arm)
VARIANT_ORDER = ("bare", "model", "xds")
def variants_of(res):
return [v for v in VARIANT_ORDER if any(r["variant"] == v for r in res)]
def common_variants(res_a, res_b):
"""The (arm, variant) pairs both runs ran - what a comparison of the two covers."""
pairs = lambda res: {(r["arm"], r["variant"]) for r in res}
return sorted(pairs(res_a) & pairs(res_b), key=lambda p: (p[0], VARIANT_ORDER.index(p[1])))
def pairs_text(pairs):
return ", ".join(f"{arm} {v}" for arm, v in pairs) or "none"
def f(x, fmt="{:.2f}", dash="-"):
@@ -123,9 +167,12 @@ def changes(ra, rb):
def compare_rows(res_a, res_b):
"""[(key, ra, rb, changes)] over the sets both runs have, then those only one has."""
a = {key(r): r for r in res_a}
b = {key(r): r for r in res_b}
"""[(key, ra, rb, changes)] over the (set, variant) rows both runs have, then those only one
has. Only the variants both runs ran on an arm are compared: a run of just `bare` against a
full one is a comparison of `bare`."""
common = common_variants(res_a, res_b)
a = {key(r): r for r in res_a if (r["arm"], r["variant"]) in common}
b = {key(r): r for r in res_b if (r["arm"], r["variant"]) in common}
rows = [(k, a[k], b[k], changes(a[k], b[k])) for k in sorted(a) if k in b]
rows += [(k, a[k], None, ["only in A"]) for k in sorted(a) if k not in b]
rows += [(k, None, b[k], ["only in B"]) for k in sorted(b) if k not in a]
@@ -139,14 +186,14 @@ def arrow(x, y, fmt=lambda v: str(v)):
def compare_table(doc, rows, only_changed=True):
head = ["set", "arm", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
head = ["set", "arm", "variant", "verdict", "space group", "d_min", "ISa", "R_meas", "CC1/2",
"cell dev %", "R_free", "time s", "beyond noise"]
out = []
for (arm, s), ra, rb, ch in rows:
for (arm, s, v), ra, rb, ch in rows:
if only_changed and not ch:
continue
ra, rb = ra or {}, rb or {}
out.append([s, arm, arrow(ra.get("verdict"), rb.get("verdict")),
out.append([s, arm, v, arrow(ra.get("verdict"), rb.get("verdict")),
arrow(ra.get("sg"), rb.get("sg")),
arrow(ra.get("d_min"), rb.get("d_min"), lambda v: f"{v:.2f}"),
arrow(ra.get("isa"), rb.get("isa"), lambda v: f"{v:.1f}"),
@@ -187,67 +234,118 @@ def build(run_dir, baseline=None, allow_incomplete=False):
state = run_state(man)
if state != "complete":
why = f" ({man['aborted']['reason']})" if man.get("aborted") else ""
doc.p(f"RUN {state.upper()}{why}: {len(res)} of {len(man['sets'])} sets have a result. "
"Not a reference and not comparable with a finished run.")
doc.p(f"RUN {state.upper()}{why}: {len(res)} of {expected_rows(man)} set/variant rows have a "
"result. Not a reference and not comparable with a finished run.")
variants = variants_of(res)
doc.ul([
f"rugnux {b.get('version', '?')}, sha256 {b['sha256'][:16]}, build flags: {b.get('flags') or 'unknown'}",
f"runner {man['runner_git']}, host {man['host']}, {man['start']} -> {man.get('end') or 'NOT FINISHED'}",
f"arms: {', '.join(man['arms'])}; {len(man['sets'])} sets"
+ (f" (subset: {man['subset']})" if man.get("subset") else " (full arm)"),
"resolution: XDS's range forced on the XDS-referenced arms" if man["options"]["xds_range"]
else "resolution: rugnux's own choice everywhere (unforced)",
"variants: " + "; ".join(f"{arm}: {', '.join(man['variants'][arm])}" for arm in man["arms"]
if man["variants"].get(arm)),
"timing: NOT a reference - the GPU was shared" if man.get("gpu_shared")
else "timing: GPU was not shared during the run",
f"command options: {' '.join(man['options']['extra_args']) or '(none beyond output selection)'}",
])
] + [f"variant {v}: {man['variant_doc'][v]}" for v in variants if man["variant_doc"].get(v)])
forced = {e["arm"] for e in man["sets"]
if man["options"]["xds_range"] and e.get("reference") == "xds"}
order = {arm: i for i, arm in enumerate(man["arms"])}
doc.h(2, "Timing")
rows = []
for v in variants:
rs = [r for r in res if r["variant"] == v and r.get("wall_s") is not None]
rows.append([v, len(rs), f"{sum(bool(r.get('first_read')) for r in rs)}/{len(rs)}",
f(median(r["wall_s"] for r in rs), "{:.0f}"),
f(sum(r["wall_s"] for r in rs) / 60, "{:.1f}")])
doc.table(["variant", "rows run", "first read of the set", "median time s", "total time min"],
rows, num=range(1, 5))
doc.p("The variants of a set run back to back, in the order listed. The first one to run reads "
"the images from disk (unless they were still in the page cache from before the run); the "
"later ones find them in the page cache. So only the first variant's times include reading "
"the data - normally `bare`, which runs first on every arm - and times are comparable only "
"between runs of the same variant.")
def gain(rs, arm):
return "forced" if arm in forced else f(median(r.get("res_gain_pct") for r in rs), "{:+.1f}%")
derived = {r["set"]: r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"}
if derived:
doc.p("XDS merged past its own signal (CC1/2 of its finest shell not significant), so the "
"reference d_min is where XDS's CC1/2 falls through 0.30, and that is also the range "
"the xds variant forces on rugnux: "
+ "; ".join(f"{s} (XDS range {f(r.get('d_min_xds'))} A, reference "
f"{f(r.get('d_min_ref'))} A)" for s, r in sorted(derived.items()))
+ ". XDS's pooled R_meas, CC1/2 and completeness are over its whole range.")
doc.h(2, "Summary")
for v in variants:
variant_section(doc, man, [r for r in res if r["variant"] == v], v)
if baseline:
bman, bres = load_run(baseline)
doc.h(2, f"Delta vs baseline {os.path.basename(os.path.normpath(baseline))}")
common = common_variants(bres, res)
doc.p(f"Baseline rugnux {bman['binary'].get('version', '?')}. Pass rates on the set/variant "
f"rows both runs have (variants compared: {pairs_text(common)}):")
both = {key(r) for r in res} & {key(r) for r in bres}
rows = []
for arm, v in common:
ra = [r for r in bres if key(r) in both and r["arm"] == arm and r["variant"] == v]
rb = [r for r in res if key(r) in both and r["arm"] == arm and r["variant"] == v]
rows.append([arm, v, len(rb), rate(ra), rate(rb)])
doc.table(["arm", "variant", "common sets", "baseline", "this run"], rows, num=(2,))
doc.p("Rows whose result moved beyond noise (thresholds in report.py NOISE; confirm with "
"`compare --rerun-changed` before believing a single-set change):")
compare_table(doc, compare_rows(bres, res))
return doc
def variant_section(doc, man, res, v):
"""Summary, populations, distributions, plots, failures and the per-set table of one variant."""
forced = v == "xds" # XDS's resolution range imposed: d_min ratio 1 by construction
arms = [arm for arm in man["arms"] if any(r["arm"] == arm for r in res)]
order = {arm: i for i, arm in enumerate(arms)}
def gain(rs):
return "forced" if forced else f(median(r.get("res_gain_pct") for r in rs), "{:+.1f}%")
doc.h(2, f"Variant {v}")
if man["variant_doc"].get(v):
doc.p(man["variant_doc"][v][0].upper() + man["variant_doc"][v][1:] + ".")
head = ["arm", "sets", "pass", "fail", "unscored", "not run", "pass rate",
"median res. gain", "median ISa", "median R_meas", "median time s", "total time min"]
rows = []
for arm in man["arms"]:
for arm in arms:
rs = [r for r in res if r["arm"] == arm]
cnt = {v: sum(r["verdict"] == v for r in rs) for v in ("pass", "fail", "unscored", "not_run")}
cnt = {x: sum(r["verdict"] == x for r in rs) for x in ("pass", "fail", "unscored", "not_run")}
xs = crystals(rs)
rows.append([arm, len(rs), cnt["pass"], cnt["fail"], cnt["unscored"], cnt["not_run"], rate(rs),
gain(xs, arm), f(median(r.get("isa") for r in xs), "{:.1f}"),
gain(xs), f(median(r.get("isa") for r in xs), "{:.1f}"),
pct(median(r.get("r_meas") for r in xs)),
f(median(r.get("wall_s") for r in rs), "{:.0f}"),
f(sum(r.get("wall_s") or 0 for r in rs) / 60, "{:.1f}")])
doc.table(head, rows, num=range(1, 12))
doc.svg(verdict_bars([(arm, {v: sum(r["verdict"] == v and r["arm"] == arm for r in res)
for v in ("pass", "fail", "unscored", "not_run")})
for arm in man["arms"]]), "verdicts per arm")
doc.svg(verdict_bars([(arm, {x: sum(r["verdict"] == x and r["arm"] == arm for r in res)
for x in ("pass", "fail", "unscored", "not_run")})
for arm in arms]), f"{v}: verdicts per arm")
doc.p("Pass rate is over the scored sets (pass + fail). Resolution gain is (reference d_min - "
"our d_min) / reference d_min: positive = finer than the deposition, or than XDS; 'forced' "
"where XDS's resolution range was imposed, which makes it zero by construction. Medians "
"leave out the no-crystal controls. R_meas and CC1/2 are pooled over each run's own "
"resolution range, so they describe a run and do not rank two.")
doc.h(3, "By population")
doc.h(3, f"{v}: by population")
rows = []
for arm in man["arms"]:
for arm in arms:
rs = [r for r in res if r["arm"] == arm]
for tag in sorted({t for r in rs for t in r.get("tags", [])}):
sub = [r for r in rs if tag in r.get("tags", [])]
rows.append([arm, tag, len(sub), rate(sub), gain(sub, arm)])
rows.append([arm, tag, len(sub), rate(sub), gain(sub)])
doc.table(["arm", "population", "sets", "pass rate", "median res. gain"], rows, num=(2, 4))
doc.h(3, "Distributions (sets that merged, controls left out)")
doc.h(3, f"{v}: distributions (sets that merged, controls left out)")
rows = []
for arm in man["arms"]:
for arm in arms:
rs = crystals(r for r in res if r["arm"] == arm)
for name, lab, fm in (("isa", "ISa", "{:.1f}"), ("r_meas", "R_meas", None),
("cc_half", "CC1/2", "{:.3f}"), ("res_gain_pct", "res. gain %", "{:+.1f}"),
("wall_s", "time s", "{:.0f}")):
if name == "res_gain_pct" and arm in forced:
("rfree_ratio", "R_free ratio", "{:.3f}"), ("wall_s", "time s", "{:.0f}")):
if name == "res_gain_pct" and forced:
continue
q = quartiles(r.get(name) for r in rs)
if q:
@@ -255,37 +353,36 @@ def build(run_dir, baseline=None, allow_incomplete=False):
rows.append([arm, lab, sum(r.get(name) is not None for r in rs)] + cells)
doc.table(["arm", "metric", "n", "min", "q1", "median", "q3", "max"], rows, num=range(2, 8))
doc.h(3, "Per set, against the reference")
for arm in man["arms"]:
if arm in forced:
doc.h(3, f"{v}: per set, against the reference")
for arm in arms:
if forced:
doc.p(f"{arm}: d_min forced to the reference's, so its ratio is 1 by construction.")
continue
pts = [(r["set"], r["d_min"] / r["d_min_ref"], r["verdict"],
f"d_min {r['d_min']:.2f} vs {r['d_min_ref']:.2f} A, {r['verdict']}")
for r in res if r["arm"] == arm and r.get("d_min") and r.get("d_min_ref")]
if pts:
doc.svg(ratio_dots(pts, f"{arm}: d_min ratio"),
f"{arm}: d_min(rugnux) / d_min(reference), one point per set, sorted; below 1 = "
doc.svg(ratio_dots(pts, f"{arm} {v}: d_min ratio"),
f"{arm}, {v}: d_min(rugnux) / d_min(reference), one point per set, sorted; below 1 = "
"finer than the reference")
pts = [(r["set"], r["rfree_ratio"], r["verdict"],
f"R_free {r['rfree_depflags']:.3f} vs {r['rfree_depdata']:.3f} on the deposited data")
f"R_free {r['rfree']:.3f} vs deposited {r['rfree_deposited']:.3f}")
for r in res if r.get("rfree_ratio")]
if pts:
doc.svg(ratio_dots(pts, "R_free ratio"),
"open: R_free of the deposited model against our merge / against the deposited structure "
"factors, both on the depositor's free set with the same protocol, one point per set, "
"sorted; below 1 = our data fit the model better")
doc.svg(ratio_dots(pts, f"{v}: R_free ratio"),
"R_free of the deposited model against our merge, as rugnux reports it with --model / "
"the R_free the depositor published (from full refinement against their own data), one "
"point per set, sorted; below 1 = lower than published")
pts = [(r["set"], r["refmac_rfree_ratio"], r["verdict"],
f"R_free {r['refmac_rfree_depflags']:.3f} vs {r['refmac_rfree_depdata']:.3f} on the deposited data")
for r in res if r.get("refmac_rfree_ratio")]
if pts:
doc.svg(ratio_dots(pts, f"{v}: REFMAC R_free ratio"),
"REFMAC check (--model-check): R_free of the deposited model against our merge / against "
"the deposited structure factors, both on the depositor's free set with the same protocol, "
"one point per set, sorted; below 1 = our data fit the model better")
derived = [r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"]
if derived:
doc.p("XDS merged past its own signal (CC1/2 of its finest shell not significant), so the "
"reference d_min is where XDS's CC1/2 falls through 0.30 (rule "
f"{derived[0]['d_min_ref_rule']}), and that is also the range forced on rugnux: "
+ "; ".join(f"{r['set']} (XDS range {f(r.get('d_min_xds'))} A, reference "
f"{f(r.get('d_min_ref'))} A)" for r in derived)
+ ". XDS's pooled R_meas, CC1/2 and completeness are over its whole range.")
doc.h(2, "Failures")
doc.h(3, f"{v}: failures")
fails = [r for r in res if r["verdict"] == "fail"]
doc.table(["set", "arm", "cause", "reason"],
[[r["set"], r["arm"], r["cause"], r["reason"]] for r in fails])
@@ -293,26 +390,14 @@ def build(run_dir, baseline=None, allow_incomplete=False):
if other:
doc.p("Not scored: " + "; ".join(f"{r['set']} ({r['reason']})" for r in other))
if baseline:
bman, bres = load_run(baseline)
doc.h(2, f"Delta vs baseline {os.path.basename(os.path.normpath(baseline))}")
doc.p(f"Baseline rugnux {bman['binary'].get('version', '?')}. Pass rates on the sets both "
"runs have:")
both = {key(r) for r in res} & {key(r) for r in bres}
rows = []
for arm in man["arms"]:
ra = [r for r in bres if key(r) in both and r["arm"] == arm]
rb = [r for r in res if key(r) in both and r["arm"] == arm]
rows.append([arm, len(rb), rate(ra), rate(rb)])
doc.table(["arm", "common sets", "baseline", "this run"], rows, num=(1,))
doc.p("Sets whose result moved beyond noise (thresholds in report.py NOISE; confirm with "
"`compare --rerun-changed` before believing a single-set change):")
compare_table(doc, compare_rows(bres, res))
doc.h(2, "Per set")
doc.h(3, f"{v}: per set")
model = [("R_free", "rfree"), ("R_work", "rwork"), ("dep R_free", "rfree_deposited"),
("R_free ratio", "rfree_ratio")] if any(r.get("rfree") for r in res) else []
model += [("REFMAC R_free", "refmac_rfree_depflags"), ("REFMAC dep data", "refmac_rfree_depdata"),
("REFMAC ratio", "refmac_rfree_ratio")] if any(r.get("refmac_status") for r in res) else []
head = ["set", "arm", "verdict", "space group", "ref", "cell dev %", "V ratio", "d_min", "ref d_min",
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx", "R_free", "dep R_free",
"time s", "note"]
"gain", "compl %", "mult", "R_meas", "CC1/2", "ISa", "ref ISa", "idx"] + \
[h for h, _ in model] + ["time s", "note"]
rows = []
for r in sorted(res, key=lambda r: (order.get(r["arm"], 9), r["set"])):
rows.append([r["set"], r["arm"], r["verdict"], r.get("sg") or "-", r.get("sg_ref") or "-",
@@ -320,10 +405,9 @@ def build(run_dir, baseline=None, allow_incomplete=False):
f(r.get("d_min")), f(r.get("d_min_ref")), f(r.get("res_gain_pct"), "{:+.1f}%"),
f(r.get("completeness"), "{:.1f}"), f(r.get("multiplicity"), "{:.1f}"),
pct(r.get("r_meas")), f(r.get("cc_half"), "{:.3f}"), f(r.get("isa"), "{:.1f}"),
f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0),
f(r.get("rfree"), "{:.3f}"), f(r.get("rfree_deposited"), "{:.3f}"),
f(r.get("wall_s"), "{:.0f}"), r.get("reason") or ""])
doc.table(head, rows, num=range(5, 20),
f(r.get("isa_ref"), "{:.1f}"), pct(r.get("indexing_rate"), 0)]
+ [f(r.get(k), "{:.3f}") for _, k in model]
+ [f(r.get("wall_s"), "{:.0f}"), r.get("reason") or ""])
doc.table(head, rows, num=range(5, len(head) - 1),
row_class=lambda row: "fail" if row[2] == "fail" else "",
cell_class=lambda i, v: {"pass": "pass", "fail": "fail"}.get(v) if i == 2 else None)
return doc
cell_class=lambda i, x: {"pass": "pass", "fail": "fail"}.get(x) if i == 2 else None)
+6 -1
View File
@@ -78,7 +78,7 @@ def sg_name(sgno):
def judge(entry, rep, run_note):
"""Score one set. entry is its manifest row (with 'ref'), rep its parsed report, run_note
what the runner saw ('', 'timeout', 'exit N: message', 'no input')."""
what the runner saw ('', 'timeout', 'exit N: message', 'no input', 'no model: why')."""
ref = dict(entry.get("ref") or {})
if entry.get("ref_override"): # a reference measured by hand, replacing the file's
ref.update(entry["ref_override"])
@@ -96,6 +96,9 @@ def judge(entry, rep, run_note):
"rugnux_wall_s": fnum(rep.get("WALL_TIME")), "rugnux_verdict": rep.get("VERDICT_TEXT"),
"r_meas_ref": ref.get("r_meas"), "cc_half_ref": ref.get("cc_half"), "isa_ref": ref.get("isa"),
"completeness_ref": ref.get("completeness"), "sg_relation": None,
# rugnux's own R-factors of the deposited model, when it was given one (--model)
"rfree": fnum(rep.get("R_FREE")), "rwork": fnum(rep.get("R_WORK")),
"model_fit": rep.get("MODEL_FIT"),
}
if rep.get("SPACE_GROUP_NUMBER"):
r["sgno"] = int(fnum(rep["SPACE_GROUP_NUMBER"]))
@@ -125,6 +128,8 @@ def judge(entry, rep, run_note):
if not have_lattice:
if run_note == "no input":
return verdict("not_run", "no_input", "input file not found")
if run_note.startswith("no model"):
return verdict("not_run", "no_model", run_note)
if run_note == "timeout":
return verdict("fail", "timeout", "timed out")
if re.search(r"Error reading input|Cannot open", run_note):
+1
View File
@@ -5,6 +5,7 @@
"gpulock": null,
"baseline": null,
"baseline_private": null,
"pdb_cache": "/path/to/battery/pdb_cache",
"arms": {
"open": {"manifest": "open.json", "reference": "deposition", "data_root": "/path/to/battery/open"},
"inhouse": {"manifest": "inhouse.json", "reference": "xds", "data_root": "/path/to/battery/inhouse"}