diff --git a/tools/battery/README.md b/tools/battery/README.md index 80d33c4b0..71869800e 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -23,20 +23,12 @@ scored with `sgequiv`. On the XDS arms only the point group is scored, because X screw axis. A no-crystal control (`"expect": "no_lattice"`) passes only if no lattice is reported. `score.py` has the details. -**Merge quality is judged only against XDS, and only by CC1/2.** No R_meas is a pass/fail -criterion, and no absolute CC1/2 either. On the XDS arms the set fails `merge` when - - (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) > 2 - -with rugnux's CC1/2 from the reference-range table (`REFRES_CC_HALF`, XDS's own range) and XDS's -from the `CORRECT.LP` total. CC1/2 = S / (S + E), with S the signal variance and E the half-set -error variance, so 1/CC1/2 - 1 = E/S at any CC1/2, and E goes as 1/(observations): a ratio above 2 -is a merge noisier than XDS's would be with half of its observations. XDS prints CC1/2 to 0.1%, so -its value is taken at the bottom of that rounding (-0.0005), and a printed 100.0 does not demand a -perfect CC1/2. The ratio is stored as `cc_half_noise_ratio`. A set without the reference-range -table or without an XDS CC1/2 (the controls, a hand-measured reference) has no merge criterion. -**The open arm has no merge criterion at all**: it is scored on lattice and symmetry, and its -resolution, CC1/2 and R_meas are reported, not judged. +**Data quality is not scored.** A verdict is about the lattice and the symmetry only. Resolution, +CC1/2, R_meas (overall and lowest shell), completeness, multiplicity and ISa are reported beside +XDS's or the deposition's values, in the per-set tables and plots, for a human to judge. On the XDS +arms `cc_half_noise_ratio` = (1/CC1/2_rugnux - 1) / (1/(CC1/2_XDS - 0.0005) - 1) over XDS's own +range is shown as a guide: CC1/2 = S/(S+E), so 1/CC1/2 - 1 = E/S, and 2 means a merge as noisy as +XDS's would be with half of its observations. **Low-resolution R_meas** is reported, not scored: the lowest-resolution shell of rugnux's own table (`lowres_r_meas`), of the reference-range table (`refres_lowres_r_meas`) and of XDS's @@ -256,7 +248,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. |---|---| | `set`, `arm`, `tags`, `input`, `cmd` | the set's id, arm, population tags, input file, and the exact command that ran | | `verdict` | `pass`, `fail`, `unscored` (no reference) or `not_run` (input missing) | -| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `merge` (XDS arms: CC1/2 noise ratio above 2), `false_lattice`; or `no_reference` / `no_input` / `reference_problem` | +| `cause` | why it failed: `crash`, `reader`, `indexing`, `timeout`, `lattice_halved`, `lattice_doubled`, `lattice_other`, `sym_under`, `sym_over`, `sym_screw`, `sym_other`, `false_lattice`; or `no_reference` / `no_input` / `reference_problem` | | `reason` | one line for a human | | `sgno`, `sg`, `pg` / `sgno_ref`, `sg_ref`, `pg_ref` | space group number and name, and point group: ours (the data's own determination) / the reference's | | `sg_label` | the space group rugnux reported where `--model` put the model's enantiomorph on it, else null | @@ -268,7 +260,7 @@ was run `--unforced`), so its forced XDS-arm rows are left out the same way. | `r_meas_ref`, `cc_half_ref`, `isa_ref`, `completeness_ref`, `multiplicity_ref` | XDS's overall values, over XDS's own range | | `refres_range`, `refres_shells_past_limit`, `refres_unique_reflections`, `refres_completeness`, `refres_multiplicity`, `refres_i_over_sigma`, `refres_r_meas`, `refres_cc_half`, `refres_isa` | XDS arms: the REFRES_* keys, the same merge over the reference range | | `isa_ratio`, `r_meas_ratio` | refres_isa / isa_ref and refres_r_meas / r_meas_ref | -| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), the merge criterion | +| `cc_half_noise_ratio` | XDS arms: (1/refres_cc_half - 1) / (1/(cc_half_ref - 0.0005) - 1), reported, not scored | | `lowres_d`, `lowres_r_meas`, `refres_lowres_d`, `refres_lowres_r_meas`, `lowres_d_ref`, `lowres_r_meas_ref`, `lowres_r_meas_ratio` | the lowest-resolution shell's high-resolution limit and R_meas: rugnux's own table, the reference-range table, XDS's; and refres_lowres_r_meas / lowres_r_meas_ref | | `model`, `model_note` | open arm: the deposited coordinates given to `--model`, or why there were none | | `rfree`, `rwork`, `cc_model`, `model_fit`, `rfree_deposited`, `rfree_ratio` | open arm with a model: R_FREE, R_WORK, CC_MODEL_OVERALL and MODEL_FIT as rugnux reports them (placement-only, own free set: trend fields), the published R-free, and rfree / rfree_deposited | diff --git a/tools/battery/report.py b/tools/battery/report.py index 23ea0089a..724d0e6a2 100644 --- a/tools/battery/report.py +++ b/tools/battery/report.py @@ -355,8 +355,8 @@ def build(run_dir, baseline=None, allow_incomplete=False): "reference-range table / of XDS's table; below 1 = " "rugnux's strong reflections agree better"), ("cc_half_noise_ratio", "CC1/2 noise", "half-set noise over the reference range read off CC1/2 " - "(1 / CC1/2 - 1) / XDS's; above 2 fails (the merge is as " - "noisy as XDS's with half its observations)")): + "(1 / CC1/2 - 1) / XDS's; 2 = as noisy as XDS's merge " + "would be with half its observations (not scored)")): for arm in arms: pts = [(r["set"], r[name], r["verdict"], f"{what} ratio, {refres_note(r)}") for r in res if r["arm"] == arm and r.get(name)] @@ -405,13 +405,12 @@ def build(run_dir, baseline=None, allow_incomplete=False): doc.table(["set", "arm", "range A", "own d_min", "compl % (rugnux / XDS)", "mult", "I/sigma", "R_meas", "low-res R_meas (to d A)", "CC1/2", "CC1/2 noise ratio", "ISa", "reading"], rows, num=(3, 6, 10)) - doc.p("Merge quality is scored here and only here, by CC1/2 against XDS's over the same range. " + doc.p("Data quality is not scored; this table is for a human to judge. " "CC1/2 = S / (S + E) (signal and half-set error variances), so 1 / CC1/2 - 1 = E / S is the " "half-set noise; the noise ratio is rugnux's over XDS's, with XDS's CC1/2 taken at the " - "bottom of its printed rounding (-0.0005). A set fails `merge` when the ratio is above " - f"{score.CC_HALF_NOISE_LIMIT:g}: noisier than XDS's merge would be with half its " - "observations. The low-resolution R_meas is the lowest shell of each table (the shells " - "are XDS's, so both cover the same reflections); it is reported, not scored.") + "bottom of its printed rounding (-0.0005); 2 would mean noisier than XDS's merge with half " + "its observations. The low-resolution R_meas is the lowest shell of each table (the shells " + "are XDS's, so both cover the same reflections); reported like everything here.") derived = {r["set"]: r for r in res if (r.get("d_min_ref_rule") or "xds_range") != "xds_range"} if derived: diff --git a/tools/battery/score.py b/tools/battery/score.py index 865daf681..af44f3d57 100644 --- a/tools/battery/score.py +++ b/tools/battery/score.py @@ -1,7 +1,7 @@ """Scoring one rugnux run against its reference. One dataset, one verdict, one cause, decided in this order: did it run -> is the lattice right -> -is the symmetry right -> (XDS arms only) is the merge as good as XDS's by CC1/2. The order matters: +is the symmetry right. Data quality is not scored: it is reported for a human to judge. The order matters: a halved axis also makes the screw along it unobservable, and counting that row twice would hide the defect that cost the reflections. """ @@ -92,15 +92,14 @@ def cell_dev_pct(cell, ref): # XDS prints CC1/2 in % with one decimal; its value is taken at the bottom of that rounding, so a # printed 100.0 does not demand a perfect CC1/2 of rugnux XDS_CC_HALF_ROUNDING = 0.0005 -CC_HALF_NOISE_LIMIT = 2.0 def cc_half_noise_ratio(cc, cc_xds): """rugnux's half-set noise over XDS's, both read off CC1/2 over XDS's range; None without both. CC1/2 = S / (S + E), with S the variance of the signal and E that of the half-set error, so - E / S = 1 / CC1/2 - 1 at any CC1/2. E goes as 1 / (observations): a ratio of 2 - (CC_HALF_NOISE_LIMIT) is a merge as noisy as XDS's would be with half of its observations. + E / S = 1 / CC1/2 - 1 at any CC1/2. E goes as 1 / (observations): a ratio of 2 is a merge as + noisy as XDS's would be with half of its observations. Reported, not scored. A CC1/2 at or below 0 is no signal at all, counted as 0.001.""" if cc is None or cc_xds is None: return None @@ -233,11 +232,7 @@ def judge(entry, rep, run_note): "sym_over" if order > order_ref else "sym_other") return verdict("fail", cause, f"{r['sg']} vs reference {r['sg_ref']}") - # Merge quality is judged only against XDS, by CC1/2 over XDS's own range (cc_half_noise_ratio). - # The open arm has no merge criterion. - if r["cc_half_noise_ratio"] is not None and r["cc_half_noise_ratio"] > CC_HALF_NOISE_LIMIT: - return verdict("fail", "merge", f"CC1/2 {r['refres_cc_half']:.4f} vs XDS {r['cc_half_ref']:.3f} " - f"over XDS's range: {r['cc_half_noise_ratio']:.1f}x XDS's half-set noise") + # Data quality (CC1/2, R_meas, resolution, ISa) is reported, never scored: that is for a human. note = "" if r["sgno"] != ref["sgno"]: note = f"{r['sg']} vs reference {r['sg_ref']} ({r.get('sg_relation') or 'screws not judged against XDS'})"