diff --git a/docs/EXTERNAL_TEST_DATA.md b/docs/EXTERNAL_TEST_DATA.md index ada10d76d..815ad710c 100644 --- a/docs/EXTERNAL_TEST_DATA.md +++ b/docs/EXTERNAL_TEST_DATA.md @@ -341,6 +341,101 @@ reports released structure factors (`status_code_sf = REL`) for every one of the result from this pipeline can therefore be checked against the deposited model or against the deposited intensities. +## Rows where our reduction and the deposition disagree + +Five of the 153 rows are ones where `rugnux` does not reproduce the deposited space group or +cell, and where we have looked at the disagreement closely enough to change how the row is +scored. They are collected here because a scoring row that silently disagrees with a published +entry is not something a reader should have to discover from the code. + +**These are open questions, not errors we are attributing to the PDB.** A deposited entry was +arrived at by someone who had something we do not: a model that had to refine, and usually more +knowledge of the crystal than the images carry. Where we describe evidence below, it is evidence +about *what these images support*, which is a narrower thing than what the crystal is. In every +one of the symmetry rows the possibility that the crystal really has the lower symmetry, with a +pseudo-symmetry too exact for any test available to us to see, remains live - see the limit at +the end of this section. + +How the manifest records it, in `tools/battery/open.json`: + +- `ref` always keeps the deposited values verbatim, so the deposition is never lost. +- `ref_override` replaces only the fields the battery scores against, and `ref_override_why` + records the evidence and its limits. +- `unscored` is for a row we cannot defend a reference for in either direction: the set still + runs, and its merging statistics are still reported, but it is not scored. + +### 8XBP is a question about provenance, not about symmetry + +8XBP is different in kind from the other four and should not be read alongside them. Nothing +here concerns the deposited model or its space group. The question is whether the raw images +uploaded with the entry are the same crystal the deposited cell describes: the master file +records `data_collection_date` 2023-06-21 where the entry records a collection date of +2023-06-23, and the deposited *b* = 50.78 A is 2.0% away from the *b* these images give. Two +independent signals, one of them nothing to do with our processing. The override replaces the +cell with the one DIALS 3.29 indexes de novo on this master and keeps the deposited space group +and resolution. + +### Four trigonal and tetragonal rows where we read a higher point group + +| PDB | Deposited | `rugnux` reads | Row is now | Where it stands | +|---|---|---|---|---| +| 6TOC | P 42 | P 42 2 2 | overridden | strong, but the refinement test is not unanimous | +| 8XTE | P 32 | P 31 2 1 / P 32 2 1 | overridden | strong; rests on the centric-zone test | +| 8XTG | P 32 | P 31 2 1 / P 32 2 1 | overridden | **open** - our own best test argues against it | +| 6PXB | P 32 | P 31 1 2 / P 32 1 2 | unscored | unresolved in either direction | + +The 8XTE and 8XTG overrides predate the 8XBP and 6TOC ones and were carried in the manifest for +some time with no reason recorded against them; the reasons were written down later, when this +section was added, rather than at the time the rows were changed. + +**6TOC.** The deposited asymmetric unit holds two chains, and they are related by the very +two-fold the higher group adds, to 0.16 A C-alpha RMSD over 43 residues - coordinate error at +the deposited 1.85 A. Merging in P 42 2 2 costs 0.0006 in Rmeas for 1.75 +times the multiplicity, and correlates better with the deposited model than the P 42 +merge does. POINTLESS, run independently on our own P1 merge, reads the same point group. The +refinement test - refine in each candidate group and compare R-free, which is the one comparison +not biased toward the group the deposited model was refined in - does **not** come out +unanimous: ZANUDA 1.097 makes P 42 2 2 the better group at half the parameters and +reports the deposited assignment incorrect, while an independent Refmac 5.8.0431 comparison on a +symmetry-consistent free set makes P 42 the better one, by less than the spread +between refinement protocols. So the row is overridden on the structural and merging evidence, +with the refinement evidence recorded as split. + +**8XTE.** The distinguishing test is the twin-immune centric zone: reflections that the higher +group makes centric but the subgroup does not are their own twin mates, so a merohedral twin law +cannot make them read centric. They read `<|E^2-1|>` = 0.946 +/- 0.012 against a centric +expectation of 0.968 and an acentric one of 0.736. Re-refinement on a shared free set, with the +twin law removed from both sides, favours the higher group. The deposited entry's published R +values are themselves reproducible only with a twin law the entry does not declare, at a twin +fraction of 0.50 - and a 0.50-twinned target already has the symmetry in question. + +**8XTG.** This row is genuinely open and is flagged as such in the manifest. Every +correlation-based instrument we have - our own operator correlations, and POINTLESS on our P1 +merge - reads the higher point group, but the centric-zone test, the only one of them that can +separate real symmetry from pseudo-symmetry, reads `<|E^2-1|>` = 0.869 at -44.9 nats: between +the two expectations, and on the wrong side. The L-test indicates a twin fraction near 0.20-0.26. +Whether this crystal is partially twinned or purely pseudo-symmetric has not been established, +and our own best test argues against the promotion the row is scored on. + +**6PXB.** Unscored rather than overridden, because the evidence does not settle either way. Our +merge and POINTLESS both read a 312 point group, the added two-folds correlate at or above the +level of the three-folds nobody disputes, and merging in the higher group *lowers* +Rmeas at twice the multiplicity. Against that, the deposited asymmetric unit's six +chains pair under the added two-fold at 0.3-0.7 A, which is more than coordinate error at 1.75 A, +and ZANUDA settles on a different trigonal supergroup - 321 rather than 312 - whose operators +these data do not support. Neither answer is established, so the set runs and is not scored. + +### The limit that applies to all four + +A merohedral twin at a twin fraction of exactly 0.5 and a crystal that genuinely has the higher +symmetry predict **identical** intensities. No amount of data and no refinement R separates them, +and the same holds, approximately, for a pseudo-symmetry that is merely very exact. Every test +described above measures how nearly a symmetry operator holds on these images; none of them can +show that it holds exactly. Where the higher symmetry is right, merging in it gains multiplicity +and completeness; where it is a pseudo-symmetry that close, merging in it costs nothing +measurable either. That is why these rows are described as open questions, and why none of them +should be read as a statement that a deposited model is wrong. + ## Dataset directories whose name is not the PDB code | Directory | PDB code in the table | Why | diff --git a/tools/battery/open.json b/tools/battery/open.json index f7aeb01a7..2770a8f21 100644 --- a/tools/battery/open.json +++ b/tools/battery/open.json @@ -33,11 +33,11 @@ {"id": "6oel", "input": "6oel/1449-8_3_001.img", "ref": {"sg": "F 41 3 2", "sgno": 210, "cell": [328.1, 328.1, 328.1, 90.0, 90.0, 90.0], "dmin": 3.1}, "tags": ["SMV", "cubic"], "tiers": {"smoke": "SMV reader (ADSC), cubic F4132"}}, {"id": "6p8p", "input": "6p8p/14_12_1_0001.cbf", "ref": {"sg": "P 4", "sgno": 75, "cell": [97.525, 97.525, 60.134, 90.0, 90.0, 90.0], "dmin": 1.635}, "tags": ["cbf", "tetragonal"]}, {"id": "6pb3", "input": "6pb3/14_16_1_000001.cbf", "ref": {"sg": "P 6", "sgno": 168, "cell": [100.436, 100.436, 48.86, 90.0, 90.0, 120.0], "dmin": 2.048}, "tags": ["cbf", "hexagonal"]}, - {"id": "6pxb", "input": "6pxb/TJB1_3_rachel_1_0001.cbf", "ref": {"sg": "P 32", "sgno": 145, "cell": [64.021, 64.021, 119.447, 90.0, 90.0, 120.0], "dmin": 1.747}, "tags": ["cbf", "trigonal"]}, + {"id": "6pxb", "input": "6pxb/TJB1_3_rachel_1_0001.cbf", "ref": {"sg": "P 32", "sgno": 145, "cell": [64.021, 64.021, 119.447, 90.0, 90.0, 120.0], "dmin": 1.747}, "unscored": "disputed and open in both directions - see unscored_why", "unscored_why": "our merge and POINTLESS (Laue confidence 0.999) both read point group 312 on these images where the deposition is P 32, the added two-folds correlate at or above the level of the 3-folds nobody disputes, and merging in P 32 1 2 lowers R_meas from 0.0716 to 0.0662 at twice the multiplicity. Against that, the deposited model's six chains pair under the added two-fold at 0.3-0.7 A, which is more than coordinate error at 1.75 A, and ZANUDA settles on a different trigonal supergroup (P 32 2 1) whose operators these data do not support (CC 0.18-0.47 against 0.75-0.99 for P 32 1 2's). Neither answer is established, so the set runs but is not scored. A perfect merohedral twin and true higher symmetry predict identical data, so pseudo-symmetry remains a live explanation either way", "tags": ["cbf", "trigonal"]}, {"id": "6pxc", "input": "6pxc/TJB6_1_Rachel_1_0001.cbf", "ref": {"sg": "I 2 2 2", "sgno": 23, "cell": [44.19, 64.827, 87.239, 90.0, 90.0, 90.0], "dmin": 1.6}, "tags": ["cbf", "orthorhombic"]}, {"id": "6r72", "input": "6r72/V-CK63-8-ld_1_master.h5", "ref": {"sg": "P 1 21 1", "sgno": 4, "cell": [117.805, 110.754, 155.615, 90.0, 93.23, 90.0], "dmin": 3.95}, "pinned": true, "tags": ["h5", "monoclinic"]}, {"id": "6rlr", "input": "6rlr/_b4_1_master.h5", "ref": {"sg": "P 1", "sgno": 1, "cell": [39.986, 39.998, 63.643, 80.39, 76.29, 68.15], "dmin": 2.0}, "tags": ["h5", "triclinic", "twin"]}, - {"id": "6toc", "input": "6toc/zenodo/dts_1_00001.cbf", "ref": {"sg": "P 42", "sgno": 77, "cell": [31.523, 31.523, 81.599, 90.0, 90.0, 90.0], "dmin": 1.853}, "tags": ["cbf", "tetragonal", "twin"]}, + {"id": "6toc", "input": "6toc/zenodo/dts_1_00001.cbf", "ref": {"sg": "P 42", "sgno": 77, "cell": [31.523, 31.523, 81.599, 90.0, 90.0, 90.0], "dmin": 1.853}, "ref_override": {"sg": "P 42 2 2", "sgno": 93}, "ref_override_why": "our merge and an independent POINTLESS run on the same P1 merge both read point group 422, and the deposited model's own two chains are related by the added two-fold to 0.16 A CA RMSD over 43 residues, which is coordinate error at 1.85 A. Merging in P 42 2 2 costs 0.0006 in R_meas for 1.75x the multiplicity and correlates better with the deposited model (0.9802 vs 0.9764 at 1.853 A). ZANUDA 1.097 refines P 42 2 2 to R-free 0.2561 against P 42's 0.2636 at half the parameters and calls the deposited assignment incorrect, but an independent refmac 5.8.0431 comparison on a symmetry-consistent free set puts P 42 ahead by 0.004-0.020 depending on cycle count, so the refinement test is NOT unanimous and its protocol spread exceeds the difference being measured. Pseudo-symmetry exact enough to escape every test we have remains a live explanation. The deposited cell and d_min are kept", "tags": ["cbf", "tetragonal", "twin"]}, {"id": "6ttn", "input": "6ttn/6ttn/data/H6H_33_01_hyo_full_1_0001.cbf", "ref": {"sg": "P 21 21 21", "sgno": 19, "cell": [39.894, 79.841, 104.701, 90.0, 90.0, 90.0], "dmin": 1.12}, "tags": ["cbf", "orthorhombic"]}, {"id": "6u7g", "input": "6u7g/6u7g/data/SNB02_11_8_1_master.h5", "ref": {"sg": "P 1 21 1", "sgno": 4, "cell": [99.555, 98.682, 147.525, 90.0, 104.6, 90.0], "dmin": 2.35}, "pinned": true, "tags": ["h5", "monoclinic"]}, {"id": "6ukf", "input": "6ukf/6ukf/data/XDC-7_Pn6_000001.cbf", "ref": {"sg": "P 1 21 1", "sgno": 4, "cell": [61.018, 37.317, 69.027, 90.0, 109.768, 90.0], "dmin": 1.0}, "tags": ["cbf", "monoclinic"]}, @@ -97,9 +97,9 @@ {"id": "8u0i", "input": "8u0i/8u0i/data/dI-2-05a_4_00001.cbf", "ref": {"sg": "P 43 21 2", "sgno": 96, "cell": [50.317, 50.317, 90.596, 90.0, 90.0, 90.0], "dmin": 1.541}, "tags": ["cbf", "tetragonal"]}, {"id": "8v4o", "input": "8v4o/CaalA_00629_a_FS11-AMP_8v4o/data/PSL-1602_2180_master.h5", "ref": {"sg": "P 61 2 2", "sgno": 178, "cell": [139.46, 139.46, 544.99, 90.0, 90.0, 120.0], "dmin": 2.7}, "tags": ["h5", "hexagonal"]}, {"id": "8xbp", "input": "8xbp/8xbp/data/puck2872_3_1_master.h5", "ref": {"sg": "C 1 2 1", "sgno": 5, "cell": [148.29, 50.78, 60.21, 90.0, 92.33, 90.0], "dmin": 1.99}, "ref_override": {"cell": [148.736, 51.777, 60.181, 90.0, 92.36, 90.0]}, "ref_override_why": "the uploaded sweep is a different crystal from the one the deposited cell describes: master data_collection_date 2023-06-21, deposited _diffrn_detector.pdbx_collection_date 2023-06-23, and the deposited b = 50.78 is 2.0% from the b these images give. The override is the cell DIALS 3.29 indexes de novo on this master, b = 51.777(11); the deposited space group and d_min are kept", "tags": ["h5", "monoclinic"]}, - {"id": "8xte", "input": "8xte/1101/G2_1_00001.cbf", "ref": {"sg": "P 32 2 1", "sgno": 154, "cell": [208.77, 208.77, 67.22, 90.0, 90.0, 120.0], "dmin": 1.99}, "tags": ["cbf", "trigonal"]}, + {"id": "8xte", "input": "8xte/1101/G2_1_00001.cbf", "ref": {"sg": "P 32", "sgno": 145, "cell": [208.77, 208.77, 67.22, 90.0, 90.0, 120.0], "dmin": 1.99}, "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "deposited P 32; this override predates the 8xbp and 6toc ones and was recorded here later. Our merge and POINTLESS both read point group 321, and the twin-immune centric zone of the added two-folds reads <|E^2-1|> = 0.946 +- 0.012 at +707.3 nats - essentially centric, which a merohedral twin law cannot produce. Re-refinement on a shared free set with the twin law removed from both sides gives P 32 2 1 0.2177/0.2322 against P 32 0.2460/0.2612, and the deposited entry's own R values reproduce only with an undeclared twin law h,-h-k,-l at alpha = 0.50, which is itself a 321-symmetric target. A perfect alpha = 0.5 merohedral twin and true 321 predict identical data, so no refinement R can close this; the case rests on the centric zone. The deposited cell and d_min are kept", "tags": ["cbf", "trigonal"]}, {"id": "8xtf", "input": "8xtf/1102/puck03_12_1_master.h5", "ref": {"sg": "H 3 2", "sgno": 155, "cell": [211.75, 211.75, 67.42, 90.0, 90.0, 120.0], "dmin": 2.13}, "tags": ["h5", "trigonal"]}, - {"id": "8xtg", "input": "8xtg/1100/mpag15_1_00001.cbf", "ref": {"sg": "P 32 2 1", "sgno": 154, "cell": [199.54, 199.54, 67.15, 90.0, 90.0, 120.0], "dmin": 2.0}, "tags": ["cbf", "trigonal"]}, + {"id": "8xtg", "input": "8xtg/1100/mpag15_1_00001.cbf", "ref": {"sg": "P 32", "sgno": 145, "cell": [199.54, 199.54, 67.15, 90.0, 90.0, 120.0], "dmin": 2.0}, "ref_override": {"sg": "P 32 2 1", "sgno": 154}, "ref_override_why": "deposited P 32; this override predates the 8xbp and 6toc ones and was recorded here later. THE CASE IS OPEN. Our merge and POINTLESS (0.85 on our own P1 merge) read point group 321, but the twin-immune centric zone of the added two-folds reads <|E^2-1|> = 0.869 at -44.9 nats - between the centric and acentric expectations and negative - so the one test we have that separates real symmetry from pseudo-symmetry argues AGAINST the promotion, and the L-test indicates a twin fraction near 0.20-0.26. Whether the crystal is partially twinned or purely pseudo-symmetric is not established. This row may belong unscored rather than overridden. The deposited cell and d_min are kept", "tags": ["cbf", "trigonal"]}, {"id": "8y74", "input": "8y74/1_1/1_1_master.h5", "ref": {"sg": "C 1 2 1", "sgno": 5, "cell": [125.782, 76.553, 87.143, 90.0, 92.449, 90.0], "dmin": 1.9}, "tags": ["h5", "monoclinic"]}, {"id": "8ys9", "input": "8ys9/8YS9_xray-data_8ys9/data/SDS_09_18112_master.h5", "ref": {"sg": "P 21 21 21", "sgno": 19, "cell": [71.04, 77.68, 83.16, 90.0, 90.0, 90.0], "dmin": 1.46}, "tags": ["h5", "orthorhombic"]}, {"id": "9b22", "input": "9b22/KlpnC_20447_a_B1-AR6-AMP_9b22/data/PSL-1011_1580_master.h5", "ref": {"sg": "P 1 21 1", "sgno": 4, "cell": [39.826, 92.659, 57.716, 90.0, 91.68, 90.0], "dmin": 1.3}, "tags": ["h5", "monoclinic"]},