leonarski_fandClaude Opus 5
0c91749a41
A handful of open-arm rows disagree with their deposition on a knife edge that no test available to us settles. They were handled three different ways - silently overridden to our answer, marked unscored, or left failing - and none of the three says what is true: either answer is acceptable as long as the program picks one of them. A manifest row can now list `ref_alternatives`. Each entry replaces the reference fields it names - a space group, a cell, or both - and the row passes if the answer matches any of its references, the deposition included. `ref` keeps the deposited values verbatim in every case. Every alternative must carry `why`: an accepted alternative with no stated reason raises rather than passing, so the mechanism cannot be used to launder a failure. The report keeps these rows visible rather than folding them into the passes: a summary column counting them, their own segment in the verdict bars, and a section naming each row, what we read, what was deposited and the reason both are accepted. Five rows use it. Four are symmetry: a tetragonal row where the refinement test is split and its spread exceeds the effect, and three trigonal rows where we read a higher point group - one where the evidence favours our answer, one where our own twin-immune test favours the deposition, one unresolved in either direction. The fifth is a cell: a real tNCS supercell whose (0,1/2,1/2) sublattice is what was deposited, both being correct descriptions of the same lattice. The documentation frames all of them as open questions, not as errors in a deposition, and states the limit: a merohedral twin at exactly one half and true higher symmetry predict identical intensities, so no test can close them even in principle. `test_score.py` covers the new path: both answers accepted, the other hand of an alternative, a third answer still failing, the cell case, and the missing-justification schema error. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013nW6FNRP1bBJJ8pfHiByAT
98 lines
5.0 KiB
Python
98 lines
5.0 KiB
Python
"""Targeted tests of score.py, the part a wrong answer would be laundered through.
|
|
|
|
Plain stdlib unittest, so it runs with `python3 tools/battery/test_score.py` (or
|
|
`python3 -m unittest`) from this directory, without pytest and without a battery run.
|
|
"""
|
|
import unittest
|
|
|
|
import score
|
|
|
|
|
|
def report(sg, sgno, cell, **extra):
|
|
"""The three report keys the scorer needs, plus whatever a case adds."""
|
|
return dict({"SPACE_GROUP_NAME": sg, "SPACE_GROUP_NUMBER": str(sgno),
|
|
"UNIT_CELL_CONSTANTS": " ".join(f"{x:.3f}" for x in cell)}, **extra)
|
|
|
|
|
|
# A deposited trigonal reference and the higher group our reduction reads on it: the shape of
|
|
# the accepted-alternative rows, with a synthetic cell, not a real one.
|
|
TRIGONAL = [80.0, 80.0, 150.0, 90.0, 90.0, 120.0]
|
|
DEPOSITED = {"sg": "P 32", "sgno": 145, "cell": TRIGONAL, "dmin": 2.0}
|
|
|
|
|
|
def entry(**extra):
|
|
return dict({"id": "test_set", "arm": "open", "ref": DEPOSITED}, **extra)
|
|
|
|
|
|
class AcceptedAlternatives(unittest.TestCase):
|
|
def test_deposition_still_decides_on_its_own(self):
|
|
"""Without alternatives the higher group is a symmetry failure, as before."""
|
|
r = score.judge(entry(), report("P 31 2 1", 152, TRIGONAL), "")
|
|
self.assertEqual((r["verdict"], r["cause"]), ("fail", "sym_over"))
|
|
|
|
def test_either_answer_passes(self):
|
|
"""Both the deposited group and the alternative pass, and only the second is marked."""
|
|
e = entry(ref_alternatives=[{"sg": "P 31 2 1", "sgno": 152, "why": "a stated reason"}])
|
|
dep = score.judge(e, report("P 32", 145, TRIGONAL), "")
|
|
self.assertEqual((dep["verdict"], dep["cause"], dep["accepted_alt"]), ("pass", None, None))
|
|
alt = score.judge(e, report("P 31 2 1", 152, TRIGONAL), "")
|
|
self.assertEqual((alt["verdict"], alt["cause"]), ("pass", "accepted_alternative"))
|
|
self.assertEqual(alt["accepted_alt"], "P 31 2 1")
|
|
self.assertIn("a stated reason", alt["accepted_alt_why"])
|
|
# the row still describes the answer against the DEPOSITION, not against the alternative
|
|
self.assertEqual(alt["sg_ref"], "P 32")
|
|
|
|
def test_the_other_hand_of_an_alternative_passes_too(self):
|
|
"""P 31 2 1 and P 32 2 1 differ by hand only, which intensities cannot decide."""
|
|
e = entry(ref_alternatives=[{"sg": "P 31 2 1", "sgno": 152, "why": "a stated reason"}])
|
|
r = score.judge(e, report("P 32 2 1", 154, TRIGONAL), "")
|
|
self.assertEqual((r["verdict"], r["cause"]), ("pass", "accepted_alternative"))
|
|
|
|
def test_a_third_answer_still_fails(self):
|
|
"""An alternative accepts one other answer, not any other answer."""
|
|
e = entry(ref_alternatives=[{"sg": "P 31 2 1", "sgno": 152, "why": "a stated reason"}])
|
|
r = score.judge(e, report("P 31 1 2", 151, TRIGONAL), "")
|
|
self.assertEqual(r["verdict"], "fail")
|
|
|
|
def test_alternative_cell(self):
|
|
"""A supercell alternative: the deposited cell is its (0,1/2,1/2) sublattice, so the
|
|
deposition alone scores the answer as a doubled lattice and the alternative accepts it."""
|
|
sub = [35.869, 39.297, 100.916, 98.3, 90.32, 90.09]
|
|
sup = [35.869, 39.297, 199.976, 87.087, 90.341, 90.09] # c' = b + 2c, centring removed
|
|
e = {"id": "test_cell", "arm": "open", "ref": {"sg": "P 1", "sgno": 1, "cell": sub}}
|
|
r = score.judge(e, report("P 1", 1, sup), "")
|
|
self.assertEqual((r["verdict"], r["cause"]), ("fail", "lattice_doubled"))
|
|
e["ref_alternatives"] = [{"cell": sup, "why": "a stated reason"}]
|
|
r = score.judge(e, report("P 1", 1, sup), "")
|
|
self.assertEqual((r["verdict"], r["cause"]), ("pass", "accepted_alternative"))
|
|
self.assertTrue(r["accepted_alt"].startswith("cell "))
|
|
self.assertEqual(r["volume_ratio"], 2.0) # still reported against the deposition
|
|
|
|
def test_a_justification_is_mandatory(self):
|
|
"""An accepted alternative with no reason is a schema error, not a silent pass: this is
|
|
the check that keeps the mechanism from becoming a way to launder a failure."""
|
|
for alt in ({"sg": "P 31 2 1", "sgno": 152},
|
|
{"sg": "P 31 2 1", "sgno": 152, "why": " "}):
|
|
with self.assertRaises(ValueError) as cm:
|
|
score.judge(entry(ref_alternatives=[alt]), report("P 31 2 1", 152, TRIGONAL), "")
|
|
self.assertIn("why", str(cm.exception))
|
|
|
|
def test_an_alternative_must_replace_something(self):
|
|
with self.assertRaises(ValueError):
|
|
score.judge(entry(ref_alternatives=[{"why": "a stated reason"}]),
|
|
report("P 32", 145, TRIGONAL), "")
|
|
|
|
def test_the_manifest_rows_are_valid(self):
|
|
"""Every committed row's alternatives pass the schema check."""
|
|
import json
|
|
import os
|
|
for name in ("open.json", "inhouse.json"):
|
|
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), name)
|
|
with open(path) as f:
|
|
for e in json.load(f)["sets"]:
|
|
score.alternatives(e)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|