From 7b1caa6ce5b1a935f89939d72a136dbe45775ac1 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 2 Sep 2026 10:48:07 +0200 Subject: [PATCH] mmcif: the free set is _refln.status = f, not a column of our own The merged mmCIF carried the R-free flag in `_refln.status_free` as 1/0 and wrote `o` into `_refln.status` on every row. `_refln.status_free` is not in the PDBx/mmCIF dictionary - it is absent from mmcif_pdbx v4.0, v5.0, v5.288 and v5.362, from CCP4's and phenix's shipped copies, and its wwPDB item page is a 404 - and no deposited structure-factor file uses it. `_refln.status` is the item that carries the free set, with `f` for a test reflection and `o` for a working one; on a deposition that also carries `_refln.pdbx_r_free_flag` the two agree exactly. Measured on a real merged file this run wrote: CCP4 cif2mtz refuses the file outright - "Unexpected context type for category REFLN" from its dictionary-validating parser, exit 1, a 12-byte truncated MTZ. Dropping the non-dictionary column is what fixes it: the same file without it converts. gemmi converts, but its cif2mtz spec knows only `status` and `pdbx_r_free_flag`, so FreeR_flag comes out 1 everywhere and the free set is silently lost - R-free would then be computed on the working set. phenix worked, but only by a filename heuristic matching the words "status" and "free". Writing `f` while keeping the extra column is worse than either, because phenix then finds two candidate free-set arrays and refuses the file, so the column goes in the same change. After it, all three read the same 5% test set. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 1 + docs/CPU_DATA_ANALYSIS_INTEGRATION.md | 2 +- image_analysis/WriteReflections.cpp | 7 ++++--- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a389b71d0..705aa8663 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -10,6 +10,7 @@ * `rugnux` writes a batch header in the unmerged MTZ for every image the observations span, not only for the images that produced one, so a scaling program reads one run per sweep. * `rugnux` leaves an event out of the unmerged MTZ when less of its rocking curve was captured than `--min-captured-fraction`, as the merge does, since the exported rows are declared full. * `rugnux` writes `FreeR_flag` with 0 for the test set and 1 for the working set, the CCP4 convention REFMAC5 defaults to; it was the other way round. +* `rugnux` marks the free set in the merged mmCIF as `_refln.status` = `f` instead of a `_refln.status_free` column that is not in the PDBx dictionary and that `cif2mtz` refused to read. * `rugnux -S` refuses or re-seats a fixed space group whose symmetry axes the indexed cell does not carry, not only one whose centring differs; `--mode scale` refuses it too. * The rugnux results report names the groups the data cannot separate, the enantiomorph state and any refused higher point group as `SPACE_GROUP_ALTERNATIVES`, `SPACE_GROUP_ENANTIOMORPH`, `SPACE_GROUP_REFUSED_POINT_GROUP` and `SPACE_GROUP_REFUSED_REASON`; `REPORT_VERSION` is 5. * `rugnux` handles symmetry better: the lattice, the point group, the setting and the systematic absences. diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index ff2a64c5c..ea15fa94b 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -308,7 +308,7 @@ Each batch's $B$ is fitted on **resolution-shell means**, not on single observat ### 10.7 R-free test-set flags -A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged as a **free (test) set**, written to the output (MTZ `FreeR_flag`, mmCIF `_refln.status_free`, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection's **Friedel-merged (Laue) ASU index**, which gives three properties: +A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged as a **free (test) set**, written to the output (MTZ `FreeR_flag`, mmCIF `_refln.status` = `f`, a text-HKL column) for model validation (§14) and for downstream refinement. The flag is a pure function of the reflection's **Friedel-merged (Laue) ASU index**, which gives three properties: - all symmetry- and Friedel-equivalent reflections share one flag — in particular a Bijvoet pair $I(+)/I(-)$, kept as two separate merged rows in anomalous mode, is **never split** across the work and free sets (which would bias R-free); - the free/work decision is a deterministic hash of that key, so the same reflection always lands in the same set — reproducible run-to-run and independent of the order in which observations were merged; diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index 35ecc1678..c1de19809 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -382,7 +382,6 @@ void WriteMmcifReflections(const std::vector &reflections, out << "_refln.pdbx_F_plus_sigma\n"; out << "_refln.pdbx_F_minus\n"; out << "_refln.pdbx_F_minus_sigma\n"; - out << "_refln.status_free\n"; out << "_refln.status\n"; // One row per unique reflection, twelve formatted floats each - tens of megabytes on a crowded @@ -422,8 +421,10 @@ void WriteMmcifReflections(const std::vector &reflections, column(Fmt(r.sigmaF_plus, 4), 14); column(Fmt(r.F_minus, 4), 14); column(Fmt(r.sigmaF_minus, 4), 14); - s.push_back(r.rfree_flag ? '1' : '0'); - s.append(" o\n"); // 'o' = observed + // PDBx/mmCIF _refln.status: 'f' marks the free (test) set, 'o' the working + // set. This is the only place the free set is carried in the file - cif2mtz and + // gemmi both read it, and neither knows any other reflection-level free-set item. + s.append(r.rfree_flag ? "f\n" : "o\n"); } } });