From d5cf3a6e581a6375562c2091ab4caf9f8465e553 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Wed, 2 Sep 2026 12:33:18 +0200 Subject: [PATCH] docs: worked Phaser and SHELX routes, and the MTZ base dataset they need Both programs were run on merged rugnux output and the commands are the ones that worked, error messages included. Phaser takes myrun.mtz with no LABIN, picking IMEAN/SIGIMEAN and the cell, group and resolution out of the file; MODE MR_AUTO already searches both hands of an enantiomorphic pair and returned the hand opposite the one in the header on a tetragonal test. SGALTERNATIVE SELECT ALL covers the screw variants the report lists as indistinguishable, MODE CCA prints the list without searching, and a wrong point group has to be re-merged instead. mmCIF is refused by 2.8.3 in both the CCP4 and the phenix build; gemmi cif2mtz gives the same solution. SHELXC reads only myrun.hkl - it refuses an MTZ and exits 0 while doing it - and needs CELL and SPAG repeated, since HKLF 4 carries no metadata. A default rotation merge already writes the Bijvoet split, so no flag is needed and --no-export-unmerged hides nothing. On a 5 keV cubic sweep SHELXD separated a space-group pair the merged intensities could not (CFOM 69.4 against 52.0) and SHELXE solved it, 42.9 % against 15.3 % autotrace CC between the two hands; a tetragonal sweep at lower completeness and multiplicity did not separate at all, and that is recorded too. Verifying it exposed one defect. The merged MTZ was written with its data dataset at id 0, the id MTZ reserves for HKL_base, so mtzlib dropped the wavelength and mtzinfo reported the 1.54187 A default - the wrong edge for anything taking f'/f'' from the file. Writing HKL_base first, as the unmerged writer already does, puts the real wavelength back; Phaser's solution is unchanged either way. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EFEJG6WBQv8th4UJFNe53N --- docs/CHANGELOG.md | 2 + docs/RUGNUX_INTEGRATION.md | 201 +++++++++++++++++++++++++++- image_analysis/WriteReflections.cpp | 9 +- 3 files changed, 202 insertions(+), 10 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 58f73f86a..c6c9de516 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -11,6 +11,7 @@ * `rugnux` writes a batch header in the unmerged MTZ for every image the observations span, not only for the images that produced one, so a scaling program reads one run per sweep. * `rugnux` leaves an event out of the unmerged MTZ when less of its rocking curve was captured than `--min-captured-fraction`, as the merge does, since the exported rows are declared full. * `rugnux` writes `FreeR_flag` with 0 for the test set and 1 for the working set, the CCP4 convention REFMAC5 defaults to; it was the other way round. +* `rugnux` writes the merged and P1 MTZ with the reserved `HKL_base` dataset ahead of its own, so a CCP4 program reads the wavelength from the file instead of falling back to 1.54187 A. * `rugnux` marks the free set in the merged mmCIF as `_refln.status` = `f` instead of a `_refln.status_free` column that is not in the PDBx dictionary and that `cif2mtz` refused to read. * `rugnux -S` refuses or re-seats a fixed space group whose symmetry axes the indexed cell does not carry, not only one whose centring differs; `--mode scale` refuses it too. * The rugnux results report names the groups the data cannot separate, the enantiomorph state and any refused higher point group as `SPACE_GROUP_ALTERNATIVES`, `SPACE_GROUP_ENANTIOMORPH`, `SPACE_GROUP_REFUSED_POINT_GROUP` and `SPACE_GROUP_REFUSED_REASON`; `REPORT_VERSION` is 5. @@ -34,6 +35,7 @@ * `jfjoch_viewer` reads PILATUS miniCBF sweeps natively, and draws grid scan cells in the proportion of the scan steps. * The HDF5 and API documentation say how a grid scan records the angle its spindle stood at: send the goniometer axis with a step of 0. * The rugnux manual is reorganised into task pages with a run overview, and gains worked phenix / REFMAC5 / POINTLESS-AIMLESS / careless examples. +* The rugnux manual gains worked Phaser and SHELXC/D/E examples, and says how to hand each of them the space groups the merged intensities could not separate. * `jfjoch_viewer` labels the merge-statistics plot over the range the axis is drawn on, so the CC1/2 curve is no longer read against tick labels covering only part of it. ### 1.0.0-rc.165 diff --git a/docs/RUGNUX_INTEGRATION.md b/docs/RUGNUX_INTEGRATION.md index 846c6fc2d..3033d38c6 100644 --- a/docs/RUGNUX_INTEGRATION.md +++ b/docs/RUGNUX_INTEGRATION.md @@ -49,6 +49,20 @@ expect. Two properties worth knowing before using it: untouched, but the absolute scale is not meaningful. This matters only if you intend to compare magnitudes with another file; SHELXC and ANODE use ratios alone. +**MTZ** (`.mtz`, and `_P1.mtz` beside it). The CCP4 anomalous layout, with the +column types CCP4 programs dispatch on: + +``` +H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag +H H H J Q K M K M F Q G L G L I +``` + +`F` is the French–Wilson amplitude. The header carries the determined space group, the refined cell +and the wavelength, on a dataset of its own behind the reserved `HKL_base` — so `mtzinfo` and +`mtzdmp` report the wavelength the data were collected at, and a program that takes *f′*/*f″* from +the file is given the right edge. The Bijvoet columns are present on any rotation merge, with or +without `-A`; a stills merge has no Bijvoet split and the file then stops after `F SIGF FreeR_flag`. + ## The unmerged export `_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column @@ -173,13 +187,188 @@ careless mono --anomalous "BATCH,dHKL,Hobs,Kobs,Lobs,XDET,YDET,BGMEAN,BGVAR,LP,F myrun_unmerged.mtz out/myrun ``` -**Molecular replacement.** When the report names an enantiomorphic pair (`P 41 or P 43`), let -Phaser try both hands: `SGALTERNATIVE SELECT HAND`. When section 4's candidate table shows a close -runner-up, `SGALTERNATIVE SELECT ALL` searches the alternatives, and `myrun_P1.mtz` is there for -re-merging in a subgroup when the call itself is in doubt. +**Molecular replacement and experimental phasing** each get a section of their own below — +[Phaser](#molecular-replacement-with-phaser) and [SHELXC/D/E](#experimental-phasing-with-shelx). +Both are where rugnux stops and the next program starts, and both meet the one thing the merged +intensities could not decide: which of several space groups the data are in. -**`iotbx.merging_statistics myrun_unmerged.mtz`** and **SHELXC on `myrun.hkl`** need no arguments -or label choices at all. +**`iotbx.merging_statistics myrun_unmerged.mtz`** needs no arguments or label choices at all. + +## Molecular replacement with Phaser + +rugnux does not do molecular replacement, so Phaser is the next program for anyone who has a search +model. Both CCP4 and phenix ship it — `phaser` and `phenix.phaser`, the same 2.8.3 build in the +versions this was checked against — and either takes the merged `myrun.mtz` as it is written. + +**No `LABIN`, no label choices.** Phaser reads the cell, the space group and the resolution range +out of the file and picks the intensity columns itself. Where phenix stops on a merged file because +it cannot choose between two equally usable observation arrays (see above), Phaser simply announces +what it took: + +``` + Data read from mtz file: myrun.mtz + Space-Group Name (Hall Symbol): P 41 21 2 ( P 4abw 2nw) + Unit Cell: 78.06 78.06 37.70 90.00 90.00 90.00 + Column Labels Selected: IMEAN SIGIMEAN + Resolution on Mtz file: 0.99 39.03 +``` + +So the whole run is the model and the cell contents: + +``` +phaser <` should be +about 0.80 where there is no anomalous signal. Two sweeps are quoted below, both collected at 5 keV +for the sulfur signal: a cubic one that went all the way, and a tetragonal one that did not. The +cubic one, 2.5 Å at 95 % completeness and multiplicity 30, reads: + +``` + Resl. Inf. 13.02 8.01 6.03 4.93 4.22 3.71 3.33 3.04 2.80 2.60 2.43 + 108.8 91.4 63.0 64.7 70.6 61.8 45.5 34.7 23.7 12.4 5.0 + %Complete 96.2 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 99.0 72.7 + 2.58 5.06 3.97 2.92 2.36 1.68 1.50 1.33 1.48 1.38 1.79 +``` + +**SHELXD will separate space groups the merged intensities could not.** That sweep's report named a +body-centred cubic pair as indistinguishable, so SHELXC and SHELXD were run once per candidate — +same reflections, same `FIND`, only `SPAG` different. One gave `CC 37.93 / CC(weak) 14.05 / +CFOM 51.98` and the other `CC 46.76 / CC(weak) 22.61 / CFOM 69.37`. The substructure is where the +screw axis shows itself, and the second group is the right one. This is the same handover as +Phaser's arrived at from the other side, and it is worth doing whenever `SPACE_GROUP_ALTERNATIVES` +is not `NONE` — SHELXD takes seconds, and the pair of runs costs less than reprocessing anything. + +**SHELXE decides the hand**, and says so. Run it twice, `-i` inverting the substructure. `-s` is the +solvent fraction, `-h` says the substructure atoms belong to the native structure, as sulfur does, +and `-a` turns on autotracing, which is what actually makes the two hands separate. The two runs +write `sad.pdb` and `sad_i.pdb`, so they can share a directory: + +``` +shelxe sad sad_fa -h -s0.62 -m20 -a15 -q +shelxe sad sad_fa -h -s0.62 -m20 -a15 -q -i +``` + +At 63 % solvent the two hands came out at 42.93 % and 15.28 % for the autotrace CC against the +native data — pseudo-free CC 66.49 against 37.12, map contrast 0.87 against 0.44, 215 traced atoms +— which is a solved structure, from `myrun.hkl` and nothing else. Where the group is one of the 22 +that come in enantiomorphic pairs, SHELXE makes the group change itself: the inverted run prints +`** Space group converted to enantiomorph **` and writes the changed group into the `CRYST1` of its +traced model, so the answer is readable off the output file the same way it is off Phaser's. + +**A negative result, for calibration.** A tetragonal dataset at the same wavelength with the same +kind of substructure, but 87 % complete at multiplicity 20 rather than 95 % at 30, gave a plausible +SHELXD `CFOM 47.62` and then failed at the hand: 15.33 % against 15.60 % autotrace CC, map contrast +0.33 either way. That is not a discrimination and it is not a solution. Nothing about the file was +the limit — the anomalous signal SHELXC measured on it was real, `` reaching 4.2 — so the +reading is that sulfur phasing wants the completeness and the multiplicity, and a `.hkl` from a +sweep that does not have them will get this far and no further. ## Comparing the geometry with XDS diff --git a/image_analysis/WriteReflections.cpp b/image_analysis/WriteReflections.cpp index c1de19809..8332288e4 100644 --- a/image_analysis/WriteReflections.cpp +++ b/image_analysis/WriteReflections.cpp @@ -451,7 +451,11 @@ void WriteMtzReflections(const std::vector &reflections, mtz.title = "Rugnux merged reflections"; mtz.history.push_back("From Rugnux " + jfjoch_version() + ", data reduction"); - // Add dataset + // Add dataset. HKL_base first, so the data dataset is id 1: a dataset written as id 0 is the + // reserved base dataset as far as CCP4's mtzlib is concerned, and the wavelength on it is + // dropped - mtzinfo then reports the Cu K-alpha default 1.54187 A, which is what a program that + // takes f'/f'' from the file would phase on. + mtz.add_base(); // the HKL_base dataset and the H K L columns gemmi::Mtz::Dataset& ds = mtz.add_dataset("native"); ds.crystal_name = experiment.GetSampleName(); ds.wavelength = experiment.GetWavelength_A(); @@ -463,9 +467,6 @@ void WriteMtzReflections(const std::vector &reflections, bool has_anom = true; const std::vector out_rows = BuildMergedRows(reflections, experiment, has_anom); - mtz.add_column("H", 'H', dataset_id, -1, false); - mtz.add_column("K", 'H', dataset_id, -1, false); - mtz.add_column("L", 'H', dataset_id, -1, false); mtz.add_column("IMEAN", 'J', dataset_id, -1, false); mtz.add_column("SIGIMEAN", 'Q', dataset_id, -1, false); if (has_anom) {