* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform. * Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models. * Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own. * Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log. * Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell. * jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card. * jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own. * jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob. * jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then. * Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates. * Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option. * Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes. Reviewed-on: #98 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
42 KiB
Models and reference data
What Rugnux does with outside knowledge — an atomic model, a predicted model, a reference dataset, a cell or a space group you already know — and, decision by decision, what that knowledge is and is not allowed to change. The first section is the short version; the tables after it are the precise one; the rest walks through a run step by step.
:local:
:depth: 2
In short
- Rugnux needs nothing but the images. Indexing, the cell, the space group, the resolution cut, scaling, merging and the error model all come from the data. Everything on this page is optional.
- A model (
--model,--mr) does not change how your data are processed. The images are indexed, integrated, scaled and merged the same way with or without it, and the space group is determined before the model is used. The merged intensities and their sigmas in<prefix>.mtzare the same numbers either way. - The model can only relabel, and only if the data accept it. It is first tested against
copies of itself placed in random orientations. A model that fits no better than those changes
nothing at all: the reflection files are byte for byte what a run without a model writes. A model
that does fit may change three labels in
<prefix>.mtz: the hand of the space group (P4₁2₁2 against P4₃2₁2, which the intensities cannot tell apart), the indexing where the lattice allows two equally valid ones, and the axes the files are written on. No intensity is changed, no reflection is added or removed, and no other space group is ever taken. - A different space group chosen by the model goes to a separate file. If the model prefers
another space group, or a twinned subgroup, that merge is written beside the answer as
<prefix>_mr.mtz/.cif, with the model as placed there.<prefix>.mtzstays the data's own answer. - A placed model is then refined, after every decision has been taken, and written beside the
answer as
<prefix>_refined_model.cif/.pdbwith its maps (Refinement);--no-refineskips it. The refinement changes nothing else the run writes. - A reference dataset (
-z) and a space group or cell you give (-S,-C) are different: they are taken as facts about the crystal. They fix the space group and the cell before processing, so with them the space group is yours, not the data's. - Serial stills are no exception: the model is not used before the merge there either. The
per-crystal indexing ambiguity is resolved from the data (and against
-zwhere given), see Serial stills; the model only relabels the merged reflections into its own indexing afterwards, as on rotation data.
Every option that brings in a model or reference data
| Option | What you give | What Rugnux uses it for | Can it change <prefix>.mtz? |
|---|---|---|---|
| (none) | — | Everything from the data | — |
--model <file> |
An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then refinement of the placed model (REFINE_ keys, <prefix>_refined_model.cif/.pdb, _refined_maps.mtz, _blobs.json), which decides nothing |
Labels only (hand, indexing, axes), and only where the model fits. Other space groups → <prefix>_mr.* |
--mr <file> |
The same kinds of model | As --model, but molecular replacement straight away, without the rigid body. --mode mx only, not together with --model |
Hand label only. Other space groups → <prefix>_mr.* |
| PAE file beside a prediction | AF-<id>-F1-predicted_aligned_error_v<n>.json next to AF-<id>-F1-model_v<n>.cif (read automatically, no option) |
Cuts the prediction into rigid domains for molecular replacement | No (a domain alone can only confirm the answer's group) |
-z, --reference <file> (--reference-mtz) |
Merged data of the same crystal form: an MTZ, or a PDB structure-factor mmCIF (-sf.cif) |
Fixes the space group (in its own setting) and the cell (softly), puts the merge on its axes, resolves the indexing ambiguity, hands over its R-free set, reports CCref on stills. Never a scale anchor | Yes: the space group, the axes, the indexing and the free set are the reference's |
--reference-column <label> |
An MTZ column label | Which column of the reference is read | Through -z only |
--free-flags <file> |
Merged data carrying an R-free set: an MTZ, or a structure-factor mmCIF - typically the deposited reflections of the --model |
Its R-free set and nothing else: the model is validated and refined against it, on the model's axes, where the merge matches the file's intensities (R_FREE_FLAGS= FILE) |
The free flags only |
-S, --space-group |
A space group | Fixes it: no space-group search | Yes: the group is yours (a fitting model may still relabel its hand) |
-C, --unit-cell |
A cell | Indexing looks for this cell (within tolerance); the FFT search range is widened to cover it; on stills the shared-geometry refinement anchors on it as a prior (without it, on the cell the frames index to), and warns, keeping it, where it puts materially fewer of a sample's spots on the lattice than the frames indexed de novo | Yes: the cell, and the axis order the files are written in |
--force-rotation-lattice |
Nine numbers: a lattice | Skips the first-pass indexing and uses this lattice | Yes: the lattice |
--report-resolution <dmin>[,<dmax>] |
Another program's resolution range | A second statistics table at that range (REFRES_*) |
No — report only |
Without the options. A model and a reference reflection file can also be given among the inputs, in any
order: rugnux data_master.h5 model.cif, rugnux model.pdb.gz data_master.h5 ref.mtz. Each file is told by
its contents, never its name: an MTZ (its MTZ magic) or an mmCIF with a _refln or _diffrn_refln loop
(a PDB -sf.cif) is reference data, taken as -z - in --mode refine it is the input itself; a PDB file (a
PDB record first, with an ATOM or HETATM record) or an mmCIF with an _atom_site loop is a model, taken
as --model; anything else is images, for the readers. An mmCIF holding both coordinates and reflections is
refused as ambiguous and has to be given with --model or -z. At most one model and one reference, counting
--model/--mr and -z as well; gzipped files are read through.
-S, -C, -z and --force-rotation-lattice are assertions: Rugnux takes them as true and
does not test them against a null. Use them when you know; leave them out to let the data decide
(a de-novo run is the recommended default — see Running Rugnux).
--model and --mr are hypotheses: Rugnux tests them, and what it lets them change is limited
and stated below. In --mode calibration a cell given with -C names the calibrant instead
(Detector calibration); nothing on this page applies there.
Four kinds of influence
Every decision and every output of a run falls in one of four classes. The decision table below uses these letters.
- (a) Data only. Never influenced by any model. The same with and without
--model/--mr. - (b) Model-informed label in
<prefix>.mtz. A choice between alternatives the data cannot separate and that are all equally valid descriptions of the same measurement: the hand of an enantiomorphic pair, an indexing ambiguity, the setting (axes). The model may make it only after it has passed its null; the intensities do not change. Reported asSPACE_GROUP_ENANTIOMORPH= ASSUMED_FROM_MODEL,MODEL_DECISIONS_TAKEN=,MODEL_ENANTIOMORPH_ADOPTED=,MODEL_INDEXING_OPERATOR=,SETTING_SOURCE= MODEL, and the withdrawal ofINDEXING_AMBIGUITY_OPERATORS=. - (c) Model-driven, separate files only. A choice the data could in principle make but did not
make the model's way: another space group with the same rotations (other screw axes), or the twin
hypothesis' subgroup. Written only as
<prefix>_mr.mtz/.cifwith<prefix>_mr_model.pdb/.cif(and named byMODEL_CHOICE_FILE_PREFIX=,MODEL_CHOICE_SPACE_GROUP=,MR_DECISION=). Never in<prefix>.mtz. - (d) Report-only model statistics. Numbers about the model against the data, which nothing in
the pipeline acts on:
R_WORK/R_FREE,R_MODEL_SHELL_SCALED,CC_MODEL_*,MODEL_FIT*,MODEL_PLACEMENT, theMR_*keys, the anomalous sites, the maps.
Decision by decision
"Reference" means -z. A dash means the option plays no part.
| Decision | Without anything | With -S / -C |
With -z |
With --model / --mr |
Class |
|---|---|---|---|---|---|
| Spot finding, beam centre, detector geometry | data | — | — | — | (a) |
| Indexing, lattice, cell | data (de novo) | your cell / lattice | the reference's cell, softly | — | (a) |
| Integration | data | — | — | — | (a) |
| Scaling, error model, ISa | data | — | — (never a scale anchor) | — | (a) |
| Outlier rejection, frame disposition | data | — | — | — | (a) |
| Resolution cut | data (CC1/2 fall-off) | — | — | — (CC_MODEL_CONFIRMED_TO_D_MIN is reported, never used) |
(a) |
| Point group, Laue class, centring | data (space-group search) | -S fixes it |
the reference's | — | (a) |
| Screw axes (space group within the point group) | data | -S |
the reference's | a different choice → <prefix>_mr.* only |
(a) in p.mtz, (c) beside it |
| Hand (enantiomorph) | not decidable: SPACE_GROUP_ENANTIOMORPH= UNDETERMINED |
GIVEN |
GIVEN |
relabelled to the model's where it fits, past the anomalous veto: ASSUMED_FROM_MODEL |
(b) |
| Indexing ambiguity (merohedral reindex) | arbitrary on rotation (INDEXING_AMBIGUITY_OPERATORS=); on stills with -S the crystals made consistent from the data, otherwise mixed (warning) |
— | resolved against the reference | resolved by the model's fit to the merge where its lead beats its null, rotation and stills alike. Never by molecular replacement | (b) |
| Setting (which axis is a, b, c) | standard setting | -C's axis order; -S's symbol |
the reference's | the model's, where it fits and there is no -z (SETTING_SOURCE= MODEL) |
(b) |
| Twinning check (L-test, twin laws) | data | — | — | — | (a) |
Twin hypothesis <prefix>_twin_<SG>.* |
data (when the group may have absorbed a twin law) | — | — | the model may also ask for this merge; written then only where the model chooses it | (a) / (c) |
| Anomalous signal, Friedel handling | data (-A) |
— | — | — (the anomalous map is report-only, and the veto on the hand) | (a) |
| R-free test set | hash of the index | — | the reference's flags, where it matches | — (drawn again on the model's axes if the setting moved; never chosen by the model; --free-flags puts a file's set on those axes where the rigid body placed the model) |
(a) |
Merged intensities and sigmas in p.mtz |
data | — | — | — | (a) |
p.mtz space-group label |
data | yours | the reference's | hand relabel only | (b) |
<prefix>_mr.mtz/.cif, <prefix>_mr_model.* |
not written | — | — | another group or the twin subgroup the model chose | (c) |
Maps, <prefix>_model.*, section 5 of the report |
not written | — | — | always, whatever the verdict | (d) |
Refinement, <prefix>_refined_*, <prefix>_blobs.json, REFINE_ keys |
not written | — | — | where the model was placed (not with --no-refine) |
(d) |
A run with --model, step by step
- The file is read and classified. PDB or mmCIF, by content. A prediction — ModelCIF's
quality metrics, or no cell and one B value per residue between 0 and 100 (an AlphaFold file) —
is told apart from an experimental structure (
MR_MODEL_KIND=). A model that cannot be read does not fail the run; the report saysMODEL_VALIDATION= NOT_PERFORMEDand carries a warning. - The data are processed without the model. Indexing, integration, scaling, the space-group search, the resolution cut, the error model, outlier rejection: none of them reads the model, on rotation data and serial stills alike. The only thing the model asks for here is one extra merge: the merge in a subgroup (the twin hypothesis) where the indexed lattice's group was promoted by a metric re-ask, or where the model's own group has fewer rotations than the adopted one — so that the model can be scored there. That merge is written only if the model chooses it.
- Rigid-body placement (experimental structures, not with
--mr). The model is put into the data's cell and refined as one rigid body on the working reflections; the step is kept only if it lowers R-free. A model written in another description of the lattice (other axes, other centring) is first put into the data's (MODEL_CHANGE_OF_BASIS=). - The null. The same model is turned to nine random orientations about its own centroid and
placed the same way. The real fit has to beat them by 4.51 of their standard deviations on
R-work (3σ allowing for the null being nine draws):
MODEL_FIT= ACCEPTED,MODEL_FIT_SIGMA=. OtherwiseREJECTED, and the model decides nothing. There is no fixed threshold on R (§14.5). - What an accepted model may do (class b), applied to the reflections before they are written:
- hand — where the data's group is the enantiomorph of the model's, the files take the
model's group as their label. No reflection moves and I(+)/I(-) stay as measured. Vetoed where
the anomalous difference map read at the model's atoms is inverted (warning
MODEL_HAND, raised only for a placed model). Where the group was fixed with-S(or by the reference), the model's hand replacing it is warned about (MODEL_SPACE_GROUP); - indexing — where the lattice admits alternative indexings and no
-zalready settled it, the candidate with the lowest R-free, if its lead over the runner-up beats the lead a random placement of the same model takes (MODEL_INDEXING_MARGIN_SIGMA=≥ 3). The data are reindexed into the model's frame, never the model into the data's; - setting — a model written on other axes takes the files to its setting
(
SETTING_SOURCE= MODEL), so that reflections and coordinates can be used together.
- hand — where the data's group is the enantiomorph of the model's, the files take the
model's group as their label. No reflection moves and I(+)/I(-) stay as measured. Vetoed where
the anomalous difference map read at the model's atoms is inverted (warning
- The twin hypothesis. Where a subgroup merge exists, the model is scored against it plainly
and as twinned by its law. The maps come from whichever merge the model fits better on R-free
(
MAPS_FROM=). Where the model also fits there against its null, it has chosen the subgroup, and that merge is written again as<prefix>_mr.mtz/.cif(class c). - Molecular replacement, for a prediction, with
--mr, or for an experimental structure the rigid body did not place (in the answer's merge or in the subgroup's). As many copies as the cell most probably holds are searched for in every space group the merge cannot tell apart and in the twin hypothesis' subgroup, and the choice has its own null.MR_DECISION=is one of:AGREES(the answer's group — nothing changes);ENANTIOMORPH(the answer's files take the hand label, past the same anomalous veto, elseENANTIOMORPH_VETOED);SPACE_GROUPorTWIN_SUBGROUP(written as<prefix>_mr.*only);UNDECIDEDorNOT_PLACED(nothing is written). Molecular replacement never reindexes and never setsMODEL_FIT. Not placing a model is a result, not a warning (Molecular replacement). - Validation and files. R-factors, CC(model, data), maps and the placed model are computed
for the placement that was kept, against the merge it was chosen on, and written:
<prefix>_maps.mtz(map coefficients),_model.cif/.pdb— or the<prefix>_mr_*equivalents where the choice was<prefix>_mr. A model placed nowhere (MODEL_PLACEMENT=NOT_PLACED) gets no model or maps files: its R-factors stay in the report as the negative result, but its coordinates and maps would read as a placement. A placed model is then refined (<prefix>_refined_model.cif/.pdb,_refined_maps.mtz,_blobs.json). - The reflection files —
<prefix>.mtz,.cif,.hkl,_unmerged.mtz— are written last, carrying only the class-(b) labels from step 5 and the hand from step 7.
MODEL_PLACEMENT= summarises steps 3–7: RIGID_BODY, MR or NOT_PLACED.
In --mode scale, --model is steps 3–5 only: the rigid body, its null, and the hand and indexing
it may settle. There is no molecular replacement, no setting change, no twin hypothesis and no refinement, and
--mr is refused.
Serial stills: the indexing ambiguity
On serial stills each crystal is indexed on its own, so where the lattice admits two indexings, each
crystal lands in one of them at random and a merge averages reflections that are not equivalent.
That has to be settled per crystal, before the merge. --model plays no part in it: intensities
computed from a model are never used as the per-crystal reference, so a run with a model merges
exactly what a run without one does. After the merge, the model is validated as in the steps above
and may relabel the merged reflections into its own indexing.
A stills run given -S settles the same question from the data: it
cannot say which indexing is the true one, but it can say which crystals share one, and puts each
crystal into the indexing in which it agrees best with all the others, its own cell weighed
beside that (INDEXING_AMBIGUITY_RESOLVED= data). Without -S - the space group then
determined by the run, after the merge - nothing resolves it, and the run warns. With a reference MTZ,
that same resolution runs and the reference fixes the indexing the merge is written in: AMBIGUITY_DATA_DISAGREES= in the report counts the
crystals the data and their cells put into another indexing than the reference alone would.
The model and the space group
Spelled out, because this is the question that matters most:
- The space group in
<prefix>.mtzis decided without the model — by the space-group search, or by you (-S,-z). The model is read after that decision. - The model can change one thing about that group: its hand, between the two members of an
enantiomorphic pair (P3₁/P3₂, P4₁2₁2/P4₃2₁2, P6₁22/P6₅22, …). The intensities are identical for
both, so this is the one group choice no data can make; a model that fits carries chemical
knowledge (the hand of its amino acids) the intensities do not. It is a label: no reflection
moves. It applies to a group you gave with
-Sor-ztoo. The report saysSPACE_GROUP_ENANTIOMORPH= ASSUMED_FROM_MODEL— assumed, not determined — also where the model simply agrees with the hand the run wrote. - Everything else the model prefers goes beside the answer. Other screw axes with the same
rotations (e.g. P2 against P2₁ where the absences did not decide), or the twin hypothesis'
subgroup:
<prefix>_mr.mtz/.cif. The model never changes the point group, the lattice or the centring of<prefix>.mtz.
The safeguards behind 2 and 3:
- The null. The model must beat its own random placements by 4.51 sd, rigid body and molecular
replacement alike, before it may relabel anything or have a
<prefix>_mrfile written. - The anomalous veto. The hand is taken only where the anomalous difference map, read at the
model's atoms, is not inverted. The veto can act only where the merge kept the Bijvoet split —
every rotation merge does; serial stills only with
-A. R-free cannot see the hand at all. - The indexing margin. An indexing is taken only where the model's preference beats what a randomly placed copy of it would show.
- Molecular replacement's own rules. Its choice of group has its own null (
NULL_SIGMA); a placement must pack (CLASH,MATTHEWS); a choice that translational NCS could explain, or one where the merge's own group was searched to fewer copies, is leftUNDECIDED; a single domain of a prediction cut by its PAE can only confirm the answer's group, never change it; a model of several chains is searched for as a whole and, beside it, one chain of each molecule by itself, and a chain replaces the whole only where it fits better by 0.03 in R-free or the whole was placed nowhere; in P1 the first copy's origin is fixed and it counts only on its rotation function. The details are in Molecular replacement. - The wrong-model test. The validation battery checks that a model of an unrelated protein
(similar size, unrelated sequence and fold, or the other hand) leaves the reflection data and
every report key outside section 5 identical to a run without a model, is not accepted by the
null, is not placed by molecular replacement and takes no decision ("The wrong-model null" in
tools/battery/README.md).
Refinement
Giving a model is the opt-in: wherever the model is placed - it fits where it is, or molecular replacement
placed it - rugnux refines it against the merge it was placed on; a model that is not placed is not refined.
A run that should not spend the time is given no model; --no-refine skips the refinement of a placed model
(for debugging). It takes seconds on a GPU and typically 2-15 minutes on the CPU (no GPU, or --no-gpu).
Refinement comes after every processing decision and changes none of them: R_WORK, R_FREE and every
other key outside REFINE_ stay those of the model as placed, and <prefix>.mtz, <prefix>_model.cif and
<prefix>_maps.mtz are the same as they would be without it. The broker's live mode refines an uploaded model
in the same way (see Rugnux live mode).
The built-in dictionary is a subset of the CCP4 Monomer Library compiled into rugnux: the 20 standard amino acids and MSE, the DNA/RNA nucleotides and the common modified RNA nucleotides (PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, 2MG, M2G, 7MG, 5MU), ATP/ADP/AMP/GTP/GDP/GMP, the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM and SAH, dioxygen (OXY), N-acetyl-D-glucosamine (NAG, NDG, with the glycosidic links), the photoreceptor chromophores retinal (RET, with its Schiff base to a lysine), p-coumaric acid (HC4; its thioester to the cysteine is not restrained), the GFP-type chromophores CRO, GYS, CR2 and NRQ and the bilins BLA, LBV and CYC (phycocyanobilin, with its thioether to a cysteine), DMSO, PEG fragments (PEG, PGE, PG4, 1PE), glycerol, ethylene glycol, sulfate, phosphate, acetate, MPD, water and common ions. Carbon monoxide (CMO) is left out on purpose: the library restrains its C-O bond to 1.25 A, where bound CO is near 1.15 A.
Further compounds, or other restraints for one of these, are given as restraint dictionaries: a .cif among the
inputs (recognised by its _chem_comp_atom, _chem_link or _chem_mod loop) or with --restraints, as many as
needed. A monomer-library, AceDRG, grade or eLBOW restraint file is taken as it is - its monomers
(data_comp_<CODE>), and its links (data_link_list, _chem_link_*) and modifications where it has them; an entry
of the user's replaces the built-in one of the same code, and of two files the later one wins. An entry of the
PDB Chemical Component Dictionary has ideal coordinates but no restraint values: its bonds, the angles at each atom
and a plane through each atom with a double or aromatic bond are restrained to the geometry of its ideal
coordinates (sigmas 0.02 A, 3 degrees, 0.02 A), with no torsions and no chiral volumes, and the log says so -
enough to keep a ligand in shape, not a replacement for a restraint file from AceDRG or grade. A CCD entry
without ideal coordinates is refused.
rugnux -o refined data.mtz model.cif LIG.cif # --mode refine: no images given
rugnux -o myrun dataset_master.h5 model.cif LIG.cif # the full pipeline, the model refined with it
Any residue neither dictionary describes, or with an atom its entry does not know, is held fixed: it stays in
the structure factors and the bulk-solvent mask, and the refined atoms are kept from clashing with it;
REFINE_RESIDUES_FIXED_NAMES lists them. The dictionary's hydrogens are added and ride on their parents. Metal ions, and the iron of a heme, are
restrained to the donor atoms (O, N, S) the model's LINK / _struct_conn records bind them to: by the dictionary's own Zn and Fe
links to Cys and His where they apply, otherwise at the library's typical distance for the metal, the donor
element and the number of such records of the ion (Mg-O 2.07 A, Na-O 2.42 A, K-O 2.82 A at their
usual coordination), where the model's distance is within 0.33 A of that (Mg-O up to 2.4 A); a record at an
outer-sphere distance, a contact without a record, or one to a symmetry mate, is not restrained as a bond. The B factors of bonded atoms and of
atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of
the X-ray term against the geometry and against that B restraint are chosen on the free set among short
refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free
-ln L is lower and its R-free not higher than the incumbent's by more than the standard error of their paired
difference over the free reflections: three weights from the placed
model, each with the B factors held and refined; then a trajectory of further short refinements from the
chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the
refined target at the chosen weights); then a short final ladder. The free set decides the weights, the
displacement model, the twin law against its null and whether the model as given is kept over the refinement;
every other step is taken on the work-set target
(where the free set is used tabulates them).
The free set is the merge's own, so the refined R-free judges the refinement against reflections it
never fitted - but where the model was deposited, those reflections were most likely its depositor's working
reflections, and a refinement against new data then moves R-free up from the unrefined value rather than
down. R_WORK/R_FREE of the model as given are never replaced. On a GPU a refinement takes seconds; the CPU
engine computes the same and is several times slower.
How a model is scored. R-work, R-free and the likelihood target of every stage - the model as given
(REFINE_R_WORK_START) as much as the refined one - come from one model of the structure factors,
Fmodel = k exp(-sTBs/4) (Fcalc + ksol exp(-Bsols2/4) Fmask):
the atoms with the dictionary's riding hydrogens and their B made isotropic; a flat bulk solvent masked with the
REFMAC radii (probe 1.0 A, shrink 0.8 A) on a grid of its own, at most 0.4 A apart, so that the shrink acts at any
resolution; ksol and Bsol searched over the physical range on the working reflections, and the
overall scale and anisotropic B fitted to them by robust least squares, so that a few reflections the model cannot
explain (a beam-stop shadow, an ice ring, a missing part of the model) do not set the scale of the rest. On depositions
from 1.1 to 3.6 A this gives REFMAC 5.8's 0-cycle R-work and R-free on the same model and data within 0.006
(tools/battery/refine_parity.py runs the comparison; --refine-score-only prints each term under REFINE_SCORE_,
see --mode refine).
The algorithms - the target and its gradient, the likelihood's parameters, the restraints, the minimiser and the weight trials - are described in §15, and the keys in The results report.
What it does not do yet. The refinement is a first, automatic pass on a placed model, not a replacement for a refinement program. Not implemented yet:
- restraints for a new ligand from its chemistry alone: without a restraint file (or a CCD entry's ideal coordinates) a residue the built-in dictionary lacks is held fixed;
- water building and ion relabelling by default: they are opt-in (
--refine-solvent); without it the blobs only point at candidates; - occupancy refinement and new alternative conformations (existing ones are refined in place);
- rotamer and peptide-flip search, Ramachandran restraints;
- real-space refinement, jelly-body and reference-model restraints;
- a full Metal (Apple GPU) path: on macOS only the model density is put on the grid on the GPU, the rest of the refinement runs on the CPU;
- validation beyond the geometry's rmsd and clash count: no MolProbity-style report.
TLS groups, anisotropic U (an ANISOU record is kept as the atom's own U), NCS restraints and twin refinement are implemented and decided by the weight trials, see §15.
Refining a model against structure factors (rugnux --mode refine)
--mode refine refines a model against merged structure factors that already exist - processed elsewhere, a
deposition's, or a rugnux .mtz - and processes nothing:
rugnux --mode refine --model model.cif -o refined data.mtz
A model and reflections given among the inputs with no images, and no --mode, are this mode too
(rugnux -o refined data.mtz model.cif); restraint dictionaries for ligands are given beside them
(Refinement).
The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by
content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its
own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The
columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and
map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN,
I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from
a FreeR_flag-like column (flag 0 is the test set; where 0 marks the majority, as phenix and CNS write it, the
minority value is; in a structure-factor mmCIF, _refln.status 'f' first); a file without one gets rugnux's own 5 % set.
Where a structure-factor mmCIF's flags are classes (_refln.pdbx_r_free_flag 0-19), the class the file marks is not
always the one the depositors held out, so the model as given is scored on each class: the mark is kept where the model
fits that class worst or the test-set count the model file records (REMARK 3, _refine.ls_number_reflns_R_free) is
nearest its count; otherwise the class the model fits worst is the test set where that count is nearest it, or where
it is above the marked class by more than twice the scatter of the other classes' R (REFINE_FREE_SET_SOURCE= STATUS|REMARK3_COUNT|WORST_FIT, REFINE_FREE_SET_CLASS). The refinement is the one run after a merge on a placed model, and
writes the same files - <prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz, <prefix>_blobs.json - with
the REFINE_ keys in <prefix>_report.txt, beside REFINE_FRENCH_WILSON= and REFINE_FREE_FLAGS= FILE|GENERATED
saying what had to be made of the input.
| Option | Description |
|---|---|
--model <file> |
The model (required; it can also be given among the inputs, told by content) |
--refine-score-only |
Score the model as given and refine nothing; every term of the score - R-work and R-free overall and in 20 resolution shells, the overall scale, B and anisotropic B, ksol and Bsol, the likelihood over the working and the free reflections and its figure of merit over the working reflections, and the geometry's rmsd and rmsZ - under REFINE_SCORE_ keys (--developer prints them all). The twin test is the refinement's: the model is scored against the data twinned by each law too, and the twinned score taken by the same rule (REFINE_TWIN) |
--refine-d-min <A> |
Leave out reflections finer than this |
Free set and model bias
How the free set is made. Rugnux draws its own test set (5 %, raised towards about 500 reflections on small data but never above 10 %):
each reflection's flag is a hash of its index, taken over its orbit under the lattice's metric
symmetry, so symmetry mates, Friedel mates and twin-law mates share a flag, every dataset of one
crystal form written on the same axes gets the same set, and the set does not depend on the space
group a file is merged in
(R-free test-set flags). The model
plays no part in it. Where a fitting model moves the files to its setting, the set is drawn again on
the new axes by the same rule; an indexing-ambiguity reindex leaves it unchanged, since twin-law
mates share a flag. With -z, the reference's flags are inherited where the merge matches it
(REFERENCE_FREE_FLAGS_INHERITED=).
A model refined against these reflections. A set drawn here is free only for a model that was never refined
against the crystal's reflections. A deposited model, or one refined against an earlier dataset of the same
crystal, has been fitted to every reflection, so its R-free on a fresh set reads close to its R-work - below it,
sometimes - and a refinement from it reads no cross-validation at all. --free-flags <file> takes the test set of
the reflections it was refined against (the deposited -sf.cif, or an MTZ) and nothing else from them: unlike
-z it fixes no group, cell or frame and steers no processing. The flags are put on the merge once it is on the
model's axes, where the rigid body placed the model (a model placed by molecular replacement, or a prediction,
keeps the drawn set) and the merge matches the file's intensities as -z requires (CC at least 0.5 over at least
half of its reflections); the validation is then made again on them, and the refinement and the merged files carry
them. R_FREE_FLAGS= says which set R_FREE and the refinement used: DRAWN, REFERENCE (-z) or FILE.
What is fitted on which reflections. Everything fitted to place and scale the model — the
overall scale, the anisotropic B, the bulk solvent, the rigid body — is fitted on the working
reflections only. The free set is then used for a few discrete selections: whether the rigid-body
step is kept, which indexing wins, which merge the maps come from where a twin hypothesis was written, and in
molecular replacement which placement and which group
(where the free set is used lists every one).
Each of those leaves R-free very slightly optimistic where it fires. For R_WORK/R_FREE the
coordinates, B-factors and occupancies of the model are never refined; the refinement above refines them on
the side, under REFINE_ keys of its own.
What R_FREE means here — and what it does not. It is the model as placed, scored on Rugnux's
free set: a check that the model fits these data, and a trend number between datasets of one
campaign. It is not a refinement R-free, and it is not cross-validated against the model's own
history:
- A deposited model was refined by its depositor against their working set. Rugnux's free set is a different 5 %, so about 95 % of Rugnux's free reflections were in the working set the model was refined on. R-free here therefore reads close to R-work, and it is optimistic in the usual model-bias sense.
- To score a deposited model on its own test set, give its structure factors as the reference
(
-z 1abc-sf.cif): the depositor's free flags are then inherited. But-zis an assertion — it also fixes the space group and the cell — so that is no longer a de-novo run. - A predicted model has not been refined against any diffraction data, so it carries no such memory; what remains is the selection effect above.
- For comparing two runs against one model, read
R_MODEL_SHELL_SCALED, notR_FREE(Comparing two runs).
What cannot leak, and what can. The model never enters the merged intensities, their sigmas, the
choice of reflections, the resolution cut or the weights, so <prefix>.mtz carries no model bias in
its numbers — only the class-(b) labels. The maps are model-phased, and carry the model bias any
model-phased map carries: read the mFo−DFc map, and treat density that only the 2mFo−DFc map shows
with the usual care. CC(model, data) is read in one direction only: a confirmed shell is evidence for
the data, a weak shell says nothing about them, and nothing acts on it.
Practical guidance
When to give a model.
- You have a structure of this crystal form (a fragment campaign, a ligand soak, a reference
structure): give it with
--model. You get maps, a placed model and Rugnux's own first refinement of it, the indexing of every dataset brought into the model's, and the hand fixed — the things that make a campaign's files directly comparable. A wrong or rejected model costs a few seconds and changes nothing. - You have a structure of the same protein in another crystal form:
--modeltoo. The rigid body will not place it, and molecular replacement is tried automatically. - You have only a prediction (AlphaFold, ModelCIF):
--model(or--mr). Download the PAE JSON with it and keep its AlphaFold DB file name, so it is cut into domains when the whole model does not place. A prediction is good for: confirming the space group, naming the hand where the data cannot, offering a twinned or lower-symmetry alternative where the data are ambiguous, and a first set of maps. It is not a refined model, and its R-factors read accordingly. - You have a previous dataset of the same crystal form and want the same indexing and free set:
-z, not--model. It is the stronger tool for the indexing ambiguity and the only one that carries a free set over, but it pins the space group and cell. - You want to compare with another program's statistics:
--report-resolution, which changes nothing else.
Which file to use for what.
| File | What it is | Use it for |
|---|---|---|
<prefix>.mtz / .cif / .hkl |
The answer: the data's space group, with only the class-(b) labels from the model | Refinement, deposition, everything by default |
<prefix>_model.cif / .pdb, <prefix>_maps.mtz |
The model as placed, in the frame of <prefix>.mtz, and its map coefficients (Coot/PyMOL open them as maps); only where the model was placed (MODEL_PLACEMENT= RIGID_BODY or MR) |
Starting refinement (REFMAC5, phenix.refine), PanDDA, Coot |
<prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz, <prefix>_blobs.json |
The placed model refined by rugnux, its map coefficients (with the anomalous difference map's, ANOM/PHANOM, where the merge kept the Bijvoet pairs, as in <prefix>_maps.mtz), and the unexplained blobs of its difference map |
A first look at the refined density and at what the model leaves unexplained; continuing refinement |
<prefix>_twin_<SG>.mtz / .cif |
The same data merged in a subgroup with a twin law — written from the data, model or not | Twin refinement, where you suspect the higher group absorbed a twin law |
<prefix>_mr.mtz / .cif, <prefix>_mr_model.pdb / .cif, <prefix>_mr_* maps |
The merge in the space group the model chose (MODEL_CHOICE_SPACE_GROUP=), and the model placed there |
Refinement if you accept the model's choice; compare R-free against the answer's |
<prefix>_mr and <prefix>_twin_<SG> can describe the same subgroup; the _mr file then also
carries the model placed in it.
Reading section 5 of the report, in order:
MODEL_VALIDATION=— did it run? IfNOT_PERFORMED,MODEL_VALIDATION_REASON=says why.MODEL_PLACEMENT=— how the model was placed:RIGID_BODY,MRorNOT_PLACED.MODEL_FIT=,MODEL_FIT_SIGMA=— the rigid body's verdict against its null. Absent where the rigid body was not tried (a prediction,--mr).MODEL_DECISIONS_TAKEN=—NONEmeans the reflection files are what a run without a model writes. OtherwiseENANTIOMORPH,INDEXING, both, orMR_ENANTIOMORPH, withMODEL_ENANTIOMORPH_ADOPTED=andMODEL_INDEXING_OPERATOR=.MODEL_CHOICE_FILE_PREFIX=,MODEL_CHOICE_SPACE_GROUP=,MR_DECISION=— whether a<prefix>_mrfile was written, and in which group.R_WORK=,R_FREE=,CC_MODEL_CONFIRMED_TO_D_MIN=, the anomalous sites — how well the placed model describes the data (class d).
Outside section 5, SPACE_GROUP_ENANTIOMORPH= (UNDETERMINED, GIVEN or ASSUMED_FROM_MODEL),
SETTING_SOURCE= and the presence or absence of INDEXING_AMBIGUITY_OPERATORS= say what the model
did to the answer's labels. Every key is described in The results report.
Related pages
- Advanced Rugnux — what
-zdoes, the indexing ambiguity, the setting the files are written in, and validating against a model. - The results report — model validation, molecular replacement, the twin hypothesis and the reference-range table.
- Data analysis: space group and validation — the space-group search (§13) and the model-validation algorithms (§14).
- Reference data and R-free test-set flags.