Files
Jungfraujoch/docs/RUGNUX_MODEL_AND_REFERENCE.md
leonarski_f 4809c1622c
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
v1.0.0-rc.187 (#98)
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform.
* Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models.
* Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own.
* Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log.
* Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell.
* jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card.
* jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own.
* jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob.
* jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then.
* Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates.
* Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option.
* Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes.

Reviewed-on: #98
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-19 20:49:28 +01:00

42 KiB

Models and reference data

What Rugnux does with outside knowledge — an atomic model, a predicted model, a reference dataset, a cell or a space group you already know — and, decision by decision, what that knowledge is and is not allowed to change. The first section is the short version; the tables after it are the precise one; the rest walks through a run step by step.

:local:
:depth: 2

In short

  • Rugnux needs nothing but the images. Indexing, the cell, the space group, the resolution cut, scaling, merging and the error model all come from the data. Everything on this page is optional.
  • A model (--model, --mr) does not change how your data are processed. The images are indexed, integrated, scaled and merged the same way with or without it, and the space group is determined before the model is used. The merged intensities and their sigmas in <prefix>.mtz are the same numbers either way.
  • The model can only relabel, and only if the data accept it. It is first tested against copies of itself placed in random orientations. A model that fits no better than those changes nothing at all: the reflection files are byte for byte what a run without a model writes. A model that does fit may change three labels in <prefix>.mtz: the hand of the space group (P4₁2₁2 against P4₃2₁2, which the intensities cannot tell apart), the indexing where the lattice allows two equally valid ones, and the axes the files are written on. No intensity is changed, no reflection is added or removed, and no other space group is ever taken.
  • A different space group chosen by the model goes to a separate file. If the model prefers another space group, or a twinned subgroup, that merge is written beside the answer as <prefix>_mr.mtz/.cif, with the model as placed there. <prefix>.mtz stays the data's own answer.
  • A placed model is then refined, after every decision has been taken, and written beside the answer as <prefix>_refined_model.cif/.pdb with its maps (Refinement); --no-refine skips it. The refinement changes nothing else the run writes.
  • A reference dataset (-z) and a space group or cell you give (-S, -C) are different: they are taken as facts about the crystal. They fix the space group and the cell before processing, so with them the space group is yours, not the data's.
  • Serial stills are no exception: the model is not used before the merge there either. The per-crystal indexing ambiguity is resolved from the data (and against -z where given), see Serial stills; the model only relabels the merged reflections into its own indexing afterwards, as on rotation data.

Every option that brings in a model or reference data

Option What you give What Rugnux uses it for Can it change <prefix>.mtz?
(none) — Everything from the data —
--model <file> An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then refinement of the placed model (REFINE_ keys, <prefix>_refined_model.cif/.pdb, _refined_maps.mtz, _blobs.json), which decides nothing Labels only (hand, indexing, axes), and only where the model fits. Other space groups → <prefix>_mr.*
--mr <file> The same kinds of model As --model, but molecular replacement straight away, without the rigid body. --mode mx only, not together with --model Hand label only. Other space groups → <prefix>_mr.*
PAE file beside a prediction AF-<id>-F1-predicted_aligned_error_v<n>.json next to AF-<id>-F1-model_v<n>.cif (read automatically, no option) Cuts the prediction into rigid domains for molecular replacement No (a domain alone can only confirm the answer's group)
-z, --reference <file> (--reference-mtz) Merged data of the same crystal form: an MTZ, or a PDB structure-factor mmCIF (-sf.cif) Fixes the space group (in its own setting) and the cell (softly), puts the merge on its axes, resolves the indexing ambiguity, hands over its R-free set, reports CCref on stills. Never a scale anchor Yes: the space group, the axes, the indexing and the free set are the reference's
--reference-column <label> An MTZ column label Which column of the reference is read Through -z only
--free-flags <file> Merged data carrying an R-free set: an MTZ, or a structure-factor mmCIF - typically the deposited reflections of the --model Its R-free set and nothing else: the model is validated and refined against it, on the model's axes, where the merge matches the file's intensities (R_FREE_FLAGS= FILE) The free flags only
-S, --space-group A space group Fixes it: no space-group search Yes: the group is yours (a fitting model may still relabel its hand)
-C, --unit-cell A cell Indexing looks for this cell (within tolerance); the FFT search range is widened to cover it; on stills the shared-geometry refinement anchors on it as a prior (without it, on the cell the frames index to), and warns, keeping it, where it puts materially fewer of a sample's spots on the lattice than the frames indexed de novo Yes: the cell, and the axis order the files are written in
--force-rotation-lattice Nine numbers: a lattice Skips the first-pass indexing and uses this lattice Yes: the lattice
--report-resolution <dmin>[,<dmax>] Another program's resolution range A second statistics table at that range (REFRES_*) No — report only

Without the options. A model and a reference reflection file can also be given among the inputs, in any order: rugnux data_master.h5 model.cif, rugnux model.pdb.gz data_master.h5 ref.mtz. Each file is told by its contents, never its name: an MTZ (its MTZ magic) or an mmCIF with a _refln or _diffrn_refln loop (a PDB -sf.cif) is reference data, taken as -z - in --mode refine it is the input itself; a PDB file (a PDB record first, with an ATOM or HETATM record) or an mmCIF with an _atom_site loop is a model, taken as --model; anything else is images, for the readers. An mmCIF holding both coordinates and reflections is refused as ambiguous and has to be given with --model or -z. At most one model and one reference, counting --model/--mr and -z as well; gzipped files are read through.

-S, -C, -z and --force-rotation-lattice are assertions: Rugnux takes them as true and does not test them against a null. Use them when you know; leave them out to let the data decide (a de-novo run is the recommended default — see Running Rugnux). --model and --mr are hypotheses: Rugnux tests them, and what it lets them change is limited and stated below. In --mode calibration a cell given with -C names the calibrant instead (Detector calibration); nothing on this page applies there.

Four kinds of influence

Every decision and every output of a run falls in one of four classes. The decision table below uses these letters.

  • (a) Data only. Never influenced by any model. The same with and without --model/--mr.
  • (b) Model-informed label in <prefix>.mtz. A choice between alternatives the data cannot separate and that are all equally valid descriptions of the same measurement: the hand of an enantiomorphic pair, an indexing ambiguity, the setting (axes). The model may make it only after it has passed its null; the intensities do not change. Reported as SPACE_GROUP_ENANTIOMORPH= ASSUMED_FROM_MODEL, MODEL_DECISIONS_TAKEN=, MODEL_ENANTIOMORPH_ADOPTED=, MODEL_INDEXING_OPERATOR=, SETTING_SOURCE= MODEL, and the withdrawal of INDEXING_AMBIGUITY_OPERATORS=.
  • (c) Model-driven, separate files only. A choice the data could in principle make but did not make the model's way: another space group with the same rotations (other screw axes), or the twin hypothesis' subgroup. Written only as <prefix>_mr.mtz/.cif with <prefix>_mr_model.pdb/.cif (and named by MODEL_CHOICE_FILE_PREFIX=, MODEL_CHOICE_SPACE_GROUP=, MR_DECISION=). Never in <prefix>.mtz.
  • (d) Report-only model statistics. Numbers about the model against the data, which nothing in the pipeline acts on: R_WORK/R_FREE, R_MODEL_SHELL_SCALED, CC_MODEL_*, MODEL_FIT*, MODEL_PLACEMENT, the MR_* keys, the anomalous sites, the maps.

Decision by decision

"Reference" means -z. A dash means the option plays no part.

Decision Without anything With -S / -C With -z With --model / --mr Class
Spot finding, beam centre, detector geometry data — — — (a)
Indexing, lattice, cell data (de novo) your cell / lattice the reference's cell, softly — (a)
Integration data — — — (a)
Scaling, error model, ISa data — — (never a scale anchor) — (a)
Outlier rejection, frame disposition data — — — (a)
Resolution cut data (CC1/2 fall-off) — — — (CC_MODEL_CONFIRMED_TO_D_MIN is reported, never used) (a)
Point group, Laue class, centring data (space-group search) -S fixes it the reference's — (a)
Screw axes (space group within the point group) data -S the reference's a different choice → <prefix>_mr.* only (a) in p.mtz, (c) beside it
Hand (enantiomorph) not decidable: SPACE_GROUP_ENANTIOMORPH= UNDETERMINED GIVEN GIVEN relabelled to the model's where it fits, past the anomalous veto: ASSUMED_FROM_MODEL (b)
Indexing ambiguity (merohedral reindex) arbitrary on rotation (INDEXING_AMBIGUITY_OPERATORS=); on stills with -S the crystals made consistent from the data, otherwise mixed (warning) — resolved against the reference resolved by the model's fit to the merge where its lead beats its null, rotation and stills alike. Never by molecular replacement (b)
Setting (which axis is a, b, c) standard setting -C's axis order; -S's symbol the reference's the model's, where it fits and there is no -z (SETTING_SOURCE= MODEL) (b)
Twinning check (L-test, twin laws) data — — — (a)
Twin hypothesis <prefix>_twin_<SG>.* data (when the group may have absorbed a twin law) — — the model may also ask for this merge; written then only where the model chooses it (a) / (c)
Anomalous signal, Friedel handling data (-A) — — — (the anomalous map is report-only, and the veto on the hand) (a)
R-free test set hash of the index — the reference's flags, where it matches — (drawn again on the model's axes if the setting moved; never chosen by the model; --free-flags puts a file's set on those axes where the rigid body placed the model) (a)
Merged intensities and sigmas in p.mtz data — — — (a)
p.mtz space-group label data yours the reference's hand relabel only (b)
<prefix>_mr.mtz/.cif, <prefix>_mr_model.* not written — — another group or the twin subgroup the model chose (c)
Maps, <prefix>_model.*, section 5 of the report not written — — always, whatever the verdict (d)
Refinement, <prefix>_refined_*, <prefix>_blobs.json, REFINE_ keys not written — — where the model was placed (not with --no-refine) (d)

A run with --model, step by step

  1. The file is read and classified. PDB or mmCIF, by content. A prediction — ModelCIF's quality metrics, or no cell and one B value per residue between 0 and 100 (an AlphaFold file) — is told apart from an experimental structure (MR_MODEL_KIND=). A model that cannot be read does not fail the run; the report says MODEL_VALIDATION= NOT_PERFORMED and carries a warning.
  2. The data are processed without the model. Indexing, integration, scaling, the space-group search, the resolution cut, the error model, outlier rejection: none of them reads the model, on rotation data and serial stills alike. The only thing the model asks for here is one extra merge: the merge in a subgroup (the twin hypothesis) where the indexed lattice's group was promoted by a metric re-ask, or where the model's own group has fewer rotations than the adopted one — so that the model can be scored there. That merge is written only if the model chooses it.
  3. Rigid-body placement (experimental structures, not with --mr). The model is put into the data's cell and refined as one rigid body on the working reflections; the step is kept only if it lowers R-free. A model written in another description of the lattice (other axes, other centring) is first put into the data's (MODEL_CHANGE_OF_BASIS=).
  4. The null. The same model is turned to nine random orientations about its own centroid and placed the same way. The real fit has to beat them by 4.51 of their standard deviations on R-work (3σ allowing for the null being nine draws): MODEL_FIT= ACCEPTED, MODEL_FIT_SIGMA=. Otherwise REJECTED, and the model decides nothing. There is no fixed threshold on R (§14.5).
  5. What an accepted model may do (class b), applied to the reflections before they are written:
    • hand — where the data's group is the enantiomorph of the model's, the files take the model's group as their label. No reflection moves and I(+)/I(-) stay as measured. Vetoed where the anomalous difference map read at the model's atoms is inverted (warning MODEL_HAND, raised only for a placed model). Where the group was fixed with -S (or by the reference), the model's hand replacing it is warned about (MODEL_SPACE_GROUP);
    • indexing — where the lattice admits alternative indexings and no -z already settled it, the candidate with the lowest R-free, if its lead over the runner-up beats the lead a random placement of the same model takes (MODEL_INDEXING_MARGIN_SIGMA= ≥ 3). The data are reindexed into the model's frame, never the model into the data's;
    • setting — a model written on other axes takes the files to its setting (SETTING_SOURCE= MODEL), so that reflections and coordinates can be used together.
  6. The twin hypothesis. Where a subgroup merge exists, the model is scored against it plainly and as twinned by its law. The maps come from whichever merge the model fits better on R-free (MAPS_FROM=). Where the model also fits there against its null, it has chosen the subgroup, and that merge is written again as <prefix>_mr.mtz/.cif (class c).
  7. Molecular replacement, for a prediction, with --mr, or for an experimental structure the rigid body did not place (in the answer's merge or in the subgroup's). As many copies as the cell most probably holds are searched for in every space group the merge cannot tell apart and in the twin hypothesis' subgroup, and the choice has its own null. MR_DECISION= is one of: AGREES (the answer's group — nothing changes); ENANTIOMORPH (the answer's files take the hand label, past the same anomalous veto, else ENANTIOMORPH_VETOED); SPACE_GROUP or TWIN_SUBGROUP (written as <prefix>_mr.* only); UNDECIDED or NOT_PLACED (nothing is written). Molecular replacement never reindexes and never sets MODEL_FIT. Not placing a model is a result, not a warning (Molecular replacement).
  8. Validation and files. R-factors, CC(model, data), maps and the placed model are computed for the placement that was kept, against the merge it was chosen on, and written: <prefix>_maps.mtz (map coefficients), _model.cif/.pdb — or the <prefix>_mr_* equivalents where the choice was <prefix>_mr. A model placed nowhere (MODEL_PLACEMENT=NOT_PLACED) gets no model or maps files: its R-factors stay in the report as the negative result, but its coordinates and maps would read as a placement. A placed model is then refined (<prefix>_refined_model.cif/.pdb, _refined_maps.mtz, _blobs.json).
  9. The reflection files — <prefix>.mtz, .cif, .hkl, _unmerged.mtz — are written last, carrying only the class-(b) labels from step 5 and the hand from step 7.

MODEL_PLACEMENT= summarises steps 3–7: RIGID_BODY, MR or NOT_PLACED.

In --mode scale, --model is steps 3–5 only: the rigid body, its null, and the hand and indexing it may settle. There is no molecular replacement, no setting change, no twin hypothesis and no refinement, and --mr is refused.

Serial stills: the indexing ambiguity

On serial stills each crystal is indexed on its own, so where the lattice admits two indexings, each crystal lands in one of them at random and a merge averages reflections that are not equivalent. That has to be settled per crystal, before the merge. --model plays no part in it: intensities computed from a model are never used as the per-crystal reference, so a run with a model merges exactly what a run without one does. After the merge, the model is validated as in the steps above and may relabel the merged reflections into its own indexing.

A stills run given -S settles the same question from the data: it cannot say which indexing is the true one, but it can say which crystals share one, and puts each crystal into the indexing in which it agrees best with all the others, its own cell weighed beside that (INDEXING_AMBIGUITY_RESOLVED= data). Without -S - the space group then determined by the run, after the merge - nothing resolves it, and the run warns. With a reference MTZ, that same resolution runs and the reference fixes the indexing the merge is written in: AMBIGUITY_DATA_DISAGREES= in the report counts the crystals the data and their cells put into another indexing than the reference alone would.

The model and the space group

Spelled out, because this is the question that matters most:

  1. The space group in <prefix>.mtz is decided without the model — by the space-group search, or by you (-S, -z). The model is read after that decision.
  2. The model can change one thing about that group: its hand, between the two members of an enantiomorphic pair (P3₁/P3₂, P4₁2₁2/P4₃2₁2, P6₁22/P6₅22, …). The intensities are identical for both, so this is the one group choice no data can make; a model that fits carries chemical knowledge (the hand of its amino acids) the intensities do not. It is a label: no reflection moves. It applies to a group you gave with -S or -z too. The report says SPACE_GROUP_ENANTIOMORPH= ASSUMED_FROM_MODEL — assumed, not determined — also where the model simply agrees with the hand the run wrote.
  3. Everything else the model prefers goes beside the answer. Other screw axes with the same rotations (e.g. P2 against P2₁ where the absences did not decide), or the twin hypothesis' subgroup: <prefix>_mr.mtz/.cif. The model never changes the point group, the lattice or the centring of <prefix>.mtz.

The safeguards behind 2 and 3:

  • The null. The model must beat its own random placements by 4.51 sd, rigid body and molecular replacement alike, before it may relabel anything or have a <prefix>_mr file written.
  • The anomalous veto. The hand is taken only where the anomalous difference map, read at the model's atoms, is not inverted. The veto can act only where the merge kept the Bijvoet split — every rotation merge does; serial stills only with -A. R-free cannot see the hand at all.
  • The indexing margin. An indexing is taken only where the model's preference beats what a randomly placed copy of it would show.
  • Molecular replacement's own rules. Its choice of group has its own null (NULL_SIGMA); a placement must pack (CLASH, MATTHEWS); a choice that translational NCS could explain, or one where the merge's own group was searched to fewer copies, is left UNDECIDED; a single domain of a prediction cut by its PAE can only confirm the answer's group, never change it; a model of several chains is searched for as a whole and, beside it, one chain of each molecule by itself, and a chain replaces the whole only where it fits better by 0.03 in R-free or the whole was placed nowhere; in P1 the first copy's origin is fixed and it counts only on its rotation function. The details are in Molecular replacement.
  • The wrong-model test. The validation battery checks that a model of an unrelated protein (similar size, unrelated sequence and fold, or the other hand) leaves the reflection data and every report key outside section 5 identical to a run without a model, is not accepted by the null, is not placed by molecular replacement and takes no decision ("The wrong-model null" in tools/battery/README.md).

Refinement

Giving a model is the opt-in: wherever the model is placed - it fits where it is, or molecular replacement placed it - rugnux refines it against the merge it was placed on; a model that is not placed is not refined. A run that should not spend the time is given no model; --no-refine skips the refinement of a placed model (for debugging). It takes seconds on a GPU and typically 2-15 minutes on the CPU (no GPU, or --no-gpu). Refinement comes after every processing decision and changes none of them: R_WORK, R_FREE and every other key outside REFINE_ stay those of the model as placed, and <prefix>.mtz, <prefix>_model.cif and <prefix>_maps.mtz are the same as they would be without it. The broker's live mode refines an uploaded model in the same way (see Rugnux live mode).

The built-in dictionary is a subset of the CCP4 Monomer Library compiled into rugnux: the 20 standard amino acids and MSE, the DNA/RNA nucleotides and the common modified RNA nucleotides (PSU, 5MC, 1MA, OMG, OMC, OMU, H2U, 2MG, M2G, 7MG, 5MU), ATP/ADP/AMP/GTP/GDP/GMP, the cofactors NAD, NAI, NAP, NDP, FAD, FMN, HEM, HEC, SAM and SAH, dioxygen (OXY), N-acetyl-D-glucosamine (NAG, NDG, with the glycosidic links), the photoreceptor chromophores retinal (RET, with its Schiff base to a lysine), p-coumaric acid (HC4; its thioester to the cysteine is not restrained), the GFP-type chromophores CRO, GYS, CR2 and NRQ and the bilins BLA, LBV and CYC (phycocyanobilin, with its thioether to a cysteine), DMSO, PEG fragments (PEG, PGE, PG4, 1PE), glycerol, ethylene glycol, sulfate, phosphate, acetate, MPD, water and common ions. Carbon monoxide (CMO) is left out on purpose: the library restrains its C-O bond to 1.25 A, where bound CO is near 1.15 A.

Further compounds, or other restraints for one of these, are given as restraint dictionaries: a .cif among the inputs (recognised by its _chem_comp_atom, _chem_link or _chem_mod loop) or with --restraints, as many as needed. A monomer-library, AceDRG, grade or eLBOW restraint file is taken as it is - its monomers (data_comp_<CODE>), and its links (data_link_list, _chem_link_*) and modifications where it has them; an entry of the user's replaces the built-in one of the same code, and of two files the later one wins. An entry of the PDB Chemical Component Dictionary has ideal coordinates but no restraint values: its bonds, the angles at each atom and a plane through each atom with a double or aromatic bond are restrained to the geometry of its ideal coordinates (sigmas 0.02 A, 3 degrees, 0.02 A), with no torsions and no chiral volumes, and the log says so - enough to keep a ligand in shape, not a replacement for a restraint file from AceDRG or grade. A CCD entry without ideal coordinates is refused.

rugnux -o refined data.mtz model.cif LIG.cif          # --mode refine: no images given
rugnux -o myrun dataset_master.h5 model.cif LIG.cif   # the full pipeline, the model refined with it

Any residue neither dictionary describes, or with an atom its entry does not know, is held fixed: it stays in the structure factors and the bulk-solvent mask, and the refined atoms are kept from clashing with it; REFINE_RESIDUES_FIXED_NAMES lists them. The dictionary's hydrogens are added and ride on their parents. Metal ions, and the iron of a heme, are restrained to the donor atoms (O, N, S) the model's LINK / _struct_conn records bind them to: by the dictionary's own Zn and Fe links to Cys and His where they apply, otherwise at the library's typical distance for the metal, the donor element and the number of such records of the ion (Mg-O 2.07 A, Na-O 2.42 A, K-O 2.82 A at their usual coordination), where the model's distance is within 0.33 A of that (Mg-O up to 2.4 A); a record at an outer-sphere distance, a contact without a record, or one to a symmetry mate, is not restrained as a bond. The B factors of bonded atoms and of atoms in contact (waters included) are restrained to each other in proportion to their size. The weights of the X-ray term against the geometry and against that B restraint are chosen on the free set among short refinements whose bond and angle rmsd stay within 0.02 A and 2 degrees - a candidate taken only where its free -ln L is lower and its R-free not higher than the incumbent's by more than the standard error of their paired difference over the free reflections: three weights from the placed model, each with the B factors held and refined; then a trajectory of further short refinements from the chosen one, each trying the weights as they are and changed, taken while it lowers the work-set target (the refined target at the chosen weights); then a short final ladder. The free set decides the weights, the displacement model, the twin law against its null and whether the model as given is kept over the refinement; every other step is taken on the work-set target (where the free set is used tabulates them). The free set is the merge's own, so the refined R-free judges the refinement against reflections it never fitted - but where the model was deposited, those reflections were most likely its depositor's working reflections, and a refinement against new data then moves R-free up from the unrefined value rather than down. R_WORK/R_FREE of the model as given are never replaced. On a GPU a refinement takes seconds; the CPU engine computes the same and is several times slower.

How a model is scored. R-work, R-free and the likelihood target of every stage - the model as given (REFINE_R_WORK_START) as much as the refined one - come from one model of the structure factors, Fmodel = k exp(-sTBs/4) (Fcalc + ksol exp(-Bsols2/4) Fmask): the atoms with the dictionary's riding hydrogens and their B made isotropic; a flat bulk solvent masked with the REFMAC radii (probe 1.0 A, shrink 0.8 A) on a grid of its own, at most 0.4 A apart, so that the shrink acts at any resolution; ksol and Bsol searched over the physical range on the working reflections, and the overall scale and anisotropic B fitted to them by robust least squares, so that a few reflections the model cannot explain (a beam-stop shadow, an ice ring, a missing part of the model) do not set the scale of the rest. On depositions from 1.1 to 3.6 A this gives REFMAC 5.8's 0-cycle R-work and R-free on the same model and data within 0.006 (tools/battery/refine_parity.py runs the comparison; --refine-score-only prints each term under REFINE_SCORE_, see --mode refine).

The algorithms - the target and its gradient, the likelihood's parameters, the restraints, the minimiser and the weight trials - are described in §15, and the keys in The results report.

What it does not do yet. The refinement is a first, automatic pass on a placed model, not a replacement for a refinement program. Not implemented yet:

  • restraints for a new ligand from its chemistry alone: without a restraint file (or a CCD entry's ideal coordinates) a residue the built-in dictionary lacks is held fixed;
  • water building and ion relabelling by default: they are opt-in (--refine-solvent); without it the blobs only point at candidates;
  • occupancy refinement and new alternative conformations (existing ones are refined in place);
  • rotamer and peptide-flip search, Ramachandran restraints;
  • real-space refinement, jelly-body and reference-model restraints;
  • a full Metal (Apple GPU) path: on macOS only the model density is put on the grid on the GPU, the rest of the refinement runs on the CPU;
  • validation beyond the geometry's rmsd and clash count: no MolProbity-style report.

TLS groups, anisotropic U (an ANISOU record is kept as the atom's own U), NCS restraints and twin refinement are implemented and decided by the weight trials, see §15.

Refining a model against structure factors (rugnux --mode refine)

--mode refine refines a model against merged structure factors that already exist - processed elsewhere, a deposition's, or a rugnux .mtz - and processes nothing:

rugnux --mode refine --model model.cif -o refined data.mtz

A model and reflections given among the inputs with no images, and no --mode, are this mode too (rugnux -o refined data.mtz model.cif); restraint dictionaries for ligands are given beside them (Refinement).

The input is an MTZ or a structure-factor mmCIF (as the PDB distributes them), gzipped or not, recognised by content; the model is PDB or mmCIF, taken as placed in the data's cell and refined in the data's space group (in its own where that is a subgroup of the data's, against the data expanded to it - see §15.1 of the analysis notes). The columns are found, not named: amplitudes with their sigmas (F/SIGF, FP/SIGFP, FOBS, ...; calculated amplitudes and map coefficients such as FC, FWT, DELFWT are passed over) where the file has them, else intensities (IMEAN/SIGIMEAN, I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from a FreeR_flag-like column (flag 0 is the test set; where 0 marks the majority, as phenix and CNS write it, the minority value is; in a structure-factor mmCIF, _refln.status 'f' first); a file without one gets rugnux's own 5 % set. Where a structure-factor mmCIF's flags are classes (_refln.pdbx_r_free_flag 0-19), the class the file marks is not always the one the depositors held out, so the model as given is scored on each class: the mark is kept where the model fits that class worst or the test-set count the model file records (REMARK 3, _refine.ls_number_reflns_R_free) is nearest its count; otherwise the class the model fits worst is the test set where that count is nearest it, or where it is above the marked class by more than twice the scatter of the other classes' R (REFINE_FREE_SET_SOURCE= STATUS|REMARK3_COUNT|WORST_FIT, REFINE_FREE_SET_CLASS). The refinement is the one run after a merge on a placed model, and writes the same files - <prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz, <prefix>_blobs.json - with the REFINE_ keys in <prefix>_report.txt, beside REFINE_FRENCH_WILSON= and REFINE_FREE_FLAGS= FILE|GENERATED saying what had to be made of the input.

Option Description
--model <file> The model (required; it can also be given among the inputs, told by content)
--refine-score-only Score the model as given and refine nothing; every term of the score - R-work and R-free overall and in 20 resolution shells, the overall scale, B and anisotropic B, ksol and Bsol, the likelihood over the working and the free reflections and its figure of merit over the working reflections, and the geometry's rmsd and rmsZ - under REFINE_SCORE_ keys (--developer prints them all). The twin test is the refinement's: the model is scored against the data twinned by each law too, and the twinned score taken by the same rule (REFINE_TWIN)
--refine-d-min <A> Leave out reflections finer than this

Free set and model bias

How the free set is made. Rugnux draws its own test set (5 %, raised towards about 500 reflections on small data but never above 10 %): each reflection's flag is a hash of its index, taken over its orbit under the lattice's metric symmetry, so symmetry mates, Friedel mates and twin-law mates share a flag, every dataset of one crystal form written on the same axes gets the same set, and the set does not depend on the space group a file is merged in (R-free test-set flags). The model plays no part in it. Where a fitting model moves the files to its setting, the set is drawn again on the new axes by the same rule; an indexing-ambiguity reindex leaves it unchanged, since twin-law mates share a flag. With -z, the reference's flags are inherited where the merge matches it (REFERENCE_FREE_FLAGS_INHERITED=).

A model refined against these reflections. A set drawn here is free only for a model that was never refined against the crystal's reflections. A deposited model, or one refined against an earlier dataset of the same crystal, has been fitted to every reflection, so its R-free on a fresh set reads close to its R-work - below it, sometimes - and a refinement from it reads no cross-validation at all. --free-flags <file> takes the test set of the reflections it was refined against (the deposited -sf.cif, or an MTZ) and nothing else from them: unlike -z it fixes no group, cell or frame and steers no processing. The flags are put on the merge once it is on the model's axes, where the rigid body placed the model (a model placed by molecular replacement, or a prediction, keeps the drawn set) and the merge matches the file's intensities as -z requires (CC at least 0.5 over at least half of its reflections); the validation is then made again on them, and the refinement and the merged files carry them. R_FREE_FLAGS= says which set R_FREE and the refinement used: DRAWN, REFERENCE (-z) or FILE.

What is fitted on which reflections. Everything fitted to place and scale the model — the overall scale, the anisotropic B, the bulk solvent, the rigid body — is fitted on the working reflections only. The free set is then used for a few discrete selections: whether the rigid-body step is kept, which indexing wins, which merge the maps come from where a twin hypothesis was written, and in molecular replacement which placement and which group (where the free set is used lists every one). Each of those leaves R-free very slightly optimistic where it fires. For R_WORK/R_FREE the coordinates, B-factors and occupancies of the model are never refined; the refinement above refines them on the side, under REFINE_ keys of its own.

What R_FREE means here — and what it does not. It is the model as placed, scored on Rugnux's free set: a check that the model fits these data, and a trend number between datasets of one campaign. It is not a refinement R-free, and it is not cross-validated against the model's own history:

  • A deposited model was refined by its depositor against their working set. Rugnux's free set is a different 5 %, so about 95 % of Rugnux's free reflections were in the working set the model was refined on. R-free here therefore reads close to R-work, and it is optimistic in the usual model-bias sense.
  • To score a deposited model on its own test set, give its structure factors as the reference (-z 1abc-sf.cif): the depositor's free flags are then inherited. But -z is an assertion — it also fixes the space group and the cell — so that is no longer a de-novo run.
  • A predicted model has not been refined against any diffraction data, so it carries no such memory; what remains is the selection effect above.
  • For comparing two runs against one model, read R_MODEL_SHELL_SCALED, not R_FREE (Comparing two runs).

What cannot leak, and what can. The model never enters the merged intensities, their sigmas, the choice of reflections, the resolution cut or the weights, so <prefix>.mtz carries no model bias in its numbers — only the class-(b) labels. The maps are model-phased, and carry the model bias any model-phased map carries: read the mFo−DFc map, and treat density that only the 2mFo−DFc map shows with the usual care. CC(model, data) is read in one direction only: a confirmed shell is evidence for the data, a weak shell says nothing about them, and nothing acts on it.

Practical guidance

When to give a model.

  • You have a structure of this crystal form (a fragment campaign, a ligand soak, a reference structure): give it with --model. You get maps, a placed model and Rugnux's own first refinement of it, the indexing of every dataset brought into the model's, and the hand fixed — the things that make a campaign's files directly comparable. A wrong or rejected model costs a few seconds and changes nothing.
  • You have a structure of the same protein in another crystal form: --model too. The rigid body will not place it, and molecular replacement is tried automatically.
  • You have only a prediction (AlphaFold, ModelCIF): --model (or --mr). Download the PAE JSON with it and keep its AlphaFold DB file name, so it is cut into domains when the whole model does not place. A prediction is good for: confirming the space group, naming the hand where the data cannot, offering a twinned or lower-symmetry alternative where the data are ambiguous, and a first set of maps. It is not a refined model, and its R-factors read accordingly.
  • You have a previous dataset of the same crystal form and want the same indexing and free set: -z, not --model. It is the stronger tool for the indexing ambiguity and the only one that carries a free set over, but it pins the space group and cell.
  • You want to compare with another program's statistics: --report-resolution, which changes nothing else.

Which file to use for what.

File What it is Use it for
<prefix>.mtz / .cif / .hkl The answer: the data's space group, with only the class-(b) labels from the model Refinement, deposition, everything by default
<prefix>_model.cif / .pdb, <prefix>_maps.mtz The model as placed, in the frame of <prefix>.mtz, and its map coefficients (Coot/PyMOL open them as maps); only where the model was placed (MODEL_PLACEMENT= RIGID_BODY or MR) Starting refinement (REFMAC5, phenix.refine), PanDDA, Coot
<prefix>_refined_model.cif/.pdb, <prefix>_refined_maps.mtz, <prefix>_blobs.json The placed model refined by rugnux, its map coefficients (with the anomalous difference map's, ANOM/PHANOM, where the merge kept the Bijvoet pairs, as in <prefix>_maps.mtz), and the unexplained blobs of its difference map A first look at the refined density and at what the model leaves unexplained; continuing refinement
<prefix>_twin_<SG>.mtz / .cif The same data merged in a subgroup with a twin law — written from the data, model or not Twin refinement, where you suspect the higher group absorbed a twin law
<prefix>_mr.mtz / .cif, <prefix>_mr_model.pdb / .cif, <prefix>_mr_* maps The merge in the space group the model chose (MODEL_CHOICE_SPACE_GROUP=), and the model placed there Refinement if you accept the model's choice; compare R-free against the answer's

<prefix>_mr and <prefix>_twin_<SG> can describe the same subgroup; the _mr file then also carries the model placed in it.

Reading section 5 of the report, in order:

  1. MODEL_VALIDATION= — did it run? If NOT_PERFORMED, MODEL_VALIDATION_REASON= says why.
  2. MODEL_PLACEMENT= — how the model was placed: RIGID_BODY, MR or NOT_PLACED.
  3. MODEL_FIT=, MODEL_FIT_SIGMA= — the rigid body's verdict against its null. Absent where the rigid body was not tried (a prediction, --mr).
  4. MODEL_DECISIONS_TAKEN= — NONE means the reflection files are what a run without a model writes. Otherwise ENANTIOMORPH, INDEXING, both, or MR_ENANTIOMORPH, with MODEL_ENANTIOMORPH_ADOPTED= and MODEL_INDEXING_OPERATOR=.
  5. MODEL_CHOICE_FILE_PREFIX=, MODEL_CHOICE_SPACE_GROUP=, MR_DECISION= — whether a <prefix>_mr file was written, and in which group.
  6. R_WORK=, R_FREE=, CC_MODEL_CONFIRMED_TO_D_MIN=, the anomalous sites — how well the placed model describes the data (class d).

Outside section 5, SPACE_GROUP_ENANTIOMORPH= (UNDETERMINED, GIVEN or ASSUMED_FROM_MODEL), SETTING_SOURCE= and the presence or absence of INDEXING_AMBIGUITY_OPERATORS= say what the model did to the answer's labels. Every key is described in The results report.