diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 94b886f37..2910d46d3 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -652,8 +652,8 @@ structure refinement with phenix.refine" (2012), Acta Cryst. D68, 352-367 A. T. Brünger, "Free R value: a novel statistical quantity for assessing the accuracy of crystal structures" (1992), Nature 355, 472-475 [doi:10.1038/355472a0](https://doi.org/10.1038/355472a0). -**What the free set decides** — the free set decides the weights and the displacement model alone, and only -where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher +**What the free set decides** — the free set decides the weights, the displacement model and whether the model +as given is kept, and only where its two measures agree: a candidate is taken where its free $-\ln L$ is lower and its R-free not higher than the incumbent's by more than the paired standard error of the difference; every other step of the refinement is taken on the work-set target. The approach of deciding on more than one free measure, and of keeping the free set for few choices, is PDB-REDO's: R. P. Joosten, K. Joosten, G. N. Murshudov and A. Perrakis, diff --git a/docs/CPU_DATA_ANALYSIS_DECISIONS.md b/docs/CPU_DATA_ANALYSIS_DECISIONS.md index 574bbdcee..6e72d2f07 100644 --- a/docs/CPU_DATA_ANALYSIS_DECISIONS.md +++ b/docs/CPU_DATA_ANALYSIS_DECISIONS.md @@ -158,7 +158,7 @@ $ R = \frac{\sum \big|\,|F_o| - |F_\mathrm{model}|\,\big|}{\sum |F_o|}, $ -with R-free the same sum restricted to the free set. Every parameter $F_\mathrm{model}$ carries — the scaling of §14.2 and the rigid-body placement of §14.8 alike — is fitted on the working set alone, so the free reflections are held out of the fit as well as out of the sum. Two choices are still *selected* with the free set rather than fitted to it, and both are single discrete decisions rather than continuous parameters: the rigid-body step is committed only if it lowers R-free (§14.8), and the alternative indexing of §14.6 is the candidate with the lowest R-free. Both are the conventional use of a test set to accept or reject a step; both leave R-free very slightly optimistic where they fire. +with R-free the same sum restricted to the free set. Every parameter $F_\mathrm{model}$ carries — the scaling of §14.2 and the rigid-body placement of §14.8 alike — is fitted on the working set alone, so the free reflections are held out of the fit as well as out of the sum. A few choices are still *selected* with the free set rather than fitted to it, each a single discrete decision rather than a continuous parameter: the rigid-body step is committed only if it lowers R-free (§14.8), the alternative indexing of §14.6 is the candidate with the lowest R-free, the merge the maps come from where a twin hypothesis was written is the one the model fits better on R-free (§14.9), and molecular replacement ranks its placements, copies and groups by R-free (§14.10). Each is the conventional use of a test set to accept or reject a step; each leaves R-free very slightly optimistic where it fires. Every use of the free set, here and in the refinement, is tabulated in [§15.13](CPU_DATA_ANALYSIS_REFINEMENT.md#1513-where-the-free-set-is-used). ### 14.4 Electron-density maps diff --git a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md index 90d4f25cf..7808c2e15 100644 --- a/docs/CPU_DATA_ANALYSIS_INTEGRATION.md +++ b/docs/CPU_DATA_ANALYSIS_INTEGRATION.md @@ -451,7 +451,7 @@ A fraction of the unique reflections (`rfree_fraction`, default 0.05) is flagged On small data, where `rfree_fraction` (default 0.05) would give too few test reflections for a statistically stable R-free (Brünger's ~500–2000 rule), the fraction is **floored** so at least ~500 distinct reflections are free — capped at 10 % so a large test set never steals working data. For ordinary data this floor is inactive and the fraction stays flat at `rfree_fraction`, preserving the cross-dataset-identical property above; it only lifts the fraction on genuinely small datasets, where per-dataset R-free stability outweighs cross-dataset identity (and a shared reference set is the way to keep exact identity there). -When a reference (`--reference`: an MTZ, or a PDB structure-factor mmCIF, which is converted to one with `_refln.status` `f`/`o` becoming flag 0/1) carries a `FreeR_flag` column, its test set is **imported** instead: every merged reflection whose Laue-ASU index matches the reference takes the reference's flag (reflections absent from the reference keep the hash flag). The match is made in the reference's frame — on rotation data the merge is first put onto the reference's axes, choosing among every description of the lattice on them (and every alternative indexing) by the intensity correlation with the reference — and only where the merge then matches it: an intensity CC of at least 0.5 over at least half of the merged reflections in the reference's resolution range. Below that the reference is not the same crystal form on the same axes, its flags would land on unrelated reflections, and they are not imported (`REFERENCE_MISMATCH`). This lets a whole fragment-screening campaign inherit one shared free set from the apo/reference dataset. The CCP4/refmac convention (test set = flag 0, including the historical 0–19 form) is assumed, with the complement taken automatically if flag 0 would be the majority (a phenix-style file where 1 marks free). +When a reference (`--reference`: an MTZ, or a PDB structure-factor mmCIF, which is converted to one with `_refln.status` `f`/`o` becoming flag 0/1) carries a `FreeR_flag` column, its test set is **imported** instead: every merged reflection whose Laue-ASU index matches the reference takes the reference's flag (reflections absent from the reference keep the hash flag). The match is made in the reference's frame — on rotation data the merge is first put onto the reference's axes, choosing among every description of the lattice on them (and every alternative indexing) by the intensity correlation with the reference — and only where the merge then matches it: an intensity CC of at least 0.5 over at least half of the merged reflections in the reference's resolution range. Below that the reference is not the same crystal form on the same axes, its flags would land on unrelated reflections, and they are not imported (`REFERENCE_MISMATCH`). This lets a whole fragment-screening campaign inherit one shared free set from the apo/reference dataset. The CCP4/refmac convention (test set = flag 0, including the historical 0–19 form) is assumed, with the complement taken automatically if flag 0 would be the majority (a phenix-style file where 1 marks free). `--free-flags ` imports a test set the same way from a file of which nothing else is used - the deposited reflections of a `--model`, typically - applied on the model's axes where the rigid body placed the model (§14.5). Where each flag is then used - what is fitted on the working reflections, which choices the free set takes, and what only reports it - is tabulated in [§15.13](CPU_DATA_ANALYSIS_REFINEMENT.md#1513-where-the-free-set-is-used). ### 10.8 French–Wilson amplitudes diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index f2c541579..d58b195ac 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -346,7 +346,8 @@ chosen late, on the evidence, instead of early at a rule. **What the free set decides.** The free set decides the weights - the X-ray weights of the positions and of the B factors and the NCS restraints' $\sigma$ - the displacement model (§15.7, 6.), the reading of the model's TLS groups -(§15.1) and whether the model as given is kept over the refinement (7.), and nothing else. Each of these choices asks +(§15.1) and whether the model as given is kept over the refinement (7.); the one other free-set decision of the +refinement is the twin law against its null (§15.12), and §15.13 tabulates them all. Each of these choices asks both of its measures: a candidate is preferred to the incumbent only where its $-\ln L$ of the free reflections is lower **and** its R-free is not higher by more than the standard error of the difference. The two are scored on the same free reflections, so the difference is paired: R-free is a sum of one term per free reflection (per free twin @@ -767,7 +768,72 @@ The keys are `REFINE_TWIN` (`NO_TWIN_LAW`, `FREE_SET_SPLIT`, `ZERO_FRACTION`, `U `REFINE_UNTWINNED_R_WORK`/`_R_FREE`, the last five of the two refinements where they ended. Where `REFINE_TWIN` is `TWINNED`, every other `REFINE_` key, `REFINE_R_FREE_START` and the `REFINE_SCORE_` terms included, is of the twinned refinement. -### 15.13 Not yet +### 15.13 Where the free set is used + +One table for the whole run: how the free set comes about, what is fitted on which reflections, which choices it +decides and which numbers only report it. The merge draws and writes the set (§10.7); model validation and +molecular replacement (§14) and the refinement (§15) read it. Every row states the rule as the code applies it. + +**How the free set is obtained.** `R_FREE_FLAGS` names the set a run's validation and refinement used, and +`REFINE_FREE_FLAGS` the one `--mode refine` read. + +| Source | How | Reported as | +|---|---|---| +| Drawn by the merge (§10.7) | A hash of the reflection index over its orbit under the lattice holohedry, Friedel mates included - symmetry mates, Bijvoet mates and twin-law mates share one flag, so does a re-merge in any subgroup. 5 % of the distinct reflections, lifted towards 500 on small data and never above 10 %; no seed, no resolution shells. Drawn again, by the same rule, on the model's axes where a fitting model moves the setting; an indexing-ambiguity reindex leaves it as it was | `R_FREE_FLAGS= DRAWN` | +| Inherited from the reference (`-z`) | The reference's `FreeR_flag` (or `_refln.status`), matched on the merging group's Laue ASU where the merge matches the reference (CC ≥ 0.5 over at least half of its reflections); a reflection the reference lacks keeps its drawn flag | `REFERENCE`, `REFERENCE_FREE_FLAGS_INHERITED` | +| Taken from `--free-flags` | The same match, nothing else of the file used; applied on the model's axes, and only where the rigid body placed the model (§14.5) | `FILE` | +| Read by `--mode refine` | The file's `FreeR_flag`-like column, 0 the test set (the minority value where 0 is the majority); of an sf-mmCIF's classes 0-19, the class the file marks unless the model as given fits another class as it fits its working reflections; a file without flags gets a 5 % set drawn as above | `REFINE_FREE_FLAGS= FILE` / `GENERATED`, `REFINE_FREE_SET_CLASS` | +| Closed under the lattice's twin domains (§15.12) | Before a twinned refinement, every orbit under the domain operators is put in one set; a split orbit goes free with the probability of its own free share | `REFINE_TWIN_FREE_CLOSED` | +| Carried into another group | A merge expanded to the model's subgroup (§15.1) gives every new mate its parent's flag; the twin hypothesis' file (§13.2) carries the answer's flags | - | + +**What is fitted on which reflections.** The parameters of a fit are never fitted to the free reflections, with +two exceptions that are the convention of likelihood refinement: + +| Fitted quantity | Reflections | +|---|---| +| Overall scale, anisotropic B, bulk solvent (§14.2, §15.3); the rigid body (§14.8); the rotation and translation functions and the likelihood gain of molecular replacement (§14.10); the twin fraction (§14.9, §15.12); the X-ray target and its gradient, so every position, B, U and TLS parameter (§15.2); the side-chain field's energies, the population's averaged map (§15.7); $D$ of the likelihood (§15.3); Hamilton's R-ratio test (§15.7, 6.) | working only | +| $\beta$, the variance of the likelihood, per resolution bin (§15.3) | **free**; below 100 free reflections the working ones join them | +| $\sigma_A$ per shell for the validation's maps (§14.4) | **free**; a shell below 10 free reflections takes the estimate over the whole free set | + +**Decisions the free set takes.** "Paired rule" is the rule of §15.7: the candidate's $-\ln L$ of the free +reflections is lower *and* its R-free is not higher than the incumbent's by more than the standard error of the +paired difference; ties go to the incumbent (the trial listed first). + +| Decision | Compared on | Rule | Reported as | +|---|---|---|---| +| Rigid-body step kept (§14.8) | free | R-free lower than before the step; no standard error | `R_FREE_BEFORE_RIGID_BODY` | +| Indexing ambiguity, with a model and no `-z` (§14.6) | free | the indexing with the lowest R-free, where its lead over the runner-up is 3 σ beyond the leads a randomly oriented model takes; the test's resolution is widened until it holds 1000 free reflections | `MODEL_INDEXING_MARGIN*` | +| Which merge the maps come from, with a twin hypothesis (§14.9) | free | the lower R-free, the subgroup's plain or twinned against the answer's | `MAPS_FROM` | +| Molecular replacement: first copy, further copies, the group, a chain against the whole model, the twin subgroup, a carried placement (§14.10) | free, then both | the lowest R-free; a further copy lowers it by 0.01; another group must trail by 0.03 - within that margin, with as many copies, the paired per-reflection difference over work and free together decides at 3 standard errors, else `UNDECIDED` | `MR_DECISION`, `MR_COPIES` | +| Which symmetry hypothesis the model's choice settles on, where both fit within 0.03 before refinement | free | both refined; the rival replaces the validation's choice where its refined R-free is lower by more than 0.01 | `MODEL_CHOICE_DECIDED_BY`, `RIVAL_REFINE_*` | +| Reading of the model's TLS groups (§15.1) | free | paired rule, B total against B residual against the B as given | `REFINE_ADP_INPUT` | +| X-ray weights of the positions and the B factors, B held or refined (§15.7, 1., 2., 5.) | free | paired rule among the trials of a ladder or a step; a step that keeps the weights is taken on the work-set target instead | `REFINE_XRAY_WEIGHT` | +| NCS restraints on or off, and their σ (§15.5, §15.7) | free | paired rule, each rung and step with against without, tightened against loosened | `REFINE_NCS_SIGMA` | +| Displacement model: isotropic, the model's U, TLS by chain or cut, U per atom (§15.7, 6.) | work, then free | Hamilton's test on R-work decides where it passes more than 95 % or fewer than 30 % of its tests; between those the paired rule, a larger model also with its R-free lower outright. Which trials run is gated on R-free alone (TLS) or $-\ln L$ free alone (the TLS cuts, the full U weight) | `REFINE_ADP_MODEL` | +| Second round of the weights kept (§15.7, 6.) | free or work | as a step: paired rule where its weights or σ changed, the work-set target where they did not | - | +| Model as given kept over the refinement (§15.7, 7.) | free and work | paired rule for the model as given, and the refinement not better on both R-work and R-free; only where the set is free to the model as given (not a rigid-body placement on a drawn set) | `REFINE_AS_GIVEN` | +| Twin law against its null (§15.12) | free | the paired difference of $\lvert F_o - \lvert F\rvert\rvert$ over the free twin groups, 3 standard errors (2 where the L-test reads twinned); the twinned arm is stopped after its first ladder below 0.5; added domains at 3 | `REFINE_TWIN`, `REFINE_TWIN_NULL_Z`, `REFINE_TWIN_MULTI` | + +**Decided without the free set.** Whether the model fits at all, against its null (§14.5): R-work. The placement +stage's rigid bodies and B per residue (§15.1), a trajectory step at unchanged weights, the population's child, the +solvent build's rounds and the side-chain field's step (§15.7, §15.10): the work-set target. The twin fraction +(§14.9, §15.12), and molecular replacement's tNCS half-vector moves: the working reflections, or every reflection +together. + +**Reported only.** `R_WORK`/`R_FREE` and the shell table's `R_FREE` column (the model as placed, §14.3); +`REFINE_R_FREE_START`, `_PLACEMENT`, `REFINE_R_FREE` and the `REFINE_SCORE_` terms; `REFINE_TWIN_R_FREE`, +`REFINE_UNTWINNED_R_FREE` and `REFINE_TWIN_R_FREE_NULL` (the twin decision reads the null's z, never the two +R-frees); `RIVAL_REFINE_R_FREE`; the molecular-replacement table's `R_FREE`; the live page's per-shell and +refined R-free; and the flags themselves, written as `FreeR_flag` (0 free, 1 work) in the merged MTZ and the map +MTZs' `FREE`, and `_refln.status` `f`/`o` in the mmCIF. + +**Small free sets.** Nothing is cross-validated by folds: a small set is met by the floor on the draw (500 +reflections, within 10 %), by the working reflections joining the free ones for the likelihood's variance below +100, by $\sigma_A$ falling back to one estimate below 10 per shell, and by the indexing test widening its +resolution to 1000 free reflections. The paired rule's standard error is taken as 0 where only one free group +is left. + +### 15.14 Not yet What the refinement does not do yet - each a known gap, not a design choice: @@ -785,10 +851,11 @@ What the refinement does not do yet - each a known gap, not a design choice: density, not by refining each in turn. - **Alternative conformations and occupancies**: existing conformers are refined in place; no occupancy is refined and no conformer is added. -- **Rotamer and peptide-flip search**, and **Ramachandran restraints**. +- **Peptide-flip search** and **Ramachandran restraints** (side chains are searched over their staggered rotamers + by the side-chain field, §15.7). - **Real-space refinement**, and **jelly-body or reference-model restraints**. -- **A full Metal (Apple GPU) path**: on macOS the model density is gathered on the GPU and the rest of the - refinement runs on the CPU. +- **A full Metal (Apple GPU) path**: on macOS the model density, the atom-gradient sums, the solvent mask's + shrink and the side-chain field's sums run on the GPU and the rest of the refinement runs on the CPU. - **Validation beyond geometry**: the report gives the bond and angle rmsd, the metal-bond rmsd and the number of clashes, not a MolProbity-style analysis (Ramachandran, rotamer and clashscore statistics). diff --git a/docs/RUGNUX_CLI.md b/docs/RUGNUX_CLI.md index 2232a954d..b050c2656 100644 --- a/docs/RUGNUX_CLI.md +++ b/docs/RUGNUX_CLI.md @@ -330,7 +330,7 @@ serial stills; `fft` (GPU) / `fftw` (CPU) index **de novo** and suit strong rota | `--mosaicity ` | Diagnostic: fix the scaling mosaicity (°) instead of using the per-image seed | | `--scaling-iterations ` | Cap on the per-frame scaling iterations; the loop stops earlier once the scales settle, and a final merge whose scales were still moving when its loop stopped is flagged `SCALING_NOT_CONVERGED` (default: 100) | | `-z, --reference ` | Reference data of the same crystal form — an MTZ or a PDB structure-factor mmCIF (`-sf.cif`), gzipped or not, recognised by content; `--reference-mtz` is the same option: fixes the space group, its setting and the cell, resolves the [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity), hands over the R-free set and reports CCref. Not a scale anchor | -| `--free-flags ` | Take only the R-free test set of this MTZ or structure-factor mmCIF - e.g. the deposited reflections of the `--model` - for the validation, the refinement and the merged files; nothing else of the file is used, so unlike `-z` it fixes no space group, cell or frame. Taken on the model's axes where the merge matches the file's intensities (`R_FREE_FLAGS=`, see [Free set and model bias](RUGNUX_MODEL_AND_REFERENCE.md#free-set-and-model-bias)) | +| `--free-flags ` | Take only the R-free test set of this MTZ or structure-factor mmCIF - e.g. the deposited reflections of the `--model` - for the validation, the refinement and the merged files; nothing else of the file is used, so unlike `-z` it fixes no space group, cell or frame. Taken on the model's axes where the rigid body placed the model and the merge matches the file's intensities (`R_FREE_FLAGS=`, see [Free set and model bias](RUGNUX_MODEL_AND_REFERENCE.md#free-set-and-model-bias)) | | `--reference-column