diff --git a/docs/ACKNOWLEDGEMENT.md b/docs/ACKNOWLEDGEMENT.md index 504e01a88..5216023b6 100644 --- a/docs/ACKNOWLEDGEMENT.md +++ b/docs/ACKNOWLEDGEMENT.md @@ -252,6 +252,15 @@ H = |I(h) − I(Th)|/(I(h) + I(Th)) of the pairs a law relates is uniform on [0, gives the fraction. T. O. Yeates, "Detecting and overcoming crystal twinning" (1997), Methods Enzymol. 276, 344-358 [doi:10.1016/S0076-6879(97)76068-3](https://doi.org/10.1016/S0076-6879%2897%2976068-3). +**Twin refinement with more than two domains, and the free set closed under the twin domains**, follow +REFMAC5's treatment - one fraction per domain operator of the lattice, refined with the model, small domains +dropped - and its practice of keeping twin-related reflections in one set: +G. N. Murshudov, P. Skubák, A. A. Lebedev, N. S. Pannu, R. A. Steiner, R. A. Nicholls, M. D. Winn, F. Long and +A. A. Vagin, "REFMAC5 for the refinement of macromolecular crystal structures" (2011), Acta Cryst. D67, 355-367 +[doi:10.1107/S0907444911001314](https://doi.org/10.1107/S0907444911001314); P. Roversi, E. Blanc, S. Johnson and +S. M. Lea, "Tetartohedral twinning could happen to you too" (2012), Acta Cryst. D68, 418-424 +[doi:10.1107/S0907444912006737](https://doi.org/10.1107/S0907444912006737). + **The symmetry hypotheses a refused promotion leaves** - the adopted group and every other index-2 subgroup of the refused group perfectly twinned by a law of its coset, all merging alike, carried to the model and decided by placing and refining it in each - follow the advice to refine in the candidate groups and compare, diff --git a/docs/CPU_DATA_ANALYSIS.md b/docs/CPU_DATA_ANALYSIS.md index 6ae90cf23..8df2e38fd 100644 --- a/docs/CPU_DATA_ANALYSIS.md +++ b/docs/CPU_DATA_ANALYSIS.md @@ -73,6 +73,8 @@ The methods draw on, and in places reimplement, solutions from: - D. C. Liu & J. Nocedal, "On the limited memory BFGS method for large scale optimization", *Math. Program.* **45** (1989), 503-528 (the minimiser, §15). - J. Nocedal, "Updating quasi-Newton matrices with limited storage", *Math. Comp.* **35** (1980), 773-782 (the two-loop recursion of the minimiser, §15.6). - G. N. Murshudov, P. Skubák, A. A. Lebedev et al., "REFMAC5 for the refinement of macromolecular crystal structures", *Acta Cryst.* **D67** (2011), 355-367 (the likelihood's D from the working set and its variance from the free, and the Kullback-Leibler B restraints, §15.3, §15.5). +- P. Roversi, E. Blanc, S. Johnson & S. M. Lea, "Tetartohedral twinning could happen to you too", *Acta Cryst.* **D68** (2012), 418-424 (twin refinement with more than two domains, and the free set closed under the twin domains, §15.12). +- K. A. Sultana, I. Alexeev, I. Kursula, P. Mäntsälä, J. Niemi & G. Schneider, "Structure determination by multiwavelength anomalous diffraction of aclacinomycin oxidoreductase: indications of multidomain pseudomerohedral twinning", *Acta Cryst.* **D63** (2007), 149-159 (a six-domain twin of a monoclinic crystal on a hexagonal metric, §15.12). - A. T. Brünger, M. Karplus & G. A. Petsko, "Crystallographic refinement by simulated annealing: application to crambin", *Acta Cryst.* **A45** (1989), 50-61 (the starting weight from the ratio of the gradients, §15.7). - P. J. Huber, "Robust estimation of a location parameter", *Ann. Math. Statist.* **35** (1964), 73-101, and P. Charbonnier, L. Blanc-Féraud, G. Aubert & M. Barlaud, "Deterministic edge-preserving regularization in computed imaging", *IEEE Trans. Image Process.* **6** (1997), 298-311 (the robust, pseudo-Huber scale fit of the refinement's score, §15.3). - J.-S. Jiang & A. T. Brünger, "Protein hydration observed by X-ray diffraction", *J. Mol. Biol.* **243** (1994), 100-115 (the flat bulk-solvent mask, §14.2, §15.3). diff --git a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md index 252707360..830f3aec5 100644 --- a/docs/CPU_DATA_ANALYSIS_REFINEMENT.md +++ b/docs/CPU_DATA_ANALYSIS_REFINEMENT.md @@ -537,7 +537,8 @@ intensities is narrower; $\Sigma$, fitted to the free set, absorbs that, and REF (Murshudov et al. 2011) is the treatment this one simplifies. The mates $T\mathbf{h}$ that were not observed are composed as rows of their own, with no observation: their structure factors come from the same transform, so a twinned evaluation costs no extra FFT, only one more kernel per evaluation on the GPU (the twinned term per -reflection and the gather of its mate's derivative, the law being an involution on the rows). +reflection and the gather of its mates' derivatives, through the converse of the mate table, as a three-fold of +several domains is not an involution). **The fraction, the scale and the solvent** are fitted inside the twin sum at every macro-cycle: the scale and the solvent as in §15.3, then $\alpha$, then the scale and the solvent again against $F_o$ detwinned by the model as it @@ -571,15 +572,40 @@ leaves the L-test unreadable): two independent pieces of evidence, the data's an pseudo-merohedral law or a near-perfect twin among them - only further evidence where the null alone is short. `--mode refine` has no merge statistics, so there the null alone decides. -A law is tested only where the free set keeps its twin pairs together, and is otherwise reported -(`REFINE_TWIN` `FREE_SET_SPLIT`) rather than tested. The free set rugnux draws (§10.7) keeps together the pairs of -every law that is a rotation of the cell's own basis vectors; a law of a centred lattice that is not (its matrix -fractional in the cell's basis) can be split by it, and so can a free set `--mode refine` reads from a file. Of several laws, the one the placed model asks the largest fraction of is refined. Each of the +**The free set is closed under the lattice's domains** before either refinement, so that both use one free set +and no law splits a group of twin-related reflections between the working and the free set (REFMAC5 does the same, +Roversi et al. 2012). The groups are the orbits of the merged reflections under every domain operator of the lattice +(below); an orbit the flags already keep in one set stays there, and a split one goes free with the probability of its +own share of free flags, drawn by a hash of the orbit, so the free fraction stays what it was. A majority of each +orbit's flags would empty the free set where the flags were drawn on smaller orbits - a C-centred orthorhombic merge +of a hexagonal metric, whose flags (§10.7) are drawn on the orthorhombic holohedry of the cell's own basis vectors, lost +three quarters of it that way. The free set rugnux draws is already closed for every law that is a rotation of the +cell's own basis vectors, and nothing moves; a law of a centred lattice that is not (its matrix fractional in the +cell's basis), or a free set `--mode refine` reads from a file, can be split, and the closure is what lets such a law +be tested at all. `REFINE_TWIN_FREE_CLOSED` counts the reflections that changed set. A law still split (none is, +after the closure, unless its mates fall outside the orbits the data have) is reported (`REFINE_TWIN` +`FREE_SET_SPLIT`) rather than tested. Of several laws, the one the placed model asks the largest fraction of is refined. Each of the two refinements is a whole one - its solvent built (§15.10) and its displacement model chosen (§15.7) against its own target - and the decision is taken where they ended. `--refine-score-only` makes the same test on the model as given: it is scored against the data untwinned and twinned by the law, and the twinned score is taken by the same rule. +**More than two domains.** A lattice can hold more domains than two: a monoclinic crystal on a tetragonal metric +four, on a hexagonal one six (Sultana et al. 2007; Roversi et al. 2012). Where the single law is taken and the lattice +has more domain operators - every coset of the group in the lattice's metric symmetry, whatever the operator's order, +three-folds included - a second rung refines every domain at once, each with its own fraction: +$|F_\mathrm{twin}(\mathbf{h})|^2 \propto a_0|U(\mathbf{h})|^2 + \sum_d a_d|U(T_d\mathbf{h})|^2$, $a_0 = 1 - \sum_d a_d$, +the fractions from the same intercept regression with one column per domain, a domain whose coefficient comes out +negative taken out and the rest fitted again. It is taken where the added domains beat **their own null** - the +same model with the added domains' mates replaced by unrelated reflections and the adopted law's kept - by three +standard errors on the free set, summed over each twin group, and each added domain refines to a fraction of 0.05 or +more (as REFMAC5 drops small domains, Murshudov et al. 2011); the domains under it are dropped and the rest refined +again, and with none left the single law stands. `REFINE_TWIN_MULTI` (`MULTI_TWINNED`, `MULTI_UNTWINNED`) says what +was decided, `REFINE_TWIN_MULTI_DOMAINS` the domains and their fractions; where `MULTI_TWINNED`, `REFINE_TWIN_LAW` +lists every law and `REFINE_TWIN_FRACTION` is their sum, and every other `REFINE_` key is of that refinement. The +rung is not tried where the single law was refused: a twin that only several domains show at once is a symmetric +one, whose single-law fraction is already large. + **Maps.** Against twinned data the map coefficients use $F_o$ detwinned in proportion to the model's own share of the twinned intensity, $F_o|F_\mathrm{model}(\mathbf{h})|/|F_\mathrm{twin}(\mathbf{h})|$, and $DF_c$ of the model's own $|F_\mathrm{model}(\mathbf{h})|$; `FP` in `_refined_maps.mtz` is the detwinned amplitude. diff --git a/docs/RUGNUX_REPORT.md b/docs/RUGNUX_REPORT.md index 8beac28c2..f9f16dad5 100644 --- a/docs/RUGNUX_REPORT.md +++ b/docs/RUGNUX_REPORT.md @@ -1381,6 +1381,10 @@ Where a model was placed and refined ([Refinement](RUGNUX_MODEL_AND_REFERENCE.md | `REFINE_TWIN_R_FREE_NULL` | The twinned model's R-free at its fraction with every mate replaced by an unrelated reflection of the same resolution and set: what averaging alone buys, and the like-for-like comparison for `REFINE_TWIN_R_FREE`. `REFINE_TWIN_FRACTION_NULL` (*dev*) is the fraction those unrelated mates ask for, about 0 | | `REFINE_TWIN_NULL_Z` | The law's lead over that null on the free reflections, paired reflection by reflection (a free twin pair as one term), in standard errors; `TWINNED` needs 3, or 2 where the data read twinned (`REFINE_TWIN_RULE`) | | `REFINE_TWIN_RULE` | Where `UNTWINNED` or `TWINNED`, the rule that decided ([§15.12](CPU_DATA_ANALYSIS_REFINEMENT.md#1512-twinned-data)): `NULL` (the null alone, at 3 standard errors) or `NULL+LTEST` (taken at 2-3 standard errors because the merge's L-test reads twinned as well: `TWINNING_VERDICT` `INDICATED` with `ESTIMATED_TWIN_FRACTION_FROM` `L_TEST`). In `--mode refine` there are no merge statistics, and it is always `NULL` | +| `REFINE_TWIN_FREE_CLOSED` | Where any moved: the reflections that changed between the working and the free set when the free set was closed under the lattice's twin domains before both refinements ([§15.12](CPU_DATA_ANALYSIS_REFINEMENT.md#1512-twinned-data)) | +| `REFINE_TWIN_MULTI` | Where the single law was taken and the lattice has more domains: `MULTI_TWINNED` (every domain refined at once is the result: `REFINE_TWIN_LAW` then lists every law, `;`-separated, and `REFINE_TWIN_FRACTION` is their sum) or `MULTI_UNTWINNED` (the single law stands) | +| `REFINE_TWIN_MULTI_DOMAINS` | The domains of that refinement and their fractions, `law=fraction`, `;`-separated, the adopted law first; where `MULTI_UNTWINNED` because every added domain fell below 0.05, as they refined before being dropped | +| `REFINE_TWIN_MULTI_R_WORK`, `REFINE_TWIN_MULTI_R_FREE`, `REFINE_TWIN_MULTI_R_FREE_NULL`, `REFINE_TWIN_MULTI_NULL_Z` | Where the domains kept were refined: that refinement's R factors, its null (the added domains' mates unrelated, the adopted law's kept) and the added domains' lead over it in standard errors; `MULTI_TWINNED` needs 3 | | `REFINE_BLOBS` | The blobs of the refined mFo-DFc map (above 3 σ, peaking above 4 σ) | | `REFINE_BLOBS_LIGAND`, `REFINE_BLOBS_WATER`, `REFINE_BLOBS_ION`, `REFINE_BLOBS_MODEL_ERROR` | How many of them are read as each ([§15.9](CPU_DATA_ANALYSIS_REFINEMENT.md#159-unexplained-density-blobs)); the rest are unassigned | | `REFINE_SOLVENT` | `PERFORMED` \| `NOT_PERFORMED`: the solvent build and audit of the refined model, off unless `--refine-solvent` is given; the other `REFINE_SOLVENT_` keys and `_solvent.json` only where `PERFORMED` | diff --git a/image_analysis/structure_refinement/Refinement.cpp b/image_analysis/structure_refinement/Refinement.cpp index adb494cf1..4c31dc9b4 100644 --- a/image_analysis/structure_refinement/Refinement.cpp +++ b/image_analysis/structure_refinement/Refinement.cpp @@ -448,7 +448,7 @@ struct Context { // The scale and the solvent inside the twin sum: the fraction for the untwinned fit, then the scale // fitted again against Fo detwinned by the model as it stands, Fo |F_model(h)| / |F_twin(h)| - which // the twinned model at that scale meets as the untwinned one meets Fo - and the fraction again. - rows.twin_fraction = static_cast(FitTwinFraction(refl, rows, fcalc)); + rows.twin_fractions = FitTwinFractions(refl, rows, fcalc); const std::vector> twinned = TwinnedModel(refl, rows, fcalc); fo.v.clear(); for (size_t m = 0; m < refl.Size(); m++) { @@ -459,7 +459,7 @@ struct Context { fo.v.push_back({refl.hkl[m], {static_cast(tw > 0 ? refl.fo[m] * own / tw : refl.fo[m]), 1.0f}}); } fit_scale(); - rows.twin_fraction = static_cast(FitTwinFraction(refl, rows, fcalc)); + rows.twin_fractions = FitTwinFractions(refl, rows, fcalc); } s3.reset(); const std::vector> fmodel = TwinnedModel(refl, rows, fcalc); @@ -538,7 +538,9 @@ struct Context { } } sc.fom = fom_n > 0 ? fom_sum / fom_n : NAN; - sc.twin_fraction = rows.twin_fraction; + sc.twin_fraction = 0; + for (float a : rows.twin_fractions) + sc.twin_fraction += a; for (int p : rider_parent) sc.hydrogens += p >= 0; constexpr int SHELLS = 20; @@ -1265,6 +1267,27 @@ void MakeMaps(Context &ctx, const std::vector &merged, double map.get(); } +// The fractions of the twin laws, together. +double FractionSum(const XrayRowModel &rows) { + double sum = 0; + for (float a : rows.twin_fractions) + sum += a; + return sum; +} + +// law=fraction per twin law, ';'-separated. +std::string DomainText(const RefinementReflections &refl, const XrayRowModel &rows) { + std::string out; + for (size_t d = 0; d < refl.twin_laws.size(); d++) + out += fmt::format("{}{}={:.3f}", out.empty() ? "" : ";", refl.twin_laws[d].as_hkl().triplet('h'), + d < rows.twin_fractions.size() ? rows.twin_fractions[d] : 0.0f); + return out; +} + +// A domain added to the adopted law is kept only at this fraction or more, on top of its null. +// As REFMAC5 drops twin domains of small fraction, Murshudov et al. (2011) Acta Cryst. D67, 355-367 +constexpr double TWIN_DOMAIN_MIN = 0.05; + // The null of a twinned refinement, for the model as the last Refit() left it: its R-free at the same fraction with // every reflection's mate replaced by an unrelated observed one of the same resolution and the same set (free with // free, work with work, as a law's twin pairs are). The reflections of each set are put in order of 1/d^2 with the two @@ -1277,56 +1300,72 @@ constexpr size_t TWIN_NULL_SHIFT = 2; // evidence (AdoptTwinLaw). constexpr double TWIN_NULL_Z = 3.0; constexpr double TWIN_NULL_Z_DATA = 2.0; -void TwinNull(const Context &ctx, RefinementResult &result) { +// `first`: the first law whose mates are replaced - the laws before it keep theirs (the null of the domains added to +// them). With several laws a twin group holds up to Laws() + 1 reflections, kept next to each other, and law first + k +// is paired (Laws() + 1) (k + 1) apart: past the reflection's own group, and no two laws drawing the same partner. +void TwinNull(const Context &ctx, RefinementResult &result, size_t first = 0) { RefinementReflections null = ctx.refl; const size_t n = null.Size(); - null.mate_hkl.clear(); - null.mate.assign(n, 0); + // each observed reflection's twin group, named by its lowest observed row + std::vector group(n); + for (size_t m = 0; m < n; m++) { + group[m] = static_cast(m); + for (size_t d = 0; d < ctx.refl.Laws(); d++) + if (ctx.refl.mates[d][m] < static_cast(n)) + group[m] = std::min(group[m], ctx.refl.mates[d][m]); + } + for (size_t d = first; d < null.Laws(); d++) + std::iota(null.mates[d].begin(), null.mates[d].end(), 0); for (const bool free : {false, true}) { std::vector order; for (size_t m = 0; m < n; m++) if ((null.free[m] != 0) == free) order.push_back(static_cast(m)); - // a twin pair is ordered at the 1/d^2 of its first member - auto pair_of = [&](int m) { return ctx.refl.mate[m] < static_cast(n) ? std::min(m, ctx.refl.mate[m]) : m; }; + // a twin group is ordered at the 1/d^2 of its first member std::stable_sort(order.begin(), order.end(), [&](int a, int b) { - const int pa = pair_of(a), pb = pair_of(b); + const int pa = group[a], pb = group[b]; return std::make_pair(null.inv_d2[pa], pa) < std::make_pair(null.inv_d2[pb], pb); }); - for (size_t r = 0; r < order.size(); r++) { - // in blocks of twice the shift, the first half with the second - const size_t p = r % (2 * TWIN_NULL_SHIFT) < TWIN_NULL_SHIFT ? r + TWIN_NULL_SHIFT : r - TWIN_NULL_SHIFT; - null.mate[order[r]] = order[p < order.size() ? p : r]; + for (size_t d = first; d < null.Laws(); d++) { + const size_t shift = (null.Laws() + 1) * (d - first + 1); + for (size_t r = 0; r < order.size(); r++) { + // in blocks of twice the shift, the first half with the second + const size_t p = r % (2 * shift) < shift ? r + shift : r - shift; + null.mates[d][order[r]] = order[p < order.size() ? p : r]; + } } } // at the law's fraction: averaging any two intensities lifts the weak reflections' model amplitudes off zero // by itself (the fraction the unrelated mates ask for, reported beside it, is about 0) XrayRowModel rows = ctx.rows; - result.twin_fraction_null = FitTwinFraction(null, rows, ctx.fcalc); + { + const std::vector a = FitTwinFractions(null, rows, ctx.fcalc); + result.twin_fraction_null = 0; + for (size_t d = first; d < a.size(); d++) + result.twin_fraction_null += a[d]; + } const std::vector> f_null = TwinnedModel(null, rows, ctx.fcalc); result.twin_r_free_null = ComputeRFactors(null, f_null).r_free; - // How far the law beats its null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin|| - // over its standard error. The two members of a free twin pair share their two F_calc, so their terms are summed - // into one: the standard error counts twin pairs, not reflections. + // How far the laws beat their null on the free reflections, paired: the mean of |Fo - |F_null|| - |Fo - |F_twin|| + // over its standard error. The members of a free twin group share their F_calc, so their terms are summed into + // one: the standard error counts twin groups, not reflections. const std::vector> f_law = TwinnedModel(ctx.refl, ctx.rows, ctx.fcalc); - std::vector d(n, 0.0); + std::vector sum_group(n, 0.0); for (size_t m = 0; m < n; m++) if (ctx.refl.free[m]) - d[m] = std::fabs(ctx.refl.fo[m] - std::abs(f_null[m])) - std::fabs(ctx.refl.fo[m] - std::abs(f_law[m])); + sum_group[group[m]] += std::fabs(ctx.refl.fo[m] - std::abs(f_null[m])) + - std::fabs(ctx.refl.fo[m] - std::abs(f_law[m])); double sum = 0, sum2 = 0; - int n_pairs = 0; - for (size_t m = 0; m < n; m++) { - const size_t t = ctx.refl.mate[m]; - if (!ctx.refl.free[m] || (t < n && t < m)) - continue; // the pair was counted at its first member - const double pair = t < n && t != m ? d[m] + d[t] : d[m]; - sum += pair; - sum2 += pair * pair; - n_pairs++; - } - if (n_pairs > 1) { - const double mean = sum / n_pairs, var = (sum2 - n_pairs * mean * mean) / (n_pairs - 1); - result.twin_null_z = var > 0 ? mean / std::sqrt(var / n_pairs) : 0.0; + int n_groups = 0; + for (size_t m = 0; m < n; m++) + if (ctx.refl.free[m] && group[m] == static_cast(m)) { + sum += sum_group[m]; + sum2 += sum_group[m] * sum_group[m]; + n_groups++; + } + if (n_groups > 1) { + const double mean = sum / n_groups, var = (sum2 - n_groups * mean * mean) / (n_groups - 1); + result.twin_null_z = var > 0 ? mean / std::sqrt(var / n_groups) : 0.0; } } @@ -2087,10 +2126,12 @@ double RefinementDMin(double max_inv_d2) { } // One refinement of a model, with the solvent build in it where settings.solvent asks for it. -// `twin_law`: against the data as twinned by it, null for untwinned. +// `twin_laws`: against the data as twinned by them (one domain each beside the model's own), empty for untwinned; +// `null_first`: the first of them the null (TwinNull) replaces the mates of. RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, - Logger &logger, const RefinementSettings &settings, const gemmi::Op *twin_law = nullptr) { + Logger &logger, const RefinementSettings &settings, + const std::vector &twin_laws = {}, size_t null_first = 0) { const auto start = std::chrono::steady_clock::now(); RefinementResult result; Context ctx; @@ -2129,10 +2170,11 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vectoras_hkl().triplet('h'); + for (const gemmi::Op &law : twin_laws) + result.twin_law += (result.twin_law.empty() ? "" : ";") + law.as_hkl().triplet('h'); } ctx.mask = std::make_unique(cell, sg, ctx.d_min, settings.nthreads); @@ -2204,7 +2246,7 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, Logger &logger, const RefinementSettings &settings, RefinementResult &result) { @@ -3145,6 +3192,7 @@ void DecideTwin(const gemmi::Structure &placed, const std::vector laws = RefinementTwinLaws(cell, sg); + const std::vector domains = RefinementTwinDomains(cell, sg); result.twin_laws = static_cast(laws.size()); if (laws.empty()) { result.twin_decision = "NO_TWIN_LAW"; @@ -3175,7 +3223,7 @@ void DecideTwin(const gemmi::Structure &placed, const std::vector best) { @@ -3185,7 +3233,7 @@ void DecideTwin(const gemmi::Structure &placed, const std::vector all{law}; + for (const gemmi::Op &op : domains) + if (op.rot != law.rot) + all.push_back(op); + if (all.size() < 2) + return; + // Refined, the added domains under TWIN_DOMAIN_MIN dropped and the rest refined again, until every one left + // has it; none left: the single law stands. + RefinementResult multi; + while (true) { + multi = RefineOnce(placed, merged, cell, sg, wavelength_A, logger, settings, all, 1); + if (!multi.ok || multi.cancelled) + return; + std::vector kept{law}; + for (size_t d = 1; d < all.size(); d++) + if (d < multi.twin_domain_fractions.size() && multi.twin_domain_fractions[d] >= TWIN_DOMAIN_MIN) + kept.push_back(all[d]); + if (kept.size() == all.size()) + break; + logger.Info("Refinement, {} twin domains ({}): {} below a fraction of {:.2f}{}", all.size() + 1, + multi.twin_multi_domains, all.size() - kept.size(), TWIN_DOMAIN_MIN, + kept.size() > 1 ? ", refined again without" : " - the single law stands"); + if (kept.size() == 1) { + result.twin_multi_decision = "MULTI_UNTWINNED"; + result.twin_multi_domains = multi.twin_multi_domains; + return; + } + all = kept; + } + result.twin_multi_domains = multi.twin_multi_domains; + result.twin_multi_r_work = FinalRWork(multi); + result.twin_multi_r_free = FinalRFree(multi); + result.twin_multi_r_free_null = multi.twin_r_free_null; + result.twin_multi_null_z = multi.twin_null_z; + const bool adopt_multi = multi.twin_null_z >= TWIN_NULL_Z; + result.twin_multi_decision = adopt_multi ? "MULTI_TWINNED" : "MULTI_UNTWINNED"; + logger.Info("Refinement, {} twin domains ({}): R-work {:.4f}, R-free {:.4f}; one law R-free {:.4f}; null (the added " + "domains' mates unrelated) R-free {:.4f}, the domains ahead of it by {:.1f} standard errors -> {}", + all.size() + 1, multi.twin_multi_domains, result.twin_multi_r_work, result.twin_multi_r_free, + result.twin_r_free, result.twin_multi_r_free_null, result.twin_multi_null_z, result.twin_multi_decision); + if (!adopt_multi) + return; + multi.twin_decision = result.twin_decision; + multi.twin_rule = result.twin_rule; + multi.twin_laws = result.twin_laws; + multi.twin_free_closed = result.twin_free_closed; + multi.twin_fraction_null = result.twin_fraction_null; + multi.twin_r_free_null = result.twin_r_free_null; + multi.twin_null_z = result.twin_null_z; + multi.twin_r_work = result.twin_r_work; + multi.twin_r_free = result.twin_r_free; + multi.untwinned_r_work = result.untwinned_r_work; + multi.untwinned_r_free = result.untwinned_r_free; + multi.twin_multi_decision = result.twin_multi_decision; + multi.twin_multi_r_work = result.twin_multi_r_work; + multi.twin_multi_r_free = result.twin_multi_r_free; + multi.twin_multi_r_free_null = result.twin_multi_r_free_null; + multi.twin_multi_null_z = result.twin_multi_null_z; + result = std::move(multi); } } // namespace +int CloseFreeSet(std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg) { + const std::vector laws = RefinementTwinDomains(cell, sg); + if (laws.empty()) + return 0; + const gemmi::GroupOps gops = sg.operations(); + const gemmi::ReciprocalAsu asu(&sg); + auto key = [&](const gemmi::Miller &h) { return asu.to_asu(h, gops).first; }; + // each reflection's orbit, by the ASU index of every reflection the laws reach from it; named by its lowest index + std::map orbit_of; + std::vector name; + std::vector orbit(merged.size()); + for (size_t i = 0; i < merged.size(); i++) { + const gemmi::Miller h = key(gemmi::Miller{{merged[i].h, merged[i].k, merged[i].l}}); + if (const auto it = orbit_of.find(h); it != orbit_of.end()) { + orbit[i] = it->second; + continue; + } + const int o = static_cast(name.size()); + name.push_back(h); + std::vector todo{h}; + orbit_of[h] = o; + while (!todo.empty()) { + const gemmi::Miller x = todo.back(); + todo.pop_back(); + name[o] = std::min(name[o], x); + for (const gemmi::Op &law : laws) + if (const gemmi::Miller y = key(law.apply_to_hkl(x)); orbit_of.emplace(y, o).second) + todo.push_back(y); + } + orbit[i] = o; + } + std::vector free(name.size(), 0), all(name.size(), 0); + for (size_t i = 0; i < merged.size(); i++) { + free[orbit[i]] += merged[i].rfree_flag; + all[orbit[i]]++; + } + // splitmix64 of the orbit's name -> uniform in [0, 1), salted: unsalted it is AssignRfreeFlags' own draw of a + // reflection the name often is, and an orbit would go free whenever that member was + auto uniform = [](const gemmi::Miller &h) { + uint64_t z = (static_cast(h[0] + (1 << 20)) << 42 | static_cast(h[1] + (1 << 20)) << 21 + | static_cast(h[2] + (1 << 20))) + 0x9e3779b97f4a7c15ULL * 2; + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL; + z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL; + return static_cast((z ^ (z >> 31)) >> 11) * (1.0 / 9007199254740992.0); + }; + int moved = 0; + for (size_t i = 0; i < merged.size(); i++) { + const int o = orbit[i]; + const bool flag = free[o] == all[o] || (free[o] > 0 && uniform(name[o]) * all[o] < free[o]); + moved += merged[i].rfree_flag != flag; + merged[i].rfree_flag = flag; + } + return moved; +} + bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) { return twin_fraction > 0 && (null_z >= TWIN_NULL_Z || (data_indicated && null_z >= TWIN_NULL_Z_DATA)); } @@ -3227,12 +3392,24 @@ bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated) { RefinementResult RefineModel(const gemmi::Structure &placed, const std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double wavelength_A, Logger &logger, const RefinementSettings &settings) { - RefinementResult result = RefineOnce(placed, merged, cell, sg, wavelength_A, logger, settings); + // The free set closed under the lattice's twin domains once, before either arm, so that both share it. + std::vector closed; + int moved = 0; + if (settings.twin) { + closed = merged; + moved = CloseFreeSet(closed, cell, sg); + if (moved > 0) + logger.Info("Refinement: the free set closed under the lattice's twin domains, each group of twin-related " + "reflections in one set - {} reflections changed set", moved); + } + const std::vector &data = moved > 0 ? closed : merged; + RefinementResult result = RefineOnce(placed, data, cell, sg, wavelength_A, logger, settings); // Each arm, untwinned and twinned, is a whole refinement - the solvent built and the displacement model chosen // against its own target - and the decision is taken on where they ended. A model only scored is scored both // ways and decided by the same rule, so the R-free it reports is not an untwinned one by default. if (settings.twin && result.ok && !result.cancelled) - DecideTwin(placed, merged, cell, sg, wavelength_A, logger, settings, result); + DecideTwin(placed, data, cell, sg, wavelength_A, logger, settings, result); + result.twin_free_closed = moved; result.map.reset(); return result; } diff --git a/image_analysis/structure_refinement/Refinement.h b/image_analysis/structure_refinement/Refinement.h index 9a1fb8d78..909ccfb13 100644 --- a/image_analysis/structure_refinement/Refinement.h +++ b/image_analysis/structure_refinement/Refinement.h @@ -232,8 +232,28 @@ struct RefinementResult { // Where tested to UNTWINNED or TWINNED, the rule that decided: NULL (the null alone, 3 standard errors), or // NULL+LTEST (taken at 2-3 standard errors because the data read twinned, settings.twin_data_indicated). std::string twin_rule; + // The domains beyond two (DecideTwin's second rung): every other domain of the lattice added to the adopted law, + // its decision (MULTI_TWINNED where the model is that refinement's, MULTI_UNTWINNED, or empty: not tested), the + // laws and their fractions (law=fraction, ';'-separated), its R-free, and the null of the domains added - their + // mates unrelated, the adopted law's kept - with the domains' lead over it in standard errors. + std::string twin_multi_decision; + std::string twin_multi_domains; + double twin_multi_r_work = NAN, twin_multi_r_free = NAN, twin_multi_r_free_null = NAN, twin_multi_null_z = NAN; + int twin_free_closed = 0; // reflections moved between work and free to close the free set + // under the lattice's domains (CloseFreeSet) + std::vector twin_domain_fractions; // per law of the refinement (twin_law's order), as refined }; +// The free set of `merged` closed under the lattice's domain operators (RefinementTwinDomains): the reflections that +// any domain relates - an orbit of the lattice's holohedry, as far as the data have it - all free or all working, so +// that no twin law splits a twin group between the working and the free set. An orbit the flags already keep in one +// set stays there; a split one goes free with the probability of its own share of free flags, drawn by a hash of the +// orbit, so the free fraction stays as it was (a majority would empty the free set where the flags were drawn on +// smaller orbits). Returns how many reflections changed set - none for a free set drawn on the lattice's holohedry +// (AssignRfreeFlags), some for one a file brings or one keyed on another cell. +// As REFMAC5 keeps twin-related reflections in one set (Roversi et al. (2012) Acta Cryst. D68, 418-424) +int CloseFreeSet(std::vector &merged, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); + // Whether the twinned refinement is taken (RefineModel): a fraction above 0, and the law ahead of its null on the // free set by 3 standard errors - or by 2 where the data's own intensity statistics read twinned as well. bool AdoptTwinLaw(double twin_fraction, double null_z, bool data_indicated); diff --git a/image_analysis/structure_refinement/RefinementGPU.cpp b/image_analysis/structure_refinement/RefinementGPU.cpp index c66406922..e2dc285c1 100644 --- a/image_analysis/structure_refinement/RefinementGPU.cpp +++ b/image_analysis/structure_refinement/RefinementGPU.cpp @@ -150,7 +150,7 @@ public: setup_->work.push_back(!refl.free[m]); setup_->centric.push_back(refl.centric[m]); } - // a twin law's mates not observed: structure factors only + // the twin laws' mates not observed: structure factors only for (const gemmi::Miller &h : refl.mate_hkl) { setup_->rows.push_back({h[0], h[1], h[2]}); setup_->inv_d2.push_back(cell.calculate_1_d2(h)); @@ -158,7 +158,11 @@ public: setup_->work.push_back(0); setup_->centric.push_back(0); } - setup_->mate = refl.mate; + setup_->laws = static_cast(refl.Laws()); + for (size_t d = 0; d < refl.Laws(); d++) { + setup_->mate.insert(setup_->mate.end(), refl.mates[d].begin(), refl.mates[d].end()); + setup_->mate_from.insert(setup_->mate_from.end(), refl.mates_from[d].begin(), refl.mates_from[d].end()); + } // The adjoint of the composition on cuFFT's half-u spectrum: the term of row m and operator o, at // k = hR, goes to k where k_u > 0, conjugated to -k where k_u < 0, and to both on the planes k_u = 0 @@ -244,7 +248,7 @@ public: k_total.resize(setup_->rows.size(), 0.0f); d.resize(setup_->rows.size(), 0.0f); var.resize(setup_->rows.size(), 1.0f); - engine_->SetRowModel(k_total, fmask, d, var, rows.twin_fraction); + engine_->SetRowModel(k_total, fmask, d, var, rows.twin_fractions); } bool OnGPU() const override { return true; } diff --git a/image_analysis/structure_refinement/RefinementGPU.cu b/image_analysis/structure_refinement/RefinementGPU.cu index 866c7c2f2..91f789990 100644 --- a/image_analysis/structure_refinement/RefinementGPU.cu +++ b/image_analysis/structure_refinement/RefinementGPU.cu @@ -116,46 +116,71 @@ __global__ void compose_target_kernel(const float2 *f1, float norm, int nu, int coef[m] = make_float2(static_cast(cr), static_cast(ci)); } -// With a twin law, the target's term of a working row against the twinned model, -// |F_model| = k sqrt((1-a) |U(h)|^2 + a |U(Th)|^2) with U = F + fmask, from F of every row (compose_target_kernel -// without `evaluate`); and the conjugates of its derivatives with respect to U(h) and to U(Th), unscaled, in `own` -// and `other` - zero for a row that is not a working observation. +// With twin laws, the target's term of a working row against the twinned model, +// |F_model| = k sqrt(a_0 |U(h)|^2 + sum_d a_d |U(T_d h)|^2) with U = F + fmask and a_0 = 1 - sum_d a_d, from F of every +// row (compose_target_kernel without `evaluate`); and the conjugates of its derivatives with respect to U(h), in +// `own`, and to U(T_d h), in `other` (law * rows + row) - a mate that is the row itself counted in `own` - unscaled, +// zero for a row that is not a working observation. __global__ void twin_target_kernel(const float2 *f, const float2 *fmask, const int *mate, const float *twin_fraction, - int rows, const float *k_total, const float *ml_d, const float *ml_var, + int laws, int rows, const float *k_total, const float *ml_d, const float *ml_var, const float *fo, const char *work, const char *centric, double *nll, float2 *own, float2 *other) { const int m = blockIdx.x * blockDim.x + threadIdx.x; if (m >= rows) return; double term = 0, g = 0; - const int j = mate[m]; const double u1r = f[m].x + fmask[m].x, u1i = f[m].y + fmask[m].y; - const double u2r = f[j].x + fmask[j].x, u2i = f[j].y + fmask[j].y; - const double a = twin_fraction[0]; + double a0 = 1.0; + for (int d = 0; d < laws; d++) + a0 -= twin_fraction[d]; if (work[m]) { const double k = k_total[m]; - const double fc = k * sqrt((1.0 - a) * (u1r * u1r + u1i * u1i) + a * (u2r * u2r + u2i * u2i)); + double i_model = a0 * (u1r * u1r + u1i * u1i); + for (int d = 0; d < laws; d++) { + const int j = mate[static_cast(d) * rows + m]; + const double u2r = f[j].x + fmask[j].x, u2i = f[j].y + fmask[j].y; + i_model += twin_fraction[d] * (u2r * u2r + u2i * u2i); + } + const double fc = k * sqrt(fmax(i_model, 0.0)); double dfc = 0; term = refine_math::AmplitudeNegLogLikelihood(fo[m], fc, ml_d[m], ml_var[m], centric[m], dfc); if (fc > 0) g = dfc * k * k / fc; } nll[m] = term; - own[m] = make_float2(static_cast(g * (1.0 - a) * u1r), static_cast(-g * (1.0 - a) * u1i)); - other[m] = make_float2(static_cast(g * a * u2r), static_cast(-g * a * u2i)); + double own_r = g * a0 * u1r, own_i = -g * a0 * u1i; + for (int d = 0; d < laws; d++) { + const int j = mate[static_cast(d) * rows + m]; + const double u2r = f[j].x + fmask[j].x, u2i = f[j].y + fmask[j].y; + const double ga = g * twin_fraction[d]; + if (j == m) { + own_r += ga * u2r; + own_i -= ga * u2i; + other[static_cast(d) * rows + m] = make_float2(0.0f, 0.0f); + } else + other[static_cast(d) * rows + m] = make_float2(static_cast(ga * u2r), + static_cast(-ga * u2i)); + } + own[m] = make_float2(static_cast(own_r), static_cast(own_i)); } -// The gradient's coefficient of every row under a twin law: its own term's derivative and that of the row whose -// mate it is - its own mate, the law being an involution - times the row's scale. -__global__ void twin_gather_kernel(const float2 *own, const float2 *other, const int *mate, const double *row_scale, - int rows, float2 *coef) { +// The gradient's coefficient of every row under twin laws: its own term's derivative and, per law, that of the +// observed row whose mate it is (mate_from), times the row's scale. +__global__ void twin_gather_kernel(const float2 *own, const float2 *other, const int *mate_from, int laws, + const double *row_scale, int rows, float2 *coef) { const int n = blockIdx.x * blockDim.x + threadIdx.x; if (n >= rows) return; const double scale = row_scale[n]; - const float2 a = own[n], b = other[mate[n]]; - coef[n] = make_float2(static_cast(scale * (static_cast(a.x) + b.x)), - static_cast(scale * (static_cast(a.y) + b.y))); + double cr = own[n].x, ci = own[n].y; + for (int d = 0; d < laws; d++) { + const int m = mate_from[static_cast(d) * rows + n]; + if (m >= 0) { + cr += other[static_cast(d) * rows + m].x; + ci += other[static_cast(d) * rows + m].y; + } + } + coef[n] = make_float2(static_cast(scale * cr), static_cast(scale * ci)); } // The gradient's coefficients on the half-u spectrum: each point the sum, in order, of the terms @@ -609,7 +634,7 @@ struct RefinementGPUCard { CudaDevicePtr ops, row_hkl; CudaDevicePtr fo; CudaDevicePtr work, centric; - CudaDevicePtr mate; // a twin law's (setup.mate non-empty) + CudaDevicePtr mate, mate_from; // the twin laws' (setup.mate non-empty), law * rows + row CudaDevicePtr scatter_point, scatter_start, scatter_term, scatter_phase; CudaDevicePtr phase; // exp(2 pi i h.t) over the integer h.t from phase_min int phase_min = 0; @@ -642,8 +667,10 @@ struct RefinementGPUCard { fo = CudaDevicePtr(nr, sync); work = CudaDevicePtr(nr, sync); centric = CudaDevicePtr(nr, sync); - if (!setup.mate.empty()) - mate = CudaDevicePtr(nr, sync); + if (!setup.mate.empty()) { + mate = CudaDevicePtr(setup.mate.size(), sync); + mate_from = CudaDevicePtr(setup.mate_from.size(), sync); + } scatter_point = CudaDevicePtr(std::max(setup.scatter_point.size(), 1), sync); scatter_start = CudaDevicePtr(std::max(setup.scatter_start.size(), 1), sync); scatter_term = CudaDevicePtr(std::max(setup.scatter_term.size(), 1), sync); @@ -684,8 +711,10 @@ struct RefinementGPUCard { Upload(fo, setup.fo, stream); Upload(work, setup.work, stream); Upload(centric, setup.centric, stream); - if (!setup.mate.empty()) + if (!setup.mate.empty()) { Upload(mate, setup.mate, stream); + Upload(mate_from, setup.mate_from, stream); + } Upload(scatter_point, setup.scatter_point, stream); Upload(scatter_start, setup.scatter_start, stream); Upload(scatter_term, term, stream); @@ -737,7 +766,7 @@ struct RefinementGPUEngineImpl { float *mask_grid; float2 *mask_spectrum; // the card's tables - const int *ops, *row_hkl, *mate, *scatter_point, *scatter_start, *scatter_term, *scatter_phase; + const int *ops, *row_hkl, *mate, *mate_from, *scatter_point, *scatter_start, *scatter_term, *scatter_phase; const float *fo; const char *work, *centric; const double2 *phase; @@ -745,7 +774,8 @@ struct RefinementGPUEngineImpl { CudaDevicePtr row_scale; CudaDevicePtr k_total, ml_d, ml_var; CudaDevicePtr fmask_scaled, f, fm, coef; - // a twin law's (setup.mate non-empty): the fraction, the two parts of each row's derivative + // the twin laws' (setup.mate non-empty): their fractions, the parts of each row's derivative (own: one per row, + // other: one per law and row) CudaDevicePtr twin_fraction; CudaDevicePtr coef_own, coef_other; CudaDevicePtr nll, nll_sum; @@ -763,7 +793,7 @@ struct RefinementGPUEngineImpl { RefinementGPUEngineImpl(int dev, const RefinementGPUSetup &s, std::shared_ptr c, size_t fft_work_bytes, bool own_lane) : setup(s), card(std::move(c)), work_bytes(fft_work_bytes), device(dev), ops(card->ops), row_hkl(card->row_hkl), - mate(card->mate), scatter_point(card->scatter_point), scatter_start(card->scatter_start), + mate(card->mate), mate_from(card->mate_from), scatter_point(card->scatter_point), scatter_start(card->scatter_start), scatter_term(card->scatter_term), scatter_phase(card->scatter_phase), fo(card->fo), work(card->work), centric(card->centric), phase(card->phase), phase_min(card->phase_min), lane(card->Lane(s, fft_work_bytes, own_lane)), grid(lane->grid), spectrum(lane->spectrum), @@ -790,9 +820,9 @@ struct RefinementGPUEngineImpl { nll = CudaDevicePtr(nr, sync); nll_sum = CudaDevicePtr(1, sync); if (!setup.mate.empty()) { - twin_fraction = CudaDevicePtr(1, sync); + twin_fraction = CudaDevicePtr(setup.laws, sync); coef_own = CudaDevicePtr(nr, sync); - coef_other = CudaDevicePtr(nr, sync); + coef_other = CudaDevicePtr(static_cast(setup.laws) * nr, sync); } reduce_bytes = ReduceBytes(nr); reduce_temp = CudaDevicePtr(std::max(reduce_bytes, 1), sync); @@ -852,11 +882,12 @@ struct RefinementGPUEngineImpl { Compose(true); else if (rows > 0) { Compose(false); - twin_target_kernel<<>>(f, fmask_scaled, mate, twin_fraction, rows, k_total, + twin_target_kernel<<>>(f, fmask_scaled, mate, twin_fraction, setup.laws, rows, k_total, ml_d, ml_var, fo, work, centric, nll, coef_own, coef_other); cuda_err(cudaGetLastError()); - twin_gather_kernel<<>>(coef_own, coef_other, mate, row_scale, rows, coef); + twin_gather_kernel<<>>(coef_own, coef_other, mate_from, setup.laws, + row_scale, rows, coef); cuda_err(cudaGetLastError()); } if (rows > 0) { @@ -960,7 +991,7 @@ size_t RefinementGPUEngine::CardBytes(const RefinementGPUSetup &s) { const size_t nr = s.rows.size() + 1; size_t bytes = 12 * s.sym_ops.size() * sizeof(int) + nr * (3 * sizeof(int) + sizeof(float) + 2); if (!s.mate.empty()) - bytes += nr * sizeof(int); + bytes += (s.mate.size() + s.mate_from.size()) * sizeof(int); bytes += (s.scatter_point.size() + s.scatter_start.size() + 2 * s.scatter_term.size()) * sizeof(int); int phase_min = 0, phase_max = 0; PhaseRange(s, phase_min, phase_max); @@ -975,7 +1006,7 @@ size_t RefinementGPUEngine::DeviceBytes(const RefinementGPUSetup &s, size_t reco sizeof(ModelDensityAtom)); bytes += nr * (2 * sizeof(double) + 3 * sizeof(float) + 4 * sizeof(float2)); if (!s.mate.empty()) - bytes += nr * 2 * sizeof(float2); + bytes += nr * (1 + static_cast(s.laws)) * sizeof(float2); bytes += ModelDensityGPU::DeviceBytes(s.max_atoms, s.max_pairs, ModelDensityGPU::Bricks(s.grid.nu, s.grid.nv, s.grid.nw)); // the two restraint sets of SetRestraints(), together up to these records (plane atoms counted as records) and // gradient slots @@ -1013,11 +1044,13 @@ void RefinementGPUEngine::SetBlur(double blur) { } void RefinementGPUEngine::SetRowModel(const std::vector &k_total, const std::vector> &fmask, - const std::vector &d, const std::vector &var, float twin_fraction) { + const std::vector &d, const std::vector &var, + const std::vector &twin_fractions) { RefinementGPUEngineImpl &e = *impl_; DeviceGuard guard(e.device); Upload(e.k_total, k_total, e.stream); - const std::vector fraction{twin_fraction}; // kept until the wait below + std::vector fraction = twin_fractions; // kept until the wait below + fraction.resize(e.setup.laws, 0.0f); if (e.twin_fraction) Upload(e.twin_fraction, fraction, e.stream); if (!fmask.empty()) diff --git a/image_analysis/structure_refinement/RefinementGPUEngine.h b/image_analysis/structure_refinement/RefinementGPUEngine.h index fe7bc73c3..978dc8a23 100644 --- a/image_analysis/structure_refinement/RefinementGPUEngine.h +++ b/image_analysis/structure_refinement/RefinementGPUEngine.h @@ -32,9 +32,10 @@ struct RefinementGPUSetup { std::vector inv_d2; std::vector fo; std::vector work, centric; - // With a twin law: per row the row of its twin mate (RefinementReflections::mate), an involution; empty - // without one. - std::vector mate; + // With twin laws: per law and row (law * rows + row) the row of its twin mate under the law, and the observed + // row whose mate it is (-1: none) - RefinementReflections::mates and mates_from; empty without a law. + int laws = 0; + std::vector mate, mate_from; // The adjoint of the composition, as a gather: the half-u spectrum points the gradient's coefficients // land on (ascending), and for each the terms (row * ops + op) * 2 + conj adding to it, in order. std::vector scatter_point, scatter_start, scatter_term; @@ -65,10 +66,11 @@ public: // n_cen exp(blur s^2 / 4) per reflection: prepare_asu_data()'s unblur. void SetBlur(double blur); - // Per macro-cycle: F_model = k_total (F + fmask), the likelihood's D and variance, and with a twin law the twin - // fraction (XrayRowModel). + // Per macro-cycle: F_model = k_total (F + fmask), the likelihood's D and variance, and with twin laws their + // fractions (XrayRowModel). void SetRowModel(const std::vector &k_total, const std::vector> &fmask, - const std::vector &d, const std::vector &var, float twin_fraction = 0); + const std::vector &d, const std::vector &var, + const std::vector &twin_fractions = {}); // The atoms' densities, as ModelStructureFactorsGPU takes them (the boxes of ModelDensityGPU::Boxes() made // from them): called again only where some atom's density changed. Read until the call returns; from pinned diff --git a/image_analysis/structure_refinement/RefinementXray.cpp b/image_analysis/structure_refinement/RefinementXray.cpp index 350513566..792c35d7d 100644 --- a/image_analysis/structure_refinement/RefinementXray.cpp +++ b/image_analysis/structure_refinement/RefinementXray.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -139,7 +140,10 @@ public: // The target and, per reflection, dT/dF of the atoms' structure factor (dT/dRe F + i dT/dIm F). std::vector> dt_df(n, 0.0); double sum = 0; - const double a = rows_.twin_fraction; + const size_t laws = refl_.Laws(); + double own_fraction = 1.0; + for (float a : rows_.twin_fractions) + own_fraction -= a; for (size_t m = 0; m < refl_.Size(); m++) { if (refl_.free[m]) continue; @@ -154,19 +158,25 @@ public: dt_df[m] = k * dfc * fm / fc; continue; } - // twinned: |F_model| = k sqrt((1-a) |U(h)|^2 + a |U(Th)|^2), U = F + fmask, so that - // dT/dU(h) = dT/d|F_model| k^2 (1-a) U(h) / |F_model| and dT/dU(Th) the same with a U(Th) - const int j = refl_.mate[m]; + // twinned: |F_model| = k sqrt(a_0 |U(h)|^2 + sum_d a_d |U(T_d h)|^2), U = F + fmask, so that + // dT/dU(T_d h) = dT/d|F_model| k^2 a_d U(T_d h) / |F_model| const std::complex u1 = f[m] + std::complex(rows_.fmask[m]); - const std::complex u2 = f[j] + std::complex(rows_.fmask[j]); - const double fc = k * std::sqrt((1.0 - a) * std::norm(u1) + a * std::norm(u2)); + double i_model = own_fraction * std::norm(u1); + for (size_t d = 0; d < laws; d++) { + const int j = refl_.mates[d][m]; + i_model += rows_.twin_fractions[d] * std::norm(f[j] + std::complex(rows_.fmask[j])); + } + const double fc = k * std::sqrt(std::max(i_model, 0.0)); double dfc = 0; sum += refine_math::AmplitudeNegLogLikelihood(refl_.fo[m], fc, rows_.d[m], rows_.var[m], refl_.centric[m], dfc); if (fc > 0) { const double g = dfc * k * k / fc; - dt_df[m] += g * (1.0 - a) * u1; - dt_df[j] += g * a * u2; + dt_df[m] += g * own_fraction * u1; + for (size_t d = 0; d < laws; d++) { + const int j = refl_.mates[d][m]; + dt_df[j] += g * double(rows_.twin_fractions[d]) * (f[j] + std::complex(rows_.fmask[j])); + } } } if (grad == nullptr) @@ -311,44 +321,63 @@ std::vector RefinementTwinLaws(const gemmi::UnitCell &cell, const gem return laws; } -int SetTwinLaw(RefinementReflections &refl, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, - const gemmi::Op &law) { +std::vector RefinementTwinDomains(const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg) { + std::vector laws; + try { + for (const gemmi::Op &op : gemmi::find_twin_laws(cell, &sg, 3.0, false)) + laws.push_back(gemmi::Op{op.rot, {0, 0, 0}, 'x'}); + } catch (const std::exception &) { + } + return laws; +} + +int SetTwinLaws(RefinementReflections &refl, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, + const std::vector &laws) { const gemmi::GroupOps gops = sg.operations(); const gemmi::ReciprocalAsu asu(&sg); const size_t n = refl.Size(); std::map row; for (size_t m = 0; m < n; m++) row[refl.hkl[m]] = static_cast(m); - // each observed reflection's mate in the ASU, and those not observed - where the grid composes them - as rows + // each observed reflection's mates in the ASU, and those not observed - where the grid composes them - as rows // of their own - std::vector mate_of(n); + std::vector> mate_of(laws.size(), std::vector(n)); std::set unobserved; - for (size_t m = 0; m < n; m++) { - mate_of[m] = asu.to_asu(law.apply_to_hkl(refl.hkl[m]), gops).first; - if (row.find(mate_of[m]) == row.end()) - unobserved.insert(mate_of[m]); - } + for (size_t d = 0; d < laws.size(); d++) + for (size_t m = 0; m < n; m++) { + mate_of[d][m] = asu.to_asu(laws[d].apply_to_hkl(refl.hkl[m]), gops).first; + if (row.find(mate_of[d][m]) == row.end()) + unobserved.insert(mate_of[d][m]); + } const std::vector candidates(unobserved.begin(), unobserved.end()); const SymmetryComposition composition(RigidBodyZoneGrid(cell, sg, d_min), d_min, candidates); - refl.twin_law = law; + refl.twin_laws = laws; refl.mate_hkl.clear(); for (size_t i = 0; i < candidates.size(); i++) if (composition.Row()[i] >= 0) { row[candidates[i]] = static_cast(n + refl.mate_hkl.size()); refl.mate_hkl.push_back(candidates[i]); } - refl.mate.assign(refl.Rows(), -1); - for (size_t m = 0; m < n; m++) { - const auto it = row.find(mate_of[m]); - refl.mate[m] = it == row.end() ? static_cast(m) : it->second; - if (refl.mate[m] >= static_cast(n)) - refl.mate[refl.mate[m]] = static_cast(m); - } + const size_t rows = refl.Rows(); + refl.mates.assign(laws.size(), std::vector(rows)); + refl.mates_from.assign(laws.size(), std::vector(rows, -1)); int split = 0; - for (size_t m = 0; m < n; m++) { - const int j = refl.mate[m]; - if (j > static_cast(m) && j < static_cast(n) && refl.free[m] != refl.free[j]) - split++; + for (size_t d = 0; d < laws.size(); d++) { + std::vector &mate = refl.mates[d]; + std::iota(mate.begin(), mate.end(), 0); + for (size_t m = 0; m < n; m++) { + const auto it = row.find(mate_of[d][m]); + if (it != row.end() && it->second != static_cast(m)) { + mate[m] = it->second; + refl.mates_from[d][it->second] = static_cast(m); + } + } + for (size_t m = 0; m < n; m++) { + const int j = mate[m]; + if (j != static_cast(m) && j < static_cast(n) && refl.free[m] != refl.free[j] && + (refl.mates[d][j] != static_cast(m) || j > static_cast(m))) + split++; + } } return split; } @@ -362,80 +391,152 @@ std::vector> TwinnedModel(const RefinementReflections &refl, out[m] = rows.k_total[m] * (f[m] + rows.fmask[m]); return out; } - const double a = rows.twin_fraction; + double own_fraction = 1.0; + for (float a : rows.twin_fractions) + own_fraction -= a; for (size_t m = 0; m < n; m++) { - const int j = refl.mate[m]; const std::complex u1 = std::complex(f[m]) + std::complex(rows.fmask[m]); - const std::complex u2 = std::complex(f[j]) + std::complex(rows.fmask[j]); const double i1 = std::norm(u1); - const double amplitude = rows.k_total[m] * std::sqrt((1.0 - a) * i1 + a * std::norm(u2)); + double i_model = own_fraction * i1; + for (size_t d = 0; d < refl.Laws(); d++) { + const int j = refl.mates[d][m]; + i_model += rows.twin_fractions[d] * + std::norm(std::complex(f[j]) + std::complex(rows.fmask[j])); + } + const double amplitude = rows.k_total[m] * std::sqrt(std::max(i_model, 0.0)); out[m] = i1 > 0 ? std::complex(u1 * (amplitude / std::sqrt(i1))) : std::complex(amplitude, 0); } return out; } -double FitTwinFraction(const RefinementReflections &refl, const XrayRowModel &rows, - const std::vector> &f) { - if (!refl.Twinned()) - return 0; - // the working reflections whose mate is another reflection: Fo^2 and the model's intensity at h and at Th +std::vector FitTwinFractions(const RefinementReflections &refl, const XrayRowModel &rows, + const std::vector> &f) { + const size_t laws = refl.Laws(); + std::vector out(laws, 0.0f); + if (laws == 0) + return out; + // the working reflections with a mate that is another reflection: Fo^2 and the model's intensity at h and at + // each T_d h + const size_t nx = laws + 1; struct Term { - double inv_d2, io, i1, i2; + double inv_d2, io; + std::vector x; // the model's intensity at h, then at T_d h }; std::vector terms; for (size_t m = 0; m < refl.Size(); m++) { - const int j = refl.mate[m]; - if (refl.free[m] || j == static_cast(m)) + if (refl.free[m]) + continue; + bool related = false; + for (size_t d = 0; d < laws; d++) + related |= refl.mates[d][m] != static_cast(m); + if (!related) continue; const double k2 = double(rows.k_total[m]) * rows.k_total[m]; - terms.push_back({refl.inv_d2[m], double(refl.fo[m]) * refl.fo[m], - k2 * std::norm(std::complex(f[m]) + std::complex(rows.fmask[m])), - k2 * std::norm(std::complex(f[j]) + std::complex(rows.fmask[j]))}); + Term t{refl.inv_d2[m], double(refl.fo[m]) * refl.fo[m], std::vector(nx)}; + t.x[0] = k2 * std::norm(std::complex(f[m]) + std::complex(rows.fmask[m])); + for (size_t d = 0; d < laws; d++) { + const int j = refl.mates[d][m]; + t.x[d + 1] = k2 * std::norm(std::complex(f[j]) + std::complex(rows.fmask[j])); + } + terms.push_back(std::move(t)); } - if (terms.size() < 3) - return 0; - // normalised per resolution shell of TWIN_FRACTION_SHELL reflections: Fo^2 by its mean, the model's by its own + if (terms.size() < nx + 2) + return out; + // normalised per resolution shell of TWIN_FRACTION_SHELL reflections: Fo^2 by its mean, the model's by the mean + // of its own intensity std::stable_sort(terms.begin(), terms.end(), [](const Term &a, const Term &b) { return a.inv_d2 < b.inv_d2; }); const size_t shells = std::max(1, terms.size() / TWIN_FRACTION_SHELL); - // the normal equations of y = b0 + b1 x1 + b2 x2 - double n[3][3] = {}, r[3] = {}; + // the normal equations of y = b0 + sum_p b_p x_p, unknowns 0 (the intercept) to nx + const size_t nu = nx + 1; + std::vector nm(nu * nu, 0.0), r(nu, 0.0); + std::vector x(nu); for (size_t k = 0; k < shells; k++) { const size_t lo = k * terms.size() / shells, hi = (k + 1) * terms.size() / shells; double so = 0, sc = 0; for (size_t i = lo; i < hi; i++) { so += terms[i].io; - sc += terms[i].i1; + sc += terms[i].x[0]; } if (so <= 0 || sc <= 0) continue; for (size_t i = lo; i < hi; i++) { - const double x[3] = {1.0, terms[i].i1 / sc, terms[i].i2 / sc}, y = terms[i].io / so; - for (int p = 0; p < 3; p++) { + x[0] = 1.0; + for (size_t p = 0; p < nx; p++) + x[p + 1] = terms[i].x[p] / sc; + const double y = terms[i].io / so; + for (size_t p = 0; p < nu; p++) { r[p] += x[p] * y; - for (int q = 0; q < 3; q++) - n[p][q] += x[p] * x[q]; + for (size_t q = 0; q < nu; q++) + nm[p * nu + q] += x[p] * x[q]; } } } - // Cramer's rule for b1 and b2 - auto det3 = [](const double m[3][3]) { - return m[0][0] * (m[1][1] * m[2][2] - m[1][2] * m[2][1]) - m[0][1] * (m[1][0] * m[2][2] - m[1][2] * m[2][0]) + - m[0][2] * (m[1][0] * m[2][1] - m[1][1] * m[2][0]); - }; - const double d = det3(n); - if (!(std::fabs(d) > 0)) - return 0; - double b[3]; - for (int c = 1; c < 3; c++) { - double m[3][3]; - for (int p = 0; p < 3; p++) - for (int q = 0; q < 3; q++) - m[p][q] = q == c ? r[p] : n[p][q]; - b[c] = det3(m) / d; + // solved over the unknowns still in, by Gaussian elimination; a law whose coefficient is negative is taken out + // (the most negative first) and the rest solved again + std::vector in(nu, 1); + std::vector b(nu, 0.0); + for (size_t round = 0; round < nu; round++) { + std::vector idx; + for (size_t p = 0; p < nu; p++) + if (in[p]) + idx.push_back(p); + const size_t k = idx.size(); + std::vector a(k * (k + 1)); + for (size_t p = 0; p < k; p++) { + for (size_t q = 0; q < k; q++) + a[p * (k + 1) + q] = nm[idx[p] * nu + idx[q]]; + a[p * (k + 1) + k] = r[idx[p]]; + } + bool singular = false; + for (size_t c = 0; c < k && !singular; c++) { + size_t piv = c; + for (size_t p = c + 1; p < k; p++) + if (std::fabs(a[p * (k + 1) + c]) > std::fabs(a[piv * (k + 1) + c])) + piv = p; + if (!(std::fabs(a[piv * (k + 1) + c]) > 0)) { + singular = true; + break; + } + for (size_t q = 0; q <= k; q++) + std::swap(a[c * (k + 1) + q], a[piv * (k + 1) + q]); + for (size_t p = 0; p < k; p++) + if (p != c) { + const double factor = a[p * (k + 1) + c] / a[c * (k + 1) + c]; + for (size_t q = c; q <= k; q++) + a[p * (k + 1) + q] -= factor * a[c * (k + 1) + q]; + } + } + if (singular) + return out; + std::fill(b.begin(), b.end(), 0.0); + for (size_t p = 0; p < k; p++) + b[idx[p]] = a[p * (k + 1) + k] / a[p * (k + 1) + p]; + size_t worst = 0; + for (size_t p = 2; p < nu; p++) + if (in[p] && b[p] < 0 && (worst == 0 || b[p] < b[worst])) + worst = p; + if (worst == 0) + break; + in[worst] = 0; } - if (!(b[1] + b[2] > 0)) - return 0; - return std::clamp(b[2] / (b[1] + b[2]), 0.0, 0.5); + double total = 0; + for (size_t p = 1; p < nu; p++) + total += b[p]; + if (!(total > 0)) + return out; + for (size_t d = 0; d < laws; d++) + out[d] = static_cast(std::max(0.0, b[d + 2] / total)); + if (laws == 1) + out[0] = std::min(out[0], 0.5f); + return out; +} + +double FitTwinFraction(const RefinementReflections &refl, const XrayRowModel &rows, + const std::vector> &f) { + double sum = 0; + for (float a : FitTwinFractions(refl, rows, f)) + sum += a; + return sum; } BulkSolventMask::BulkSolventMask(const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, diff --git a/image_analysis/structure_refinement/RefinementXray.h b/image_analysis/structure_refinement/RefinementXray.h index f1e259518..789e888c9 100644 --- a/image_analysis/structure_refinement/RefinementXray.h +++ b/image_analysis/structure_refinement/RefinementXray.h @@ -36,15 +36,18 @@ struct RefinementReflections { std::vector inv_d2; size_t Size() const { return hkl.size(); } - // With a twin law (SetTwinLaw) the structure factors are needed at every observed reflection's twin mate too: + // With twin laws (SetTwinLaws) the structure factors are needed at every observed reflection's twin mates too: // the mates not observed follow the observed rows as rows of their own, with no observation (mate_hkl), and - // `mate` holds per row, observed or not, the row of its mate - the row itself where the mate is the reflection - // or is not composed. A law whose square is in the group makes it an involution: mate[mate[m]] == m. + // `mates[d]` holds per row, observed or not, the row of its mate under law d - the row itself where the mate is + // the reflection, is not composed, or the row is not observed. `mates_from[d]` is the converse: per row, the + // observed row whose mate under law d it is (-1: none, or only itself), which the GPU gathers the gradient by. + // One law is a two-domain twin; several are as many domains more, each with its own fraction. // Empty without a law. - gemmi::Op twin_law = gemmi::Op::identity(); + std::vector twin_laws; std::vector mate_hkl; - std::vector mate; - bool Twinned() const { return !mate.empty(); } + std::vector> mates, mates_from; + bool Twinned() const { return !mates.empty(); } + size_t Laws() const { return mates.size(); } // The observed rows, then the mates': the rows the structure factors are computed at. size_t Rows() const { return hkl.size() + mate_hkl.size(); } std::vector RowHkl() const; @@ -63,24 +66,33 @@ RefinementReflections PrepareRefinementReflections(const std::vector RefinementTwinLaws(const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); -// `refl` with twin law `law` (RefinementReflections::mate). Returns how many twin pairs of observed reflections -// have one member in the working set and the other in the free set: with any, the free set is not independent -// of the working one under the law. -int SetTwinLaw(RefinementReflections &refl, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, - const gemmi::Op &law); +// Every domain operator of the lattice beyond `sg`: one rotation of the lattice's metric symmetry per coset of the +// group but its own (Le Page two-folds within 3 degrees, as RefinementTwinLaws), whatever its order - the domains a +// twin of the lattice can have, the laws of RefinementTwinLaws among them. +std::vector RefinementTwinDomains(const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg); + +// `refl` with twin laws `laws` (RefinementReflections::mates), one domain each beside the reflection's own. Returns +// how many twin pairs of observed reflections have one member in the working set and the other in the free set: +// with any, the free set is not independent of the working one under the laws. +int SetTwinLaws(RefinementReflections &refl, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, double d_min, + const std::vector &laws); +inline int SetTwinLaw(RefinementReflections &refl, const gemmi::UnitCell &cell, const gemmi::SpaceGroup &sg, + double d_min, const gemmi::Op &law) { + return SetTwinLaws(refl, cell, sg, d_min, {law}); +} // What turns the atoms' structure factors F into the model the target compares with the data, per reflection: // F_model = k_total (F + fmask), // with k_total the overall scale and anisotropic B and fmask the bulk solvent already scaled by k_sol // exp(-b_sol s^2/4); and the two parameters of the likelihood, D and the variance. Fixed for one macro-cycle. -// With a twin law, the model compared with an observation at h is the twinned one (TwinnedModel): -// |F_model(h)|^2 = k_total(h)^2 ((1-a) |F(h) + fmask(h)|^2 + a |F(Th) + fmask(Th)|^2), -// a the twin fraction; fmask is then per row of refl.Rows(), the other per observed row. +// With twin laws, the model compared with an observation at h is the twinned one (TwinnedModel): +// |F_model(h)|^2 = k_total(h)^2 ((1 - sum_d a_d) |F(h) + fmask(h)|^2 + sum_d a_d |F(T_d h) + fmask(T_d h)|^2), +// a_d the fraction of law d's domain; fmask is then per row of refl.Rows(), the other per observed row. struct XrayRowModel { std::vector k_total; std::vector> fmask; std::vector d, var; - float twin_fraction = 0; + std::vector twin_fractions; // per law of the reflections (RefinementReflections::twin_laws) // D per likelihood bin, where EstimateLikelihoodParameters() found it - the next fit's start (empty: none) std::vector bin_d; }; @@ -131,16 +143,20 @@ void EstimateLikelihoodParameters(const RefinementReflections &refl, const std:: std::vector> TwinnedModel(const RefinementReflections &refl, const XrayRowModel &rows, const std::vector> &f); -// The twin fraction a in [0, 1/2] the working observations ask of the model: Fo^2 regressed, with an intercept, on -// the model's intensities at h and at Th, all normalised per resolution shell of TWIN_FRACTION_SHELL reflections, -// a = b(Th) / (b(h) + b(Th)). The expectation of Fo^2 given the model is linear in the two, D^2 |F_model|^2 + Sigma, +// The twin fractions the working observations ask of the model, one per law: Fo^2 regressed, with an intercept, on +// the model's intensities at h and at each T_d h, all normalised per resolution shell of TWIN_FRACTION_SHELL +// reflections, a_d = b(T_d h) / (b(h) + sum_e b(T_e h)), a law whose coefficient comes out negative left out and the +// rest fitted again. The expectation of Fo^2 given the model is linear in them, D^2 |F_model|^2 + Sigma, // so an unrelated reflection takes a coefficient of 0 however poor the model is - where a least-squares fit of the // twinned amplitudes took a fraction of 0.16 for unrelated partners of one model at R-free 0.31, the average of two -// predictions being the better prediction. Reflections that are their own mate say nothing of a and are left out. `f` is F at every row, `rows` -// gives k_total and fmask. +// predictions being the better prediction. Reflections that are their own mate under every law say nothing and are +// left out. With one law the fraction is at most 1/2. `f` is F at every row, `rows` gives k_total and fmask. // Following Yeates (1997) Methods Enzymol. 276, 344-358 (the twinned intensity), refined as Murshudov et al. (2011) // Acta Cryst. D67, 355-367 refine the fractions, by least squares constexpr size_t TWIN_FRACTION_SHELL = 500; +std::vector FitTwinFractions(const RefinementReflections &refl, const XrayRowModel &rows, + const std::vector> &f); +// The sum of the fractions FitTwinFractions gives: the fraction of one law. double FitTwinFraction(const RefinementReflections &refl, const XrayRowModel &rows, const std::vector> &f); diff --git a/rugnux/ResultReport.cpp b/rugnux/ResultReport.cpp index 315567c94..bb5a96391 100644 --- a/rugnux/ResultReport.cpp +++ b/rugnux/ResultReport.cpp @@ -648,6 +648,18 @@ namespace { Add(s, KeyReal("REFINE_UNTWINNED_R_WORK", rf.untwinned_r_work, "{:.4f}")); Add(s, KeyReal("REFINE_UNTWINNED_R_FREE", rf.untwinned_r_free, "{:.4f}")); } + if (rf.twin_free_closed > 0) + Add(s, KeyInt("REFINE_TWIN_FREE_CLOSED", rf.twin_free_closed)); + if (!rf.twin_multi_decision.empty()) { + Add(s, KeyEnum("REFINE_TWIN_MULTI", rf.twin_multi_decision, {"MULTI_UNTWINNED", "MULTI_TWINNED"})); + Add(s, KeyText("REFINE_TWIN_MULTI_DOMAINS", rf.twin_multi_domains)); + if (std::isfinite(rf.twin_multi_r_free)) { // not where every added domain fell below the bar + Add(s, KeyReal("REFINE_TWIN_MULTI_R_WORK", rf.twin_multi_r_work, "{:.4f}")); + Add(s, KeyReal("REFINE_TWIN_MULTI_R_FREE", rf.twin_multi_r_free, "{:.4f}")); + Add(s, KeyReal("REFINE_TWIN_MULTI_R_FREE_NULL", rf.twin_multi_r_free_null, "{:.4f}")); + Add(s, KeyReal("REFINE_TWIN_MULTI_NULL_Z", rf.twin_multi_null_z, "{:.1f}")); + } + } int kinds[5] = {0, 0, 0, 0, 0}; for (const DensityBlob &b : rf.blobs) kinds[static_cast(b.kind)]++; diff --git a/tests/RefinementTest.cpp b/tests/RefinementTest.cpp index 942455a4e..16bf2c89f 100644 --- a/tests/RefinementTest.cpp +++ b/tests/RefinementTest.cpp @@ -1528,13 +1528,17 @@ TEST_CASE("RefinementTwin_laws_and_mates", "[Refinement]") { const gemmi::GroupOps gops = t.sg.operations(); const gemmi::ReciprocalAsu asu(&t.sg); const std::vector rows = t.refl.RowHkl(); - for (size_t m = 0; m < t.refl.Rows(); m++) { - const int j = t.refl.mate[m]; + REQUIRE(t.refl.Laws() == 1); + for (size_t m = 0; m < t.refl.Size(); m++) { + const int j = t.refl.mates[0][m]; CAPTURE(m, j); REQUIRE(j >= 0); - CHECK(t.refl.mate[j] == static_cast(m)); // an involution - if (m < t.refl.Size()) + if (j != static_cast(m)) { CHECK(rows[j] == asu.to_asu(t.law.apply_to_hkl(rows[m]), gops).first); + CHECK(t.refl.mates_from[0][j] == static_cast(m)); + if (j < static_cast(t.refl.Size())) + CHECK(t.refl.mates[0][j] == static_cast(m)); // an involution + } } // a free set drawn on the merging group alone splits twin pairs RefinementReflections split = t.refl; @@ -1543,6 +1547,39 @@ TEST_CASE("RefinementTwin_laws_and_mates", "[Refinement]") { CHECK(SetTwinLaw(split, t.cell, t.sg, 2.2, t.law) > 0); } +// The free set closed under the lattice's domains: one drawn on the holohedry is left as it is; one drawn on +// another cell's (P 2 2 2 on a tetragonal metric, its flags drawn on an orthorhombic one: twin pairs split) ends with +// every pair in one set and about the free fraction it had - where a majority of each pair would empty it, and a +// draw that repeats the flag of one member would inflate it. +TEST_CASE("RefinementTwin_free_set_closed", "[Refinement]") { + const gemmi::UnitCell cell{30, 40, 30, 90, 90, 90}; + const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 2 2 2"); + std::vector merged = asu_reflections(cell, sg, 1.5); + AssignRfreeFlags(merged, sg, 0.05, 0, UnitCell{30, 40, 30, 90, 90, 90}); + CHECK(CloseFreeSet(merged, cell, sg) == 0); + + AssignRfreeFlags(merged, sg, 0.05, 0, UnitCell{30, 40, 36, 90, 90, 90}); + const auto free_count = [&] { + return std::count_if(merged.begin(), merged.end(), [](const MergedReflection &r) { return r.rfree_flag; }); + }; + const auto before = free_count(); + CHECK(CloseFreeSet(merged, cell, sg) > 0); + CHECK(free_count() == Catch::Approx(before).epsilon(0.1)); + std::map flag; + const gemmi::GroupOps gops = sg.operations(); + const gemmi::ReciprocalAsu asu(&sg); + for (const MergedReflection &r : merged) + flag[asu.to_asu(gemmi::Miller{{r.h, r.k, r.l}}, gops).first] = r.rfree_flag; + int split = 0; + for (const MergedReflection &r : merged) + for (const gemmi::Op &law : RefinementTwinDomains(cell, sg)) { + const auto it = flag.find(asu.to_asu(law.apply_to_hkl({{r.h, r.k, r.l}}), gops).first); + split += it != flag.end() && it->second != r.rfree_flag; + } + CHECK(split == 0); + CHECK(CloseFreeSet(merged, cell, sg) == 0); +} + TEST_CASE("RefinementTwin_rule_with_data_evidence", "[Refinement]") { // the null alone: 3 standard errors CHECK(AdoptTwinLaw(0.1, 3.0, false)); @@ -1564,7 +1601,7 @@ TEST_CASE("RefinementTwin_gradient", "[Refinement]") { XrayRowModel rows; rows.k_total.assign(t.refl.Size(), 1.0f); rows.fmask.assign(t.refl.Rows(), {0, 0}); - rows.twin_fraction = 0.3f; + rows.twin_fractions = {0.3f}; std::vector> f; cpu->Fcalc(truth.models[0], f); REQUIRE(f.size() == t.refl.Rows()); @@ -1601,6 +1638,65 @@ TEST_CASE("RefinementTwin_gradient", "[Refinement]") { #endif } +TEST_CASE("RefinementTwin_domains_gradient", "[Refinement]") { + // P 2 on a tetragonal metric (a = c, beta = 90): three domain operators beside the identity, a two-fold whose + // square is in the group and two four-fold cosets + const gemmi::UnitCell cell{30, 40, 30, 90, 90, 90}; + const gemmi::SpaceGroup &sg = *gemmi::find_spacegroup_by_name("P 1 2 1"); + const std::vector laws = RefinementTwinDomains(cell, sg); + REQUIRE(laws.size() == 3); + std::vector merged = asu_reflections(cell, sg, 2.4); + int i = 0; + std::erase_if(merged, [&](const MergedReflection &r) { return r.k == 1 && i++ % 4 == 0; }); + AssignRfreeFlags(merged, sg, 0.05, 0, UnitCell{30, 40, 30, 90, 90, 90}); + RefinementReflections refl = PrepareRefinementReflections(merged, cell, sg, 2.4); + CHECK(SetTwinLaws(refl, cell, sg, 2.4, laws) == 0); + REQUIRE(refl.Laws() == 3); + CHECK(!refl.mate_hkl.empty()); + gemmi::Structure truth = random_crystal(cell, "P 1 2 1", 240, 27); + const gemmi::Model model = shaken(truth.models[0], 0.3, 4.0, 28); + auto cpu = MakeRefinementEngineCPU(cell, sg, 2.4, refl, 4); + cpu->SetBlur(RefinementBlur(cell, sg, 2.4, 5.0)); + XrayRowModel rows; + rows.k_total.assign(refl.Size(), 1.0f); + rows.fmask.assign(refl.Rows(), {0, 0}); + rows.twin_fractions = {0.2f, 0.1f, 0.15f}; + std::vector> f; + cpu->Fcalc(truth.models[0], f); + const std::vector> twinned = TwinnedModel(refl, rows, f); + for (size_t m = 0; m < refl.Size(); m++) + refl.fo[m] = std::abs(twinned[m]); + const std::vector fitted = FitTwinFractions(refl, rows, f); + REQUIRE(fitted.size() == 3); + for (size_t d = 0; d < 3; d++) + CHECK(fitted[d] == Catch::Approx(rows.twin_fractions[d]).margin(0.01)); + cpu->Fcalc(model, f); + EstimateLikelihoodParameters(refl, TwinnedModel(refl, rows, f), rows); + cpu->SetRowModel(rows); + check_xray_gradient(*cpu, model, 0.03); +#ifdef JFJOCH_USE_CUDA + if (get_gpu_count() > 0) { + auto gpu = MakeRefinementEngineGPU(0, cell, sg, 2.4, refl, model, 300.0, SIZE_MAX); + REQUIRE(gpu); + gpu->SetBlur(RefinementBlur(cell, sg, 2.4, 5.0)); + gpu->SetRowModel(rows); + std::vector> gc, gg; + const double tc = cpu->Evaluate(model, &gc); + const double tg = gpu->Evaluate(model, &gg); + CHECK(tg == Catch::Approx(tc).epsilon(1e-4)); + double max_g = 0; + for (const auto &g : gc) + for (int c = 0; c < 4; c++) + max_g = std::max(max_g, std::fabs(g[c])); + for (size_t a = 0; a < gc.size(); a++) + for (int c = 0; c < 4; c++) { + CAPTURE(a, c); + CHECK(std::fabs(gg[a][c] - gc[a][c]) < 0.01 * max_g); + } + } +#endif +} + TEST_CASE("Refinement_twin_decided", "[Refinement]") { TwinCase t; gemmi::Structure truth = residue_crystal({"PHE", "TRP", "DA", "DG", "TYR", "HOH", "HOH", "HOH"}, t.cell, "P 4", 21); @@ -1629,7 +1725,7 @@ TEST_CASE("Refinement_twin_decided", "[Refinement]") { XrayRowModel rows; rows.k_total.assign(t.refl.Size(), 1.0f); rows.fmask.assign(t.refl.Rows(), {0, 0}); - rows.twin_fraction = static_cast(fraction); + rows.twin_fractions = {static_cast(fraction)}; const std::vector> twinned = TwinnedModel(t.refl, rows, f); std::map amplitude; for (size_t m = 0; m < t.refl.Size(); m++)