RESULT-CHANGING: Refinement: the U at its balanced weight also where the U at a quarter lowers -ln L free

The displacement model is chosen on R-free (beyond the tie) and -ln L of the free reflections together, but the
fewer-trials rule ran the U at its balanced weight only where the U at a quarter of it lowered R-free. Where the
quarter fitted the free reflections better than where the weights ended without lowering R-free, the balanced weight
was left out, and with it the trial the choice would have kept: on a twinned refine-only case the quarter had the best
-ln L free of the displacement trials and an R-free a little above the isotropic one, so isotropic was kept, while
the balanced weight - run before the rule - lowers both and is kept. The condition is now the decision's own: R-free
or -ln L free lower than where the weights ended. The -ln L free there is logged.

Subset battery (5 refine-arm, 9 open, 2 inhouse sets) against rc181 e56b19abb: refined R-free identical on all 16;
the balanced-weight U trial ran on one more set (both twin arms, no choice changed); target evaluations
49023 -> 49339 (+0.6%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQqN9qRD57vnShzTC37

docs/CPU_DATA_ANALYSIS_REFINEMENT.md

@@ -313,8 +313,10 @@ are:
    as TLSMD cuts them (by dynamic programming over the least-squares fit of an isotropic TLS model, a quadratic B(r),
    to the refined B factors), each cut where it gives more groups than the one before; and a U per atom, at the U
    weight balanced as $w_{B,0}$ is (on the U shaken by 20% of its B) times $\{1/4, 1\}$. The cuts between one group
-   per chain and the finest run only where the finest fits the free set better (lower $-\ln L$) than one per chain;
-   on 49 refinements of 33 structures neither this nor leaving out the U weight $\times 4$ changed a choice taken. A
+   per chain and the finest run only where the finest fits the free set better (lower $-\ln L$) than one per chain,
+   and the U at its balanced weight only where the U at a quarter of it lowers R-free or $-\ln L$ of the free
+   reflections from where the weights ended; on 49 refinements of 33 structures neither the cuts' rule nor leaving
+   out the U weight $\times 4$ changed a choice taken. A
    trial displaces the one before it (the first: where the weights ended) only where its geometry is within bounds
    and both its R-free (beyond the tie) and the $-\ln L$ of the free reflections are lower: the free set tests the
    parameters a model adds, so the choice is made on the evidence of each run, not at a resolution (Merritt 2012).

image_analysis/structure_refinement/Refinement.cpp

@@ -2870,8 +2870,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
         //    The trials that would not change the choice are left out: the model's own TLS groups are tried only where
         //    one body per chain or the finest cut lowers R-free, the cuts between those only where also the finest
         //    fits the free set better, the U at its balanced weight only where the U at a quarter of it lowers R-free
-        //    (run ahead where the slots take them), and the U at four times its balanced weight not at all (on 49
-        //    refinements of 33 structures, no choice taken changed).
+        //    or the -ln L of the free reflections (run ahead where the slots take them), and the U at four times its
+        //    balanced weight not at all (on 49 refinements of 33 structures, no choice taken changed).
         if (settings.adp_hypotheses && state.ran && !ctx.Cancelled()) {
             const RefinementStage adp_stage("adp_hypotheses");
             ctx.SetPositions(state.pos);
@@ -2880,6 +2880,8 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
             ctx.Restore(state.fit);
             PlaceRiders(ctx, topo);
             state.trial.nll_free = ctx.Score().nll_free;
+            logger.Info("Refinement, displacements: from where the weights ended, R-free {:.4f}, -LL free {:.1f}",
+                        state.trial.r_free, state.trial.nll_free);
             std::vector<AdpHypothesis> hyps = AdpHypotheses(ctx, topo, input_tls);
             const double w = state.trial.weight, adp = state.trial.adp_weight > 0 ? state.trial.adp_weight : adp_w0;
             // the U weight balanced on the model with every refinable atom given a U
@@ -2955,13 +2957,19 @@ RefinementResult RefineOnce(const gemmi::Structure &placed, const std::vector<Me
             run(now);
             // the model's own groups where one body per chain or the finest cut lowers R-free from where the weights
             // ended, the cuts between where also the finest fits the free set better than one body per chain, the U
-            // at its balanced weight where the U at a quarter of it lowers R-free (on 364 refinements of the battery
-            // that changed the choice on 7, none by more than 0.003 in R-free, and left out 6 in 10 of these trials)
+            // at its balanced weight where the U at a quarter of it lowers R-free or -ln L free (R-free alone left out
+            // the trial that would have been kept where the quarter lowered -ln L free but not R-free)
             auto lowers = [&](int k) {
                 return k >= 0 && outcomes[k].ran && outcomes[k].trial.within_bounds &&
                        outcomes[k].trial.r_free < state.trial.r_free;
             };
-            const bool tls = lowers(coarsest) || lowers(finest), u_full = lowers(u_quarter);
+            // the U at a quarter fitting the free reflections better than where the weights ended asks for the
+            // balanced weight as well: the choice below is made on -ln L free as much as on R-free
+            auto fits_better = [&](int k) {
+                return k >= 0 && outcomes[k].ran && outcomes[k].trial.within_bounds &&
+                       outcomes[k].trial.nll_free < state.trial.nll_free;
+            };
+            const bool tls = lowers(coarsest) || lowers(finest), u_full = lowers(u_quarter) || fits_better(u_quarter);
             const bool cuts = tls && finest > coarsest && outcomes[finest].ran && outcomes[coarsest].ran &&
                               outcomes[finest].trial.nll_free < outcomes[coarsest].trial.nll_free;
             std::vector<char> needed(trials.size(), 0);