Joint sweeps: the shared-cell distance re-run covers the sweeps the given cell rescued, and is adopted only beyond the errors
The distance re-run loop read each sweep's decision from the relation made before the -C re-runs, so every
sweep those runs rescued ("no merge"/"no axes" on the first look) was skipped and kept its header distance:
on a four-sweep set two rescued sweeps stayed at the header distance with cells ~3.5% short while the other
two had found a distance ~3% longer. The sweeps are now related again (and the target sweep chosen again)
after the -C re-runs, before any cell is compared.
A re-run replaced the first run whenever its held-out residual was not higher, however small the
difference: runs were taken at differences of 0.01-0.03% (inside the bootstrap error), one with 40% fewer
fulls and a cell overshooting the target, one of a second crystal whose cell legitimately differs. A re-run
is now taken only where its held-out residual is lower by more than twice the two residuals' errors combined,
and where it has no fewer fulls. The log gives both errors and both fulls counts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqXYdJDrM65gxsCn2PrbfK
This commit is contained in:
1 parent
595a94d4dc
commit
2489700184
2 files changed
+78
-64
No files matched your search
+6
-4
@@ -116,10 +116,12 @@ best; where the lattice allows more than one indexing, which one each sweep take
|
||||
of sweeps at once - and each is judged by its isomorphism ratio with it (see [RUGNUX_REPORT](RUGNUX_REPORT.md)): clearly
|
||||
about 1 it joins, clearly below it (another crystal) it is left out, and in between the merge decides - it
|
||||
joins unless it is shown to hurt: adding it raises the held-out residual of the others' merge by more than 1%,
|
||||
two standard errors down (a sweep left out is named in the log). The crystal has one cell: a joined sweep whose own cell is more than 0.3% off that of
|
||||
the joined sweep reaching the finest resolution (where the cell is measured rather than the distance) is
|
||||
processed a second time, from the distance that would make the two cells agree, and that run replaces the
|
||||
first where its intensities agree better with that sweep's (a lower held-out residual). Their fulls are
|
||||
two standard errors down (a sweep left out is named in the log). A sweep that found no lattice of the crystal on its own (it merged nothing, or correlates on no axes)
|
||||
is processed again given the cell of the joined sweep reaching the finest resolution. The crystal has one cell:
|
||||
any sweep on the reference's axes - the ones just rescued too - whose own cell is more than 0.3% off that sweep's
|
||||
(where the cell is measured rather than the distance) is processed a second time, from the distance that would
|
||||
make the two cells agree, and that run replaces the first only where its intensities agree better with the
|
||||
joined sweeps' (a lower held-out residual by more than twice its error) and it has no fewer fulls. Their fulls are
|
||||
scaled and merged together: a per-frame scale within each sweep, the sweeps' dose order read from
|
||||
their relative B factors (not from the input order or the header times), a decay along that order or
|
||||
one B per sweep, whichever predicts held-out frames better, then the space-group search, the
|
||||
|
||||
+72
-60
@@ -4181,70 +4181,82 @@ int RugnuxCli::RunJointSweeps(const RugnuxSweepPlan &plan) {
|
||||
inputs.push_back(plan.sweeps[k].input);
|
||||
}
|
||||
|
||||
// One cell for the crystal (SharedCellTarget): a sweep whose own cell disagrees with it - joined or not
|
||||
// on the first look - is processed again from the distance that would make the two agree, and that run replaces its first one where it
|
||||
// agrees better with the target sweep's intensities (lower held-out residual, HeldOutAgainst).
|
||||
// One cell for the crystal (SharedCellTarget). A sweep that found no lattice of this crystal on its own -
|
||||
// it merged nothing, or nothing that correlates with the reference on any axes - is processed again given
|
||||
// the target sweep's cell (-C): de novo indexing of one sweep can miss a large cell that the crystal's other
|
||||
// sweeps have measured. Its join is then decided on its intensities, as every sweep's is. Only a sweep that
|
||||
// failed is given the cell: given to every sweep from the start, it moved sweeps that index well de novo
|
||||
// into another setting or centring. Where -C was given, every sweep already had a cell.
|
||||
JointSweepRelation relation = RelateJointSweeps(members);
|
||||
bool replaced = false; // a sweep's run replaced by its run again: the relation is made again
|
||||
{
|
||||
const JointSweepRelation &first = relation;
|
||||
const int target = SharedCellTarget(members, first);
|
||||
// A sweep that found no lattice of this crystal on its own - it merged nothing, or nothing that
|
||||
// correlates with the reference on any axes - is processed again given the target sweep's cell (-C):
|
||||
// de novo indexing of one sweep can miss a large cell that the crystal's other sweeps have measured.
|
||||
// Its join is then decided on its intensities, as every sweep's is. Only a sweep that failed is given
|
||||
// the cell: given to every sweep from the start, it moved sweeps that index well de novo into another
|
||||
// setting or centring. Where -C was given, every sweep already had a cell.
|
||||
for (size_t k = 0; target >= 0 && !fixed_reference_unit_cell && k < members.size(); ++k) {
|
||||
const std::string &decision = first.sweeps[k].decision;
|
||||
if (decision != "no merge" && decision != "no axes")
|
||||
continue;
|
||||
logger.Info("Sweep {}: {}; processed again given sweep {}'s cell", k + 1, first.sweeps[k].why_left_out,
|
||||
target + 1);
|
||||
JointMemberSweep again = run_member(k, std::nullopt, members[target].cell, "_known");
|
||||
if (!again.merged.empty()) {
|
||||
members[k] = std::move(again);
|
||||
replaced = true;
|
||||
}
|
||||
int target = SharedCellTarget(members, relation);
|
||||
for (size_t k = 0; target >= 0 && !fixed_reference_unit_cell && k < members.size(); ++k) {
|
||||
const std::string &decision = relation.sweeps[k].decision;
|
||||
if (decision != "no merge" && decision != "no axes")
|
||||
continue;
|
||||
logger.Info("Sweep {}: {}; processed again given sweep {}'s cell", k + 1, relation.sweeps[k].why_left_out,
|
||||
target + 1);
|
||||
JointMemberSweep again = run_member(k, std::nullopt, members[target].cell, "_known");
|
||||
if (!again.merged.empty()) {
|
||||
members[k] = std::move(again);
|
||||
replaced = true;
|
||||
}
|
||||
for (size_t k = 0; target >= 0 && k < members.size(); ++k) {
|
||||
// Every sweep on the reference's axes, joined or not: a cell off by a few percent is itself a reason
|
||||
// for intensities to disagree, so the join is decided after this, on the runs it keeps.
|
||||
const std::string &decision = first.sweeps[k].decision;
|
||||
if (decision == "no merge" || decision == "no axes" || static_cast<int>(k) == target
|
||||
|| !std::isfinite(members[k].distance_mm))
|
||||
continue;
|
||||
const double scale = VolumeLength(members[target].cell) / VolumeLength(members[k].cell);
|
||||
if (std::fabs(scale - 1.0) <= SHARED_CELL_AGREEMENT)
|
||||
continue;
|
||||
const auto distance = static_cast<float>(members[k].distance_mm * scale);
|
||||
logger.Info("Sweep {}: its cell is {:+.2f}% off sweep {}'s; processed again from {:.2f} mm", k + 1,
|
||||
100.0 * (1.0 / scale - 1.0), target + 1, distance);
|
||||
JointMemberSweep again = run_member(k, distance, std::nullopt, "_cell");
|
||||
if (again.merged.empty())
|
||||
continue;
|
||||
// Against the merge of the other joined sweeps, the target first (the scale).
|
||||
std::vector<const JointMemberSweep *> base{&members[target]};
|
||||
std::vector<gemmi::Op> base_cob{first.sweeps[target].cob};
|
||||
for (size_t j = 0; j < members.size(); ++j)
|
||||
if (first.sweeps[j].included && j != k && static_cast<int>(j) != target) {
|
||||
base.push_back(&members[j]);
|
||||
base_cob.push_back(first.sweeps[j].cob);
|
||||
}
|
||||
const JointMemberSweep &reference = members[first.reference];
|
||||
const std::vector<HeldOutGain> gains = HeldOutAgainst(base, base_cob, reference, {&members[k], &again});
|
||||
const HeldOutGain &before = gains[0], &after = gains[1];
|
||||
const bool adopt = std::isfinite(after.change) && !(after.change >= before.change);
|
||||
logger.Info("Sweep {}: against the joined sweeps (scale: sweep {}), held-out residual {:+.2f}% at its own cell ({:.3f} {:.3f} {:.3f}), "
|
||||
"{:+.2f}% processed again ({:.3f} {:.3f} {:.3f}): {}", k + 1, target + 1,
|
||||
100.0 * before.change, members[k].cell.a, members[k].cell.b, members[k].cell.c,
|
||||
100.0 * after.change, again.cell.a, again.cell.b, again.cell.c,
|
||||
adopt ? "the second run is taken" : "the first run stands");
|
||||
if (adopt) {
|
||||
again.reprocessed_from_mm = members[k].distance_mm;
|
||||
members[k] = std::move(again);
|
||||
replaced = true;
|
||||
}
|
||||
// The sweeps just rescued are related to the others before their cells are compared, as every other sweep's.
|
||||
if (replaced) {
|
||||
relation = RelateJointSweeps(members);
|
||||
target = SharedCellTarget(members, relation);
|
||||
replaced = false;
|
||||
}
|
||||
|
||||
// Every sweep on the reference's axes, joined or not - the ones just rescued too - on the target sweep's
|
||||
// cell: a cell off by a few percent is itself a reason for intensities to disagree, so the join is decided
|
||||
// after this, on the runs it keeps. A sweep whose cell is off the target's is processed again from the
|
||||
// distance that would make the two agree, and that run replaces its first one only where it agrees better
|
||||
// with the joined sweeps' intensities (HeldOutAgainst) by more than the two residuals' errors, and at no loss
|
||||
// of observations: a distance is not changed on noise, and a run that integrates fewer reflections does not
|
||||
// measure them better.
|
||||
for (size_t k = 0; target >= 0 && k < members.size(); ++k) {
|
||||
const std::string &decision = relation.sweeps[k].decision;
|
||||
if (decision == "no merge" || decision == "no axes" || static_cast<int>(k) == target
|
||||
|| !std::isfinite(members[k].distance_mm))
|
||||
continue;
|
||||
const double scale = VolumeLength(members[target].cell) / VolumeLength(members[k].cell);
|
||||
if (std::fabs(scale - 1.0) <= SHARED_CELL_AGREEMENT)
|
||||
continue;
|
||||
const auto distance = static_cast<float>(members[k].distance_mm * scale);
|
||||
logger.Info("Sweep {}: its cell is {:+.2f}% off sweep {}'s; processed again from {:.2f} mm", k + 1,
|
||||
100.0 * (1.0 / scale - 1.0), target + 1, distance);
|
||||
JointMemberSweep again = run_member(k, distance, std::nullopt, "_cell");
|
||||
if (again.merged.empty())
|
||||
continue;
|
||||
// Against the merge of the other joined sweeps, the target first (the scale).
|
||||
std::vector<const JointMemberSweep *> base{&members[target]};
|
||||
std::vector<gemmi::Op> base_cob{relation.sweeps[target].cob};
|
||||
for (size_t j = 0; j < members.size(); ++j)
|
||||
if (relation.sweeps[j].included && j != k && static_cast<int>(j) != target) {
|
||||
base.push_back(&members[j]);
|
||||
base_cob.push_back(relation.sweeps[j].cob);
|
||||
}
|
||||
const JointMemberSweep &reference = members[relation.reference];
|
||||
const std::vector<HeldOutGain> gains = HeldOutAgainst(base, base_cob, reference, {&members[k], &again});
|
||||
const HeldOutGain &before = gains[0], &after = gains[1];
|
||||
const double gain = before.change - after.change;
|
||||
const bool better = std::isfinite(gain) && gain > 2.0 * std::hypot(before.se, after.se);
|
||||
const bool no_loss = again.fulls.size() >= members[k].fulls.size();
|
||||
logger.Info("Sweep {}: against the joined sweeps (scale: sweep {}), held-out residual {:+.2f}% +- {:.2f}% at "
|
||||
"its own cell ({:.3f} {:.3f} {:.3f}, {} fulls), {:+.2f}% +- {:.2f}% processed again ({:.3f} {:.3f} "
|
||||
"{:.3f}, {} fulls): {}", k + 1, target + 1, 100.0 * before.change, 100.0 * before.se,
|
||||
members[k].cell.a, members[k].cell.b, members[k].cell.c, members[k].fulls.size(),
|
||||
100.0 * after.change, 100.0 * after.se, again.cell.a, again.cell.b, again.cell.c,
|
||||
again.fulls.size(),
|
||||
!better ? "the first run stands - the second agrees no better beyond the errors"
|
||||
: !no_loss ? "the first run stands - the second has fewer fulls"
|
||||
: "the second run is taken");
|
||||
if (better && no_loss) {
|
||||
again.reprocessed_from_mm = members[k].distance_mm;
|
||||
members[k] = std::move(again);
|
||||
replaced = true;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user