mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 66 Wiki Activity

Merge: per-crystal CC1/2-delta rejection (--reject-delta-cchalf)

Browse Source 
CrystFEL deltaCChalf-style per-crystal quality filter for heterogeneous serial
data. Each image is assigned to one CC1/2 half, so removing it perturbs only that
half's per-reflection means; deltaCChalf_i = CC1/2(all) - CC1/2(without image i).
A negative value means dropping the image RAISES CC1/2 (it disagrees with the
consensus). Images whose deltaCChalf is a low-side statistical outlier
(< mean - N*stddev) are skipped when merging. Reference-free.

Two passes over the retained integration outcomes; per-image contributions are
re-derived rather than stored, so memory stays O(unique reflections + images) for
full 200k-frame runs. New CLI flag --reject-delta-cchalf <N> (default: off).

Validation (jet FFBIDX +C+S, sigma4): removing 17/4000 (3 sigma) raises CC1/2
95.1->96.1%, CCref 54.9->55.2; 2 sigma -> 96.1/55.3. Dataset-appropriate: it HELPS
heterogeneous serial data (some crystals genuinely bad) but slightly trims a
homogeneous single rotation crystal (c2 94.6->93.8 - no bad crystals, the relative
cut still removes the tail), so it is opt-in. R-free is the real test (user's full
200k). Note: the reported overall N_obs still counts all observations; the exported
merge (and CC1/2) correctly exclude the rejected images.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-06-16 12:32:52 +02:00
co-authored by Claude Opus 4.8
parent f878fb9d5d
commit 545ebdf868
3 changed files with 129 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/jfjoch_process.cpp
+21 -3
View File
@@ -73,6 +73,7 @@ void print_usage() {
std::cout << " --scaling-high-resolution <num> High resolution limit for spot finding (default: no limit)" << std::endl;
std::cout << " --min-partiality <num> Minimum partiality to accept reflection (default: 0.02)" << std::endl;
std::cout << " --reject-outliers <num> Per-observation merge outlier rejection, N sigma from the per-reflection median (default: off; e.g. 6, XDS/DIALS-style)" << std::endl;
std::cout << " --reject-delta-cchalf <num> Per-crystal CC1/2-delta rejection: drop images with deltaCChalf below mean - N*stddev (default: off; e.g. 2.5)" << std::endl;
std::cout << " --min-image-cc <num> Per-image CC limit in percent (default: no limit)" << std::endl;
std::cout << " --scaling-iterations <num> Number of scaling iterations with no reference data (default: 3)" << std::endl;
std::cout << " --scaling-output <txt> Output format for scaling results mtz|cif|txt (default: mtz)" << std::endl;
@@ -101,7 +102,8 @@ enum {
OPT_BANDWIDTH,
OPT_INTEGRATION_RADIUS,
OPT_REJECT_OUTLIERS,
OPT_PROFILE_MULTIPLIER
OPT_PROFILE_MULTIPLIER,
OPT_REJECT_DELTA_CCHALF
};
static option long_options[] = {
@@ -140,6 +142,7 @@ static option long_options[] = {
{"bandwidth", required_argument, nullptr, OPT_BANDWIDTH},
{"integration-radius", required_argument, nullptr, OPT_INTEGRATION_RADIUS},
{"reject-outliers", required_argument, nullptr, OPT_REJECT_OUTLIERS},
{"reject-delta-cchalf", required_argument, nullptr, OPT_REJECT_DELTA_CCHALF},
{"profile-multiplier", required_argument, nullptr, OPT_PROFILE_MULTIPLIER},
{nullptr, 0, nullptr, 0}
};
@@ -323,6 +326,7 @@ int main(int argc, char **argv) {
std::optional<float> d_min_scale_merge;
std::optional<std::string> integration_radius_arg; // "r1" or "r1,r2,r3"
std::optional<double> outlier_reject_nsigma; // merge per-observation outlier rejection
std::optional<double> delta_cchalf_nsigma; // per-crystal CC1/2-delta rejection
std::optional<float> profile_multiplier; // PixelRefine Term-2 profile-width multiplier
if (argc == 1) {
@@ -519,6 +523,9 @@ int main(int argc, char **argv) {
case OPT_REJECT_OUTLIERS:
outlier_reject_nsigma = std::stod(optarg);
break;
case OPT_REJECT_DELTA_CCHALF:
delta_cchalf_nsigma = std::stod(optarg);
break;
case OPT_PROFILE_MULTIPLIER:
profile_multiplier = std::stof(optarg);
break;
@@ -1057,8 +1064,19 @@ int main(int argc, char **argv) {
const double isa = (b > 0.0) ? 1.0 / b : std::numeric_limits<double>::infinity();
logger.Info("Error model: sigma'^2 = {:.3f} sigma^2 + ({:.4f} I)^2 ISa = {:.1f}", a, b, isa);
}
for (auto &i : indexer.GetIntegrationOutcome())
merge_engine.AddImage(i);
const auto &merge_outcomes = indexer.GetIntegrationOutcome();
std::vector<char> dcch_reject;
if (delta_cchalf_nsigma) {
dcch_reject = merge_engine.DeltaCChalfReject(merge_outcomes, *delta_cchalf_nsigma);
const size_t nrej = std::count(dcch_reject.begin(), dcch_reject.end(), static_cast<char>(1));
logger.Info("CC1/2-delta rejection (deltaCChalf < mean - {:.1f} stddev) removed {} / {} images",
*delta_cchalf_nsigma, nrej, dcch_reject.size());
}
for (size_t i = 0; i < merge_outcomes.size(); ++i) {
if (!dcch_reject.empty() && dcch_reject[i])
continue;
merge_engine.AddImage(merge_outcomes[i]);
}
if (merge_engine.RejectedCount() > 0)
logger.Info("Outlier rejection (>{:.1f} sigma from the per-reflection median) removed {} observations",
experiment.GetScalingSettings().GetOutlierRejectNsigma(), merge_engine.RejectedCount());