v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

viewer/widgets/JFJochViewerSettingsDock.cpp

@@ -265,6 +265,10 @@ QWidget *JFJochViewerSettingsDock::BuildMXPage() {
     refine->addItem("None", static_cast<int>(GeomRefinementAlgorithmEnum::None));
     refine->addItem("Orientation only", static_cast<int>(GeomRefinementAlgorithmEnum::OrientationOnly));
     refine->addItem("Beam center + lattice", static_cast<int>(GeomRefinementAlgorithmEnum::BeamCenter));
+    // "Flex" (the CLI's -r flex): tries all per-image refinements and keeps whichever indexes the most
+    // spots — lets the pipeline decide. Named "Flex", not "multi", to avoid the CrystFEL sense of multi
+    // (multi-lattice integration).
+    refine->addItem("Flex (best per-image refinement)", static_cast<int>(GeomRefinementAlgorithmEnum::Flex));
     refine->setCurrentIndex(refine->findData(static_cast<int>(indexing_.GetGeomRefinementAlgorithm())));
     // One high-level mode switch instead of separate partiality / rot3d / rotation-indexing options:
     // for a rotation dataset, unchecked = the rotation good-path (rotation indexing + Rotation
@@ -442,8 +446,30 @@ QWidget *JFJochViewerSettingsDock::BuildBraggSection() {
     auto *r3 = new NumberLineEdit(1.0f, 30.0f, bragg_.GetR3(), 1, "px", this);
     auto *radii = new QHBoxLayout();
     radii->addWidget(r1); radii->addWidget(r2); radii->addWidget(r3);
+    auto *stillPartiality = new QCheckBox("Stills partiality (experimental)", this);
+    stillPartiality->setChecked(bragg_.GetStillPartiality());
+    stillPartiality->setToolTip("Experimental, stills only: weight each reflection by a Gaussian "
+                                "excitation-error partiality exp(-d_ewald²/2σ²) instead of treating it as a "
+                                "full. Pairs with \"Partiality uncertainty\" in Scaling.");
+    // Background trim: replace the r2..r3 ring mean with a symmetric trimmed mean (drop the lowest and
+    // highest fraction of ring pixels), which removes the high-side bias that over-subtracts weak
+    // high-angle reflections. Checkbox + fraction; 0 = plain mean.
+    const float trim = bragg_.GetBackgroundTrimFraction();
+    auto *bkgTrim = new QCheckBox("Background trim", this);
+    bkgTrim->setChecked(trim > 0.0f);
+    bkgTrim->setToolTip("Estimate the local Bragg background with a symmetric trimmed mean of the "
+                        "background ring (drop the lowest and highest fraction of pixels) instead of the "
+                        "plain mean, removing the high-side bias that over-subtracts weak high-angle "
+                        "reflections. 0.10 recommended; unchecked = plain mean.");
+    auto *bkgTrimFrac = new NumberLineEdit(0.01f, 0.49f, trim > 0.0f ? trim : 0.10f, 2, "", this);
+    bkgTrimFrac->setEnabled(bkgTrim->isChecked());
+    auto *trimRow = new QHBoxLayout();
+    trimRow->addWidget(bkgTrim);
+    trimRow->addWidget(bkgTrimFrac, 1);
     form->addRow("", gaussian);
     form->addRow("Radii r1/r2/r3", radii);
+    form->addRow("", stillPartiality);
+    form->addRow("", trimRow);
     section->setContentLayout(form);
     section->setExpanded(false);   // folded on start (only geometry + unit cell start open)
 
@@ -451,9 +477,15 @@ QWidget *JFJochViewerSettingsDock::BuildBraggSection() {
         bragg_.Integrator(gaussian->isChecked() ? IntegratorMode::ProfileGaussian : IntegratorMode::BoxSum);
         bragg_.R1(static_cast<float>(r1->value())).R2(static_cast<float>(r2->value()))
               .R3(static_cast<float>(r3->value()));
+        bragg_.StillPartiality(stillPartiality->isChecked());
+        bragg_.BackgroundTrimFraction(bkgTrim->isChecked() ? static_cast<float>(bkgTrimFrac->value()) : 0.0f);
         emit braggChanged(bragg_);
     };
     connect(gaussian, &QCheckBox::toggled, this, [emitBragg] { emitBragg(); });
+    connect(stillPartiality, &QCheckBox::toggled, this, [emitBragg] { emitBragg(); });
+    connect(bkgTrim, &QCheckBox::toggled, this,
+            [bkgTrim, bkgTrimFrac, emitBragg] { bkgTrimFrac->setEnabled(bkgTrim->isChecked()); emitBragg(); });
+    connect(bkgTrimFrac, &NumberLineEdit::newValue, this, [emitBragg] { emitBragg(); });
     for (auto *f : {r1, r2, r3})
         connect(f, &NumberLineEdit::newValue, this, [emitBragg] { emitBragg(); });
     return section;
@@ -475,20 +507,42 @@ QWidget *JFJochViewerSettingsDock::BuildScalingSection() {
     corrections->setToolTip("Rotation only: fit a radiation-damage decay and a goniometer-frame absorption "
                             "surface on the fulls. Cross-validated, so they no-op when their systematic is "
                             "absent. On by default.");
+    auto *modulation = new QCheckBox("Detector-plane modulation (stills)", this);
+    modulation->setChecked(scaling_.GetStillsModulation());
+    modulation->setToolTip("Stills: fit a detector-plane modulation (flat-field) surface over where each "
+                           "reflection lands, cross-validated so it no-ops when the systematic is absent. "
+                           "For rotation, modulation is part of \"Correction surfaces\" above.");
     auto *limitRes = new QCheckBox("High-resolution limit", this);
     limitRes->setChecked(scaling_.GetHighResolutionLimit_A().has_value());
     auto *highRes = new NumberLineEdit(0.3f, 5.0f, scaling_.GetHighResolutionLimit_A().value_or(2.0), 1, "Å", this);
     highRes->setEnabled(limitRes->isChecked());
 
+    // Stills partiality-uncertainty merge term: adds a systematic sigma ~c*(1-partiality)*<I> on partials,
+    // so strong low-partiality partials are not over-trusted. Pairs with "Stills partiality" (Bragg); the
+    // library auto-gates it to strong/medium data. 0 = off; ~2.5 recommended.
+    const double part_unc = scaling_.GetPartialityUncertaintyCoeff();
+    auto *partUncertain = new QCheckBox("Partiality uncertainty", this);
+    partUncertain->setChecked(part_unc > 0.0);
+    partUncertain->setToolTip("Stills: add a systematic merge σ ~c·(1−partiality)·⟨I⟩ to partials so strong "
+                              "low-partiality partials are not over-trusted. Use with \"Stills partiality\"; "
+                              "auto-gated to strong/medium data. ~2.5 recommended; unchecked = off.");
+    auto *partUncertainCoeff = new NumberLineEdit(0.1f, 10.0f, part_unc > 0.0 ? part_unc : 2.5, 1, "", this);
+    partUncertainCoeff->setEnabled(partUncertain->isChecked());
+
     form->addRow("", friedel);
     form->addRow("", refineB);
     form->addRow("", corrections);
+    form->addRow("", modulation);
     // Compact, and aligned with the checkboxes above: the limit checkbox + value sit together in the
     // field column (not as a row label, which would indent it differently).
     auto *resRow = new QHBoxLayout();
     resRow->addWidget(limitRes);
     resRow->addWidget(highRes, 1);
     form->addRow("", resRow);
+    auto *puRow = new QHBoxLayout();
+    puRow->addWidget(partUncertain);
+    puRow->addWidget(partUncertainCoeff, 1);
+    form->addRow("", puRow);
     section->setContentLayout(form);
     section->setExpanded(false);   // folded on start (only geometry + unit cell start open)
 
@@ -496,16 +550,22 @@ QWidget *JFJochViewerSettingsDock::BuildScalingSection() {
         scaling_.MergeFriedel(friedel->isChecked());
         scaling_.RefineB(refineB->isChecked());
         scaling_.CorrectionSurfaces(corrections->isChecked());
+        scaling_.StillsModulation(modulation->isChecked());
         scaling_.HighResolutionLimit_A(limitRes->isChecked()
             ? std::optional<double>(highRes->value()) : std::nullopt);
+        scaling_.PartialityUncertaintyCoeff(partUncertain->isChecked() ? partUncertainCoeff->value() : 0.0);
         emit scalingChanged(scaling_);
     };
     connect(friedel, &QCheckBox::toggled, this, [emitScaling] { emitScaling(); });
     connect(refineB, &QCheckBox::toggled, this, [emitScaling] { emitScaling(); });
     connect(corrections, &QCheckBox::toggled, this, [emitScaling] { emitScaling(); });
+    connect(modulation, &QCheckBox::toggled, this, [emitScaling] { emitScaling(); });
     connect(limitRes, &QCheckBox::toggled, this, [emitScaling, highRes](bool on) {
         highRes->setEnabled(on); emitScaling(); });
     connect(highRes, &NumberLineEdit::newValue, this, [emitScaling] { emitScaling(); });
+    connect(partUncertain, &QCheckBox::toggled, this, [emitScaling, partUncertainCoeff](bool on) {
+        partUncertainCoeff->setEnabled(on); emitScaling(); });
+    connect(partUncertainCoeff, &NumberLineEdit::newValue, this, [emitScaling] { emitScaling(); });
     return section;
 }