v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

common/BraggIntegrationSettings.cpp

@@ -96,3 +96,24 @@ float BraggIntegrationSettings::GetDMinLimit_A() const {
 float BraggIntegrationSettings::GetMinimumSigmaInRegardsToI() const {
     return minimum_sigma_in_regards_to_i;
 }
+
+BraggIntegrationSettings &BraggIntegrationSettings::StillPartiality(bool input) {
+    still_partiality = input;
+    return *this;
+}
+
+bool BraggIntegrationSettings::GetStillPartiality() const {
+    return still_partiality;
+}
+
+BraggIntegrationSettings &BraggIntegrationSettings::BackgroundTrimFraction(float input) {
+    check_finite("Background trim fraction", input);
+    check_min("Background trim fraction", input, 0.0);
+    check_max("Background trim fraction", input, 0.49);   // must leave a central majority after trimming
+    bkg_trim_fraction = input;
+    return *this;
+}
+
+float BraggIntegrationSettings::GetBackgroundTrimFraction() const {
+    return bkg_trim_fraction;
+}

common/BraggIntegrationSettings.h

@@ -7,7 +7,7 @@
 
 // Spot-intensity extraction method used by the Bragg integration engine. ProfileGaussian (default)
 // profile-fits with a measured-width Gaussian (Kabsch-style) - more accurate intensities than the
-// classical uniform BoxSum; validated on HEWL anomalous (stronger S/Cl peaks vs box-sum). BoxSum is
+// classical uniform BoxSum; validated on anomalous data (stronger S/Cl peaks vs box-sum). BoxSum is
 // the simpler, faster fallback. ProfileEmpirical learns the profile per resolution shell from strong
 // spots - see docs/CPU_DATA_ANALYSIS.md (Bragg integration).
 enum class IntegratorMode { BoxSum, ProfileGaussian, ProfileEmpirical };
@@ -20,6 +20,14 @@ class BraggIntegrationSettings {
     float d_min_limit_A = 1.0;
     std::optional<float> fixed_profile_radius;
     float minimum_sigma_in_regards_to_i = 0.02;
+    bool still_partiality = false;   // experimental stills excitation-error partiality (rugnux --still-partiality)
+    // Symmetric trimmed-mean fraction for the r2..r3 background ring: drop the lowest and highest this
+    // fraction of ring pixels before averaging. Resists the high-side contamination (neighbour-spot
+    // wings, tails, zingers) that biases the plain ring mean up and makes it over-subtract weak
+    // high-angle reflections. Applied to monochromatic data (rotation and still); the integration engine
+    // keeps the tuned high-side sigma-clip for broadband (non-zero bandwidth) data instead. 0 = plain ring
+    // mean (rugnux --background-trim).
+    float bkg_trim_fraction = 0.10f;
 
 public:
     BraggIntegrationSettings& R1(float input);
@@ -28,6 +36,8 @@ public:
     BraggIntegrationSettings& DMinLimit_A(float input);
     BraggIntegrationSettings& FixedProfileRadius_recipA(std::optional<float> input);
     BraggIntegrationSettings& Integrator(IntegratorMode input);
+    BraggIntegrationSettings& StillPartiality(bool input);
+    BraggIntegrationSettings& BackgroundTrimFraction(float input);
 
 
     [[nodiscard]] IntegratorMode GetIntegrator() const;
@@ -38,4 +48,6 @@ public:
     [[nodiscard]] float GetDMinLimit_A() const;
 
     [[nodiscard]] float GetMinimumSigmaInRegardsToI() const;
+    [[nodiscard]] bool GetStillPartiality() const;
+    [[nodiscard]] float GetBackgroundTrimFraction() const;
 };

common/DiffractionExperiment.cpp

@@ -1510,6 +1510,15 @@ DiffractionExperiment &DiffractionExperiment::IndexingAlgorithm(IndexingAlgorith
 }
 
 IndexingAlgorithmEnum DiffractionExperiment::GetIndexingAlgorithm() const {
+    // Rotation indexing accumulates a dense multi-frame reciprocal-space cloud that needs a GLOBAL
+    // lattice finder (3D FFT). ffbidx is a single-still, known-cell ORIENTATION finder: it cannot
+    // recover the global orientation from that cloud (it locks onto a local wedge -> garbage). So
+    // rotation always uses FFT (GPU) or FFTW (CPU), regardless of the requested algorithm or whether a
+    // cell is known - a reference MTZ (-z) or -C supplies a cell and used to silently flip Auto->FFBIDX,
+    // switching the rotation indexer to the one solver that cannot handle its input.
+    if (IsRotationIndexing())
+        return get_gpu_count() == 0 ? IndexingAlgorithmEnum::FFTW : IndexingAlgorithmEnum::FFT;
+
     auto cell = GetUnitCell().has_value();
 
     switch (indexing.GetAlgorithm()) {

common/FileWriterSettings.cpp

@@ -15,8 +15,6 @@ FileWriterSettings &FileWriterSettings::FileFormat(FileWriterFormat input) {
         case FileWriterFormat::NXmxLegacy:
         case FileWriterFormat::NXmxVDS:
         case FileWriterFormat::NXmxIntegrated:
-        case FileWriterFormat::TIFF:
-        case FileWriterFormat::CBF:
         case FileWriterFormat::NoFile:
             hdf5_master_format_version = input;
             return *this;

common/IndexingSettings.h

@@ -6,7 +6,9 @@
 #include <cstdint>
 
 enum class IndexingAlgorithmEnum {FFBIDX, FFT, FFTW, Auto, None};
-enum class GeomRefinementAlgorithmEnum {None, OrientationOnly, BeamCenter};
+// Flex = "let the pipeline decide": try several per-image refinements and keep whichever indexes the
+// most spots (CLI -r flex; the legacy -r multi name is still accepted as an alias).
+enum class GeomRefinementAlgorithmEnum {None, OrientationOnly, BeamCenter, Flex};
 
 class IndexingSettings {
     IndexingAlgorithmEnum algorithm;

common/JFJochMessages.h

@@ -33,8 +33,8 @@ enum class FileWriterFormat : int {
     NXmxLegacy = 1,
     NXmxVDS = 2,
     NXmxIntegrated = 3,
-    CBF = 4,
-    TIFF = 5,
+    // 4 (CBF) and 5 (TIFF) removed - only HDF5 is written now. The values are kept
+    // (deprecated) in the OpenAPI enum in broker/jfjoch_api.yaml for back compatibility.
     NoFile = 6
 };
 

common/Reflection.h

@@ -26,7 +26,6 @@ struct Reflection {
     float dist_ewald;
     float rlp;
     float partiality;     // fraction of the reflection recorded in the sampled (rocking) slice
-    float completeness = 1.0f; // fraction of the spot footprint on live pixels (1 = not clipped by edge/gap/mask)
     float zeta;
     float image_scale_corr; // I_true = scaling_correction * I; scaling_correction = rlp / (partiality * image_scale)
     bool observed = false;
@@ -45,6 +44,18 @@ struct MergedReflection {
     bool rfree_flag = false;
     float F = NAN;       // French-Wilson amplitude |F| (filled by ApplyFrenchWilson at end of merge)
     float sigmaF = NAN;  // its sigma
+    // Anomalous (Bijvoet) split of this reflection's own observations, kept even when the merge is
+    // Friedel-averaged (I above is the Friedel mean). Lets I(+)/I(-) be written and CCano reported by
+    // default without scaling anomalously; NaN when a hand was not measured or for centrics.
+    float I_plus = NAN;
+    float sigma_plus = NAN;
+    float I_minus = NAN;
+    float sigma_minus = NAN;
+    // French-Wilson amplitudes of the two hands (filled by ApplyFrenchWilson from I_plus/I_minus).
+    float F_plus = NAN;
+    float sigmaF_plus = NAN;
+    float F_minus = NAN;
+    float sigmaF_minus = NAN;
 };
 
 

common/ScalingSettings.cpp

@@ -151,6 +151,15 @@ bool ScalingSettings::GetCorrectionSurfaces() const {
     return correction_surfaces;
 }
 
+ScalingSettings &ScalingSettings::StillsModulation(bool input) {
+    stills_modulation = input;
+    return *this;
+}
+
+bool ScalingSettings::GetStillsModulation() const {
+    return stills_modulation;
+}
+
 ScalingSettings &ScalingSettings::SmoothGDegrees(double input) {
     if (input < 0)
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Smooth-G range must be non-negative");
@@ -162,6 +171,17 @@ double ScalingSettings::GetSmoothGDegrees() const {
     return smooth_g_deg;
 }
 
+ScalingSettings &ScalingSettings::RelativeBDegrees(double input) {
+    if (input < 0)
+        throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "Relative-B batch width must be non-negative");
+    relative_b_deg = input;
+    return *this;
+}
+
+double ScalingSettings::GetRelativeBDegrees() const {
+    return relative_b_deg;
+}
+
 double ScalingSettings::GetMinPartiality() const {
     return min_partiality;
 }
@@ -191,6 +211,15 @@ double ScalingSettings::GetCaptureUncertaintyCoeff() const {
     return capture_uncertainty_coeff;
 }
 
+ScalingSettings &ScalingSettings::PartialityUncertaintyCoeff(double input) {
+    partiality_uncertainty_coeff = input;
+    return *this;
+}
+
+double ScalingSettings::GetPartialityUncertaintyCoeff() const {
+    return partiality_uncertainty_coeff;
+}
+
 ScalingSettings &ScalingSettings::MinCapturedFraction(double input) {
     if (input < 0.0)
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid,
@@ -203,15 +232,6 @@ double ScalingSettings::GetMinCapturedFraction() const {
     return min_captured_fraction;
 }
 
-ScalingSettings &ScalingSettings::FileFormat(IntensityFormat input) {
-    intensity_format = input;
-    return *this;
-}
-
-IntensityFormat ScalingSettings::GetFileFormat() const {
-    return intensity_format;
-}
-
 ScalingSettings &ScalingSettings::RfreeFraction(double input) {
     if (input < 0.0 || input > 1.0)
         throw JFJochException(JFJochExceptionCategory::InputParameterInvalid, "R-free fraction must be between 0 and 1");

common/ScalingSettings.h

@@ -6,8 +6,6 @@
 #include <optional>
 #include "JFJochException.h"
 
-enum class IntensityFormat { Text, mmCIF, MTZ};
-
 // How the high-resolution cutoff for the written reflections and the reported shell table is chosen
 // when no explicit --scaling-high-resolution is given. Off = keep the full (detector-edge) range;
 // CCHalfLogistic = fit the CC1/2 fall-off and cut one shell past cc_target (DIALS-style, generous).
@@ -29,6 +27,7 @@ class ScalingSettings {
     // a fraction f<1 of its rocking curve is extrapolated, and the unobserved (1-f) carries a
     // systematic error ~coeff*(1-f)*I that plain counting sigma misses. 0 = off (baseline).
     double capture_uncertainty_coeff = 0.0;
+    double partiality_uncertainty_coeff = 0.0;
     // Full-level captured-fraction floor for the rot3d combine: drop a reconstructed full whose rocking
     // curve was only fractionally captured (sum of its partials' partiality < this). Distinct from
     // min_partiality, which gates individual partials; this gates the assembled full. 0 = off (baseline).
@@ -47,6 +46,10 @@ class ScalingSettings {
     bool correction_surfaces = true;
     // Absorption-surface refinement iteration count (used when correction_surfaces is on).
     int absorption_iter = 3;
+    // Detector-plane modulation (flat-field) correction for the STILLS merge path (ScaleOnTheFly +
+    // MergeOnTheFly), which otherwise has no correction surfaces. Off by default (experimental; the
+    // rotation path fits its own modulation via RotationScaleMerge). Enabled by rugnux --stills-modulation.
+    bool stills_modulation = false;
 
     // Smooth the per-frame scale G across frames (centered moving average of log G) before the rot3d
     // combine, so a rocking event's partials share a consistent scale. Given as a ROTATION RANGE in
@@ -54,8 +57,14 @@ class ScalingSettings {
     // the smoothing physical (independent of frame slicing). 0 = off. A no-op without rot3d.
     double smooth_g_deg = 0.0;
 
+    // Per-batch relative-B on the rot3d fulls (beyond the single global decay slope): bin frames into
+    // rotation-range batches of this width in degrees and refine one relative Debye-Waller B per batch, so
+    // NON-monotonic changes in scattering power across the run (absorption path, crystal slippage, dose
+    // bursts) are corrected the resolution-flat per-frame G and the single decay slope both miss. Cross-
+    // validated + zero-mean-anchored, so a no-op when absent. 0 = off. A no-op without rot3d.
+    double relative_b_deg = 0.0;
+
     double rfree_fraction = 0.05;
-    IntensityFormat intensity_format = IntensityFormat::mmCIF;
 
     // Automatic high-resolution cutoff for the written reflections + reported shells (not the scaling
     // or the error model, and not the per-image _process.h5). Applied only when no explicit
@@ -75,16 +84,18 @@ public:
     ScalingSettings& MinPartiality(double min_partiality);
     ScalingSettings& ForcedMosaicity(std::optional<double> input);
     ScalingSettings& CaptureUncertaintyCoeff(double input);
+    ScalingSettings& PartialityUncertaintyCoeff(double input);
     ScalingSettings& MinCapturedFraction(double input);
     ScalingSettings& MinCCForImage(double min_cc_for_image);
     ScalingSettings& OutlierRejectNsigma(double input);
     ScalingSettings& ScaleFulls(bool input);
     ScalingSettings& AbsorptionIter(int input);
     ScalingSettings& CorrectionSurfaces(bool input);
+    ScalingSettings& StillsModulation(bool input);
     ScalingSettings& SmoothGDegrees(double input);
+    ScalingSettings& RelativeBDegrees(double input);
 
     ScalingSettings& RfreeFraction(double input);
-    ScalingSettings& FileFormat(IntensityFormat input);
     ScalingSettings& ScalingRegularize(bool input);
 
     ScalingSettings& ResolutionCutoff(ResolutionCutoffMethod input);
@@ -112,16 +123,18 @@ public:
     [[nodiscard]] double GetMinPartiality() const;
     [[nodiscard]] std::optional<double> GetForcedMosaicity() const;
     [[nodiscard]] double GetCaptureUncertaintyCoeff() const;
+    [[nodiscard]] double GetPartialityUncertaintyCoeff() const;
     [[nodiscard]] double GetMinCapturedFraction() const;
     [[nodiscard]] double GetMinCCForImage() const;
     [[nodiscard]] double GetOutlierRejectNsigma() const;
     [[nodiscard]] bool GetScaleFulls() const;
     [[nodiscard]] int GetAbsorptionIter() const;
     [[nodiscard]] bool GetCorrectionSurfaces() const;
+    [[nodiscard]] bool GetStillsModulation() const;
     [[nodiscard]] double GetSmoothGDegrees() const;
+    [[nodiscard]] double GetRelativeBDegrees() const;
 
     [[nodiscard]] double GetRfreeFraction() const;
-    [[nodiscard]] IntensityFormat GetFileFormat() const;
     [[nodiscard]] bool GetScalingRegularize() const;
 
     [[nodiscard]] ResolutionCutoffMethod GetResolutionCutoff() const;