v1.0.0-rc.160 (#70)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

* rugnux: Add `--model model.pdb` - score the merged data against an atomic model and compute initial maps. It reports R-work/R-free (scaling the model to the observed amplitudes with an overall scale, an anisotropic B and a flat bulk solvent - the standard few-parameter model, so a batch of maps stays directly comparable) and writes 2Fo-Fc / Fo-Fc electron-density maps (CCP4) plus a map-coefficient MTZ. The structure itself is not refined; the model is only re-fractionalised into the data cell.
* rugnux: The merged reflection output now carries French-Wilson amplitudes (|F| and its sigma) next to the intensities - MTZ `F`/`SIGF`, mmCIF `_refln.F_meas_au`, and the text HKL - computed with the correct centric/acentric Wilson prior and epsilon multiplicity, so a downstream program (e.g. phenix.refine) can refine against amplitudes. The intensity columns are unchanged.
* rugnux: R-free test-set flags are now assigned deterministically and consistently across symmetry - a Bijvoet pair I(+)/I(-) is never split between the work and free sets, and the assignment is a reproducible per-hkl hash that depends only on the reflection index, so every dataset of one crystal form gets the same ~5% free set (what a multi-dataset campaign such as PanDDA needs). On small data the fraction is floored so the test set stays large enough for a stable R-free (~500 reflections, capped at 10%); it stays flat at 5% on ordinary data. When a reference MTZ carries a `FreeR_flag` column its test set is imported instead, letting a whole campaign inherit one shared free set.
* rugnux: A reference MTZ (`--reference-mtz`) can now fix the space group and cell for rotation data too (previously rejected), without being used to scale - the rotation merge stays self-consistent. When the crystal has an indexing (merohedral) ambiguity - a lattice symmetry higher than its Laue symmetry, e.g. P3/P4/P6/C2 - the reference also resolves it: each candidate reindexing (identity plus the twin-law cosets of the metric symmetry) is scored by its intensity correlation against the reference and the data are re-merged in the best-correlating one. This is a metric-preserving relabelling of hkl (the cell is unchanged) and a no-op for a holohedral crystal such as lysozyme.
* rugnux: `--model` validation now aligns the data to the model before scoring - the observed reflections are reindexed into the model's enantiomorph when the two differ only by hand (indistinguishable from merged intensities). A merohedral indexing ambiguity is resolved against the reference MTZ when one is given (so a whole campaign shares one indexing convention); only with a model and no reference does validation fall back to fitting each candidate reindexing and keeping the lowest R-free.
* rugnux: De-novo symmetry - recover a genuine high-symmetry group whose data are imperfectly scaled. Such a merge's within-orbit chi² lands just past the self-consistency bound (each real symmetry step adds a little systematic scatter), right where a merohedral twin also lands, so the chi² ratio alone cannot separate them. The candidate is now rescued when the extra intensity-proportional systematic error it invokes stays small relative to the confirmed subgroup - a genuine symmetry step gains multiplicity without inflating the merge error model's b, whereas a twin forces non-equivalent reflections together and b balloons. Fixes cubic insulin (I23 instead of I222) with no change to any other crystal in the test battery, including the twins that must stay in their lower symmetry.
* Docs: Document the French-Wilson amplitude estimation, R-free flagging, reference-based space-group/ambiguity resolution, and model-based validation/maps in CPU_DATA_ANALYSIS.md.
* Frontend: The status-bar pill now shows a progress bar during detector calibration (previously only during measurement), and the calibration state and its button are labelled "Calibration"/"CALIBRATE" (the internal `Pedestal` state name is unchanged for back-compatibility).Reviewed-on: #70

Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>

image_analysis/bragg_integration/BraggIntegrationEngineGPU.cu

@@ -26,10 +26,14 @@ struct BraggGpuParams {
     float c_radial;
     float F_px;
     float beam_x, beam_y;
+    float bkg_trim;       // idea 1: symmetric trimmed-mean background fraction (0 = plain ring mean)
 };
 
 __device__ inline bool valid(int32_t v) { return v != INT32_MIN && v != INT32_MAX; }
 
+// idea 1: max annulus pixels held in shared memory for the per-block trimmed-mean sort (else flat mean).
+constexpr int BKG_TRIM_MAX = 512;
+
 // --- Mark the r2 signal disk of every predicted reflection (race-free: all writes are 1). ---
 __global__ void mark_mask(const float *px_x, const float *px_y, uint8_t *mask, BraggGpuParams p, int n) {
     const int i = blockIdx.x;
@@ -111,8 +115,57 @@ __global__ void boxsum(const float *px_x, const float *px_y, const float *dd,
     }
     __syncthreads();
 
+    // Trimmed-mean background (idea 1): collect the annulus into shared memory, bitonic-sort the block, and
+    // average the middle (1 - 2*bkg_trim) fraction - robust to the high-side contamination that biases the
+    // plain ring mean. Falls back to the flat mean when the ring exceeds the shared buffer.
+    __shared__ int s_bvals[BKG_TRIM_MAX];
+    __shared__ int s_bn;
+    if (threadIdx.x == 0) s_bn = 0;
+    __syncthreads();
+    const bool do_trim = s_accept && p.bkg_trim > 0.0f && s_nbkg > 5 && s_nbkg <= BKG_TRIM_MAX;
+    if (do_trim) {
+        for (int t = threadIdx.x; t < area; t += blockDim.x) {
+            const int x = x0 + t % bw, y = y0 + t / bw;
+            const float ddx = (float) x - cx, ddy = (float) y - cy;
+            const float d2 = ddx * ddx + ddy * ddy;
+            if (!(d2 >= p.r2_sq && d2 < p.r3_sq)) continue;
+            if (mask[y * p.W + x]) continue;
+            const int32_t px = img[y * p.W + x];
+            if (!valid(px)) continue;
+            const int slot = atomicAdd(&s_bn, 1);
+            if (slot < BKG_TRIM_MAX) s_bvals[slot] = px;
+        }
+        __syncthreads();
+        const int nb = min(s_bn, BKG_TRIM_MAX);
+        int n2 = 1;
+        while (n2 < nb) n2 <<= 1;
+        for (int t = threadIdx.x + nb; t < n2; t += blockDim.x) s_bvals[t] = INT32_MAX;   // pad to power of 2
+        __syncthreads();
+        for (int k = 2; k <= n2; k <<= 1)                       // ascending bitonic sort of s_bvals[0..n2)
+            for (int j = k >> 1; j > 0; j >>= 1) {
+                for (int idx = threadIdx.x; idx < n2; idx += blockDim.x) {
+                    const int ixj = idx ^ j;
+                    if (ixj > idx) {
+                        const bool up = ((idx & k) == 0);
+                        const int a = s_bvals[idx], b = s_bvals[ixj];
+                        if ((up && a > b) || (!up && a < b)) { s_bvals[idx] = b; s_bvals[ixj] = a; }
+                    }
+                }
+                __syncthreads();
+            }
+        if (threadIdx.x == 0) {
+            const int lo = (int) (nb * p.bkg_trim), hi = nb - lo;
+            if (hi > lo) {
+                double s = 0.0;
+                for (int t = lo; t < hi; ++t) s += (double) s_bvals[t];
+                s_bkg = s / (double) (hi - lo);
+            }
+        }
+        __syncthreads();
+    }
+
     // Second ring pass for the stills sigma-clip (re-reads the annulus; avoids storing bkg values).
-    if (s_accept && p.do_clip) {
+    if (s_accept && p.do_clip && !do_trim) {
         double c_l = 0.0; int cn_l = 0;
         for (int t = threadIdx.x; t < area; t += blockDim.x) {
             const int x = x0 + t % bw, y = y0 + t / bw;
@@ -437,6 +490,7 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
         .bw_sigma = static_cast<float>(bw_sigma), .c_radial = static_cast<float>(c_radial),
         .F_px = static_cast<float>(F_px),
         .beam_x = beam_x, .beam_y = beam_y,
+        .bkg_trim = bkg_trim,   // effective trim fraction (0 for stills), set by the base ctor from settings
     };
 
     // Pass A: reset accumulators, mask, then box-sum.
@@ -464,12 +518,11 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
     cuda_err(cudaMemcpyAsync(h_sigma.data(), d_sigma, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaMemcpyAsync(h_bkg.data(), d_bkg, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaMemcpyAsync(h_ok.data(), d_ok, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
-    const bool boxsum_mode = mode == IntegratorMode::BoxSum;
-    if (boxsum_mode) {
-        cuda_err(cudaMemcpyAsync(h_obs_x.data(), d_obs_x, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
-        cuda_err(cudaMemcpyAsync(h_obs_y.data(), d_obs_y, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
-        cuda_err(cudaMemcpyAsync(h_has_obs.data(), d_has_obs, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
-    }
+    // Pass A always fills the box-sum centroid (observed spot position), so copy it back in all modes -
+    // post-refinement uses it as the observed position (beam-centre / distance).
+    cuda_err(cudaMemcpyAsync(h_obs_x.data(), d_obs_x, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
+    cuda_err(cudaMemcpyAsync(h_obs_y.data(), d_obs_y, sizeof(float) * npredicted, cudaMemcpyDeviceToHost, *stream));
+    cuda_err(cudaMemcpyAsync(h_has_obs.data(), d_has_obs, sizeof(uint8_t) * npredicted, cudaMemcpyDeviceToHost, *stream));
     cuda_err(cudaStreamSynchronize(*stream));
 
     for (size_t i = 0; i < npredicted; ++i) {
@@ -478,7 +531,7 @@ std::vector<Reflection> BraggIntegrationEngineGPU::Run(const ImagePreprocessorBu
         results[i].sigma = h_sigma[i];
         results[i].bkg = h_bkg[i];
         results[i].ok = true;
-        if (boxsum_mode && h_has_obs[i]) {
+        if (h_has_obs[i]) {
             results[i].observed_x = h_obs_x[i];
             results[i].observed_y = h_obs_y[i];
             results[i].has_observed = true;