jfjoch_process/scale: default to mmCIF, auto-detect rotation, rot3d partiality

- Default reflection format is now mmCIF (ScalingSettings.h, the single source of
  truth). jfjoch_process and jfjoch_scale no longer hard-code a local MTZ default;
  they override only when --scaling-output is given. jfjoch_viewer inherits this.
- A dataset with a rotation goniometer axis is processed as rotation data (two-pass
  indexing) by default; add --process-as-stills to force per-frame stills. -R still
  tunes the first-pass image count.
- rot3d is the default partiality model for rotation processing (fixed for stills)
  when no explicit -P is given.
- Update docs/JFJOCH_PROCESS.md: new de-novo rotation/stills examples, corrected
  -R/-P/--scaling-output defaults, --process-as-stills, and a real Integration table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

common/ScalingSettings.h

@@ -51,7 +51,7 @@ class ScalingSettings {
     double smooth_g_deg = 0.0;
 
     double rfree_fraction = 0.05;
-    IntensityFormat intensity_format = IntensityFormat::MTZ;
+    IntensityFormat intensity_format = IntensityFormat::mmCIF;
 
     bool scaling_regularize = false;
 public:

docs/JFJOCH_PROCESS.md

@@ -43,8 +43,10 @@ the first pass.
   integration, azimuthal integration, per-image statistics). See
   [HDF5 / NeXus data format](HDF5.md) for the layout.
 - When merging (`-M`, or whenever a `--reference-mtz` is supplied), the merged reflections are
-  written as `<prefix>.mtz` (default), or `<prefix>.cif` / `<prefix>.hkl` depending on
-  `--scaling-output`. No-reference scaling additionally emits per-iteration `<prefix>_iterN_scale.dat`.
+  written as `<prefix>.cif` (mmCIF — the default), or `<prefix>.mtz` / `<prefix>.hkl` depending on
+  `--scaling-output`. Both the mmCIF and the MTZ carry the **refined unit cell** (from rotation
+  indexing) and the **space group determined from systematic absences** (constrained to the indexed
+  lattice symmetry). No-reference scaling additionally emits per-iteration `<prefix>_iterN_scale.dat`.
 
 Merged statistics (⟨I/σ⟩, CC1/2, completeness, …), the error model and timing are printed to the
 console.
@@ -60,38 +62,45 @@ or to combine several processed runs into one set of merged intensities.
 
 ### Rotation data
 
-Two-pass rotation indexing, rotation partiality, scale and merge in space group 96:
+Index, integrate, scale and merge a rotation sweep, fully de novo:
 
 ```
 jfjoch_process rotation_master.h5 \
-    -o lyso_rot -N 16 \
-    -R -S 96 \
-    -M -P rot
+    -o lyso_rot -N 32 \
+    -M --scaling-high-resolution 1.4
 ```
 
-`-R` runs the two-pass rotation indexer (index the sweep once, then process every frame against
-that lattice); `-P rot` selects the rotation partiality model; `-M` scales and merges. For strong
-rotation data the de-novo FFT indexer often indexes more frames — add `-X fft` (and drop `-C` to
-let it find the cell from scratch).
+Because the dataset carries a rotation goniometer axis, it is processed as **rotation data by
+default**: two-pass rotation indexing (index the sweep once, then process every frame against that
+lattice) with the **`rot3d`** partiality model (rotation partials combined into 3D fulls). `-M`
+scales and merges; the unit cell is taken from the rotation indexer and the space group is
+determined from systematic absences, and both are written into the merged `.cif`.
+
+Run **fully de novo** (no `-C`/`-S`) for the best result — supplying a cell or space group up front
+tends to *degrade* low-symmetry cases. `--scaling-high-resolution` (set it to your expected
+resolution) sharpens both the space-group search and the error model. To tune the first pass use
+`--two-pass-rotation=100` (or `-R100` — the first-pass image count); to force the sweep to be
+treated as independent stills use `--process-as-stills`.
 
 ### Still / serial data
 
-Known-cell indexing of independent stills with the GPU fast-feedback indexer, then merge against a
-reference structure:
+A dataset with **no goniometer axis** (e.g. a serial grid scan) is processed as **independent
+stills automatically** — no flag needed. Known-cell indexing with the GPU fast-feedback indexer,
+then merge against a reference structure:
 
 ```
 jfjoch_process serial_master.h5 \
-    -o lyso_serial -N 16 \
+    -o lyso_serial -N 32 \
     -X ffbidx -C 79,79,38,90,90,90 -S 96 \
     --spot-sigma 4 \
-    -M -z reference.mtz -r pixelrefine \
+    -M -z reference.mtz \
     --scaling-high-resolution 1.8
 ```
 
-`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills.
-`-r pixelrefine` selects the experimental reference-driven still integrator (needs
-`--reference-mtz`). For weak serial data, tightening spot finding with `--spot-sigma 4` typically
-raises the indexing rate substantially.
+`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. For
+weak serial data, tightening spot finding with `--spot-sigma 4` typically raises the indexing rate
+substantially. If a dataset *does* carry a goniometer axis but you want per-frame stills processing
+anyway, add `--process-as-stills`.
 
 ## Command-line options
 
@@ -117,13 +126,19 @@ Spot finding:
 
 Indexing:
 
+A dataset with a **rotation goniometer axis** is processed as rotation data (two-pass rotation
+indexing) by default; a dataset without one is processed as independent stills. `--process-as-stills`
+overrides the former; the `-R` / `--single-pass-rotation` / `--force-rotation-lattice` flags request
+rotation explicitly and pick the pass or lattice.
+
 | Option | Description |
 | --- | --- |
+| `--process-as-stills` | Treat a rotation (goniometer) dataset as independent stills instead of rotation |
 | `-X, --indexing-algorithm <txt>` | `FFBIDX` \| `FFT` \| `FFTW` \| `Auto` \| `None` |
 | `-C, --unit-cell <cell>` | Reference unit cell `"a,b,c,alpha,beta,gamma"` (required by `ffbidx`) |
 | `-S, --space-group <num>` | Space group number (used for indexing and scaling) |
-| `-r, --refine <txt>` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) \| `pixelrefine` |
-| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (optional image count, default 30) |
+| `-r, --refine <txt>` | Geometry refinement: `none` \| `orientation` \| `beam_and_lattice` (default) |
+| `-R, --two-pass-rotation[=num]` | Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default 100) |
 | `--single-pass-rotation[=num]` | Online-like single-pass rotation indexing (optional min angular range, deg) |
 | `--redo-rotation-spots` | Redo spot finding for the two-pass rotation first pass |
 | `--force-rotation-lattice <vec>` | Force rotation lattice (9 floats, Å), skipping the first pass |
@@ -137,23 +152,24 @@ Scaling and merging:
 | Option | Description |
 | --- | --- |
 | `-M, --scale-merge` | Scale and merge |
-| `-P, --partiality <txt>` | Partiality model: `fixed` (default) \| `rot` \| `unity` |
+| `-P, --partiality <txt>` | Partiality model: `fixed` \| `rot` \| `rot3d` \| `unity` (default: `rot3d` for rotation data, `fixed` for stills). `rot3d` = `rot` + 3D combine of the per-frame partials into fulls |
 | `-A, --anomalous` | Anomalous mode (keep Friedel pairs separate) |
 | `-B, --refine-bfactor` | Refine a per-image B-factor |
 | `-w, --wedge[=num]` | Refine the per-image rotation wedge (optional starting value) |
 | `--scaling-high-resolution <num>` | High-resolution limit for scaling, Å (default: no limit) |
 | `--min-partiality <num>` | Minimum partiality to accept a reflection (default: 0.02) |
-| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: off) |
+| `--reject-outliers <num>` | Per-observation outlier rejection, N σ from the per-reflection median (default: 6 for `rot3d`, off otherwise) |
 | `--reject-delta-cchalf <num>` | Drop images with ΔCC1/2 below mean − N·stddev (default: off) |
 | `--min-image-cc <num>` | Per-image CC limit, percent (default: no limit) |
 | `--scaling-iterations <num>` | Scaling iterations with no reference data (default: 3) |
-| `--scaling-output <txt>` | Reflection output format: `mtz` (default) \| `cif` \| `txt` |
+| `--scaling-output <txt>` | Reflection output format: `cif` (mmCIF, default) \| `mtz` \| `txt` |
 | `-z, --reference-mtz <file>` | Reference MTZ (enables reference-driven scaling) |
 
-Pixel refinement (experimental; select with `-r pixelrefine`, requires `--reference-mtz`):
+Integration:
 
 | Option | Description |
 | --- | --- |
-| `--bandwidth <num>` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) |
+| `--integrator <txt>` | Spot integrator: `gaussian` (profile-fit, default) \| `empirical` \| `boxsum` (classical fallback) |
 | `--integration-radius <r>` | Signal-box radius `r1`, or `r1,r2,r3` (px). One value ⇒ `r2=r1+2`, `r3=r1+4` |
-| `--profile-multiplier <num>` | Scale the measured tangential profile width (default: 6; XDS-style generous aperture) |
+| `--reciprocal-profile` | Learn one global reciprocal-space profile width (`A+B·|q|+C·|q|²`) instead of per-shell; helps mosaic/sparse data |
+| `--bandwidth <num>` | Relative X-ray bandwidth FWHM (e.g. `0.01` for a 1% DMM); default from file or 0 (monochromatic) |

tools/jfjoch_process.cpp

@@ -45,7 +45,9 @@ void print_usage() {
     std::cout << std::endl;
 
     std::cout << "  Indexing" << std::endl;
-    std::cout << "   -R, --two-pass-rotation[=num]    Two-pass offline rotation indexing (optional: number of images, default: 100)" << std::endl;
+    std::cout << "   (A dataset with a rotation goniometer axis is processed as rotation data by default; use --process-as-stills to override)" << std::endl;
+    std::cout << "   --process-as-stills              Process a rotation (goniometer) dataset as independent stills instead of rotation" << std::endl;
+    std::cout << "   -R, --two-pass-rotation[=num]    Two-pass offline rotation indexing (default for goniometer data; optional first-pass image count, default: 100)" << std::endl;
     std::cout << "   --single-pass-rotation[=num]     Use online-like single-pass rotation indexing (optional: min angular range deg)" << std::endl;
     std::cout << "   --redo-rotation-spots            Redo spot finding for two-pass rotation indexing" << std::endl;
     std::cout << "   --force-rotation-lattice <vec>   Force rotation indexer with external lattice (in Angstrom) : \"a0x,a0y,a0z,a1x,a1y,a1z,a2x,a2y,a2z\" (9 floats, skips first pass)" << std::endl;
@@ -62,7 +64,7 @@ void print_usage() {
     std::cout << "   --write-process-h5              Also write the (large) _process.h5 when merging (default: only .mtz/.cif when merging)" << std::endl;
     std::cout << "   --absorption[=num]              rot3d: fit a smooth absorption surface on the fulls over num iterations (default 3; off by default; matters at low energy)" << std::endl;
     std::cout << "   --smooth-g[=deg]                rot3d: smooth per-frame scale G over a deg-degree rotation range (XDS DELPHI-like) before the combine (default: 5 for rot3d; 0 = off)" << std::endl;
-    std::cout << "   -P, --partiality <txt>          Partiality model fixed|rot|rot3d|unity (default: fixed). rot3d = rot + 3D combine of per-frame partials" << std::endl;
+    std::cout << "   -P, --partiality <txt>          Partiality model fixed|rot|rot3d|unity (default: rot3d for rotation data, fixed for stills). rot3d = rot + 3D combine of per-frame partials" << std::endl;
     std::cout << "   -A, --anomalous                 Anomalous mode (don't merge Friedel pairs)" << std::endl;
     std::cout << "   -B, --refine-bfactor            Refine per image B-factor" << std::endl;
     std::cout << "   -w, --wedge[=num]               Refine image wedge during scaling with starting wedge value" << std::endl;
@@ -74,7 +76,7 @@ void print_usage() {
     std::cout << "   --reject-delta-cchalf <num>     Per-crystal CC1/2-delta rejection: drop images with deltaCChalf below mean - N*stddev (default: off; e.g. 2.5)" << std::endl;
     std::cout << "   --min-image-cc <num>            Per-image CC limit in percent (default: no limit)" << std::endl;
     std::cout << "   --scaling-iterations <num>      Number of scaling iterations with no reference data (default: 3)" << std::endl;
-    std::cout << "   --scaling-output <txt>              Output format for scaling results mtz|cif|txt (default: mtz)" << std::endl;
+    std::cout << "   --scaling-output <txt>              Output format for scaling results mtz|cif|txt (default: cif)" << std::endl;
     std::cout << "   -z, --reference-mtz <file>      Reference MTZ file" << std::endl;
     std::cout << "   --reference-column <label>      Reference MTZ column to use (default: auto - F-model, else IMEAN/I/...)" << std::endl;
     std::cout << std::endl;
@@ -114,7 +116,8 @@ enum {
     OPT_DETECT_ICE_RINGS,
     OPT_NO_SCALE_FULLS,
     OPT_WRITE_PROCESS_H5,
-    OPT_ABSORPTION
+    OPT_ABSORPTION,
+    OPT_PROCESS_AS_STILLS
 };
 
 static option long_options[] = {
@@ -144,6 +147,7 @@ static option long_options[] = {
 
     {"two-pass-rotation", optional_argument, nullptr, 'R'},
     {"single-pass-rotation", optional_argument, nullptr, OPT_SINGLE_PASS_ROTATION},
+    {"process-as-stills", no_argument, nullptr, OPT_PROCESS_AS_STILLS},
     {"redo-rotation-spots", no_argument, nullptr, OPT_REDO_ROTATION_SPOTS},
     {"force-rotation-lattice", required_argument, nullptr, OPT_FORCE_ROTATION_LATTICE},
 
@@ -297,6 +301,7 @@ int main(int argc, char **argv) {
 
     bool verbose = false;
     bool rotation_indexing = false;
+    bool process_as_stills = false;     // --process-as-stills: force per-frame stills on a rotation dataset
     bool two_pass_rotation = true;
     bool reuse_rotation_spots = true;
     int rotation_indexing_image_count = 100;
@@ -331,8 +336,9 @@ int main(int argc, char **argv) {
     IndexingAlgorithmEnum indexing_algorithm = IndexingAlgorithmEnum::Auto;
     GeomRefinementAlgorithmEnum refinement_algorithm = GeomRefinementAlgorithmEnum::BeamCenter;
 
-    IntensityFormat intensity_format = IntensityFormat::MTZ;
+    std::optional<IntensityFormat> intensity_format;   // --scaling-output override; default lives in ScalingSettings (mmCIF)
     PartialityModel partiality_model = PartialityModel::Fixed;
+    bool partiality_explicit = false;   // true once -P is given, so the rotation/stills default does not override it
     bool combine_3d = false;   // -P rot3d: weight-sum per-frame partials into fulls before merging
 
     float d_min_spot_finding = 1.5;
@@ -515,6 +521,7 @@ int main(int argc, char **argv) {
                 print_usage();
                 exit(EXIT_FAILURE);
             }
+            partiality_explicit = true;
             break;
         case OPT_SPOT_SIGMA:
             sigma_spot_finding = atof(optarg);
@@ -601,6 +608,9 @@ int main(int argc, char **argv) {
                 exit(EXIT_FAILURE);
             }
             break;
+        case OPT_PROCESS_AS_STILLS:
+            process_as_stills = true;
+            break;
         case OPT_SCALING_ITERATIONS:
             scaling_iter = atoi(optarg);
             if (scaling_iter <= 0) {
@@ -753,6 +763,32 @@ int main(int argc, char **argv) {
     if (experiment.GetBandwidthFWHM())
         logger.Info("X-ray bandwidth FWHM set to {:.4f}", experiment.GetBandwidthFWHM().value());
 
+    // Rotation vs stills. A dataset collected on a rotation goniometer is processed as rotation data
+    // (two-pass indexing) by default; --process-as-stills forces per-frame stills. The rotation flags
+    // (-R / --single-pass-rotation / --force-rotation-lattice) still request rotation explicitly and
+    // choose the pass/lattice; at this point they show up as rotation_indexing already being set.
+    const bool has_goniometer = experiment.GetGoniometer().has_value();
+    if (process_as_stills) {
+        if (rotation_indexing) {
+            logger.Error("--process-as-stills conflicts with -R / --single-pass-rotation / --force-rotation-lattice");
+            exit(EXIT_FAILURE);
+        }
+        if (has_goniometer)
+            logger.Info("--process-as-stills: treating the rotation dataset as independent stills");
+    } else if (!rotation_indexing && has_goniometer) {
+        rotation_indexing = true;
+        two_pass_rotation = true;
+        logger.Info("Dataset has a rotation goniometer axis: processing as rotation data (two-pass "
+                    "indexing). Use --process-as-stills to treat it as stills.");
+    }
+
+    // Default partiality model: rot3d for rotation processing (Rotation partiality + 3D combine of the
+    // per-frame partials into fulls), fixed for stills. An explicit -P always wins.
+    if (!partiality_explicit && rotation_indexing) {
+        partiality_model = PartialityModel::Rotation;
+        combine_3d = true;   // rot3d
+    }
+
     // Configure Indexing
     IndexingSettings indexing_settings;
     indexing_settings.Algorithm(indexing_algorithm);
@@ -791,7 +827,8 @@ int main(int argc, char **argv) {
     scaling_settings.MinCCForImage(min_image_cc / 100.0);   // --min-image-cc is in percent; the setting is a fraction
     scaling_settings.OutlierRejectNsigma(
         outlier_reject_nsigma.value_or(combine_3d ? REJECT_OUTLIERS_DEFAULT_NSIGMA : 0.0));
-    scaling_settings.FileFormat(intensity_format);
+    if (intensity_format)
+        scaling_settings.FileFormat(intensity_format.value());
 
     experiment.ImportScalingSettings(scaling_settings);
 

tools/jfjoch_scale.cpp

@@ -50,7 +50,7 @@ void print_usage() {
     std::cout << "   --min-image-cc <num>                  Per-image CC limit in percent (default: no limit)" << std::endl;
     std::cout << "   --scaling-iterations <num>            Number of scaling iterations with no reference data (default: 3)"
             << std::endl;
-    std::cout << "   --scaling-output <txt>                Output format for scaling results mtz|cif|txt (default: mtz)" << std::endl;
+    std::cout << "   --scaling-output <txt>                Output format for scaling results mtz|cif|txt (default: cif)" << std::endl;
     std::cout << "   -z, --reference-mtz <file>            Reference MTZ file" << std::endl;
     std::cout << "   --reference-column <label>            Reference MTZ column to use (default: auto - F-model, else IMEAN/I/...)" << std::endl;
 }
@@ -110,7 +110,7 @@ int main(int argc, char **argv) {
     double min_image_cc = 0.0;
     int64_t scaling_iter = 3;
 
-    IntensityFormat intensity_format = IntensityFormat::MTZ;
+    std::optional<IntensityFormat> intensity_format;   // --scaling-output override; default lives in ScalingSettings (mmCIF)
     PartialityModel partiality_model = PartialityModel::Fixed;
     bool combine_3d = false;   // -P rot3d: weight-sum per-frame partials into fulls before merging
 
@@ -328,7 +328,8 @@ int main(int argc, char **argv) {
         scaling_settings.RotationWedgeForScaling(wedge_for_scaling);
     scaling_settings.MinPartiality(min_partiality);
     scaling_settings.MinCCForImage(min_image_cc);
-    scaling_settings.FileFormat(intensity_format);
+    if (intensity_format)
+        scaling_settings.FileFormat(intensity_format.value());
 
     experiment.ImportScalingSettings(scaling_settings);