Jungfraujoch/docs/RUGNUX_INTEGRATION.md

# rugnux with other programs

What the reflection files promise to a reading program, and the minimum commands that get each
downstream suite running on rugnux output.

```{contents} On this page
:local:
:depth: 2
```

## Reflection-file conventions

**mmCIF.** Standard items carry their standard meanings — `_refln.intensity_meas` / `_intensity_sigma`,
the `pdbx_I_plus`/`pdbx_I_minus` and `pdbx_F_plus`/`pdbx_F_minus` anomalous pairs, `_reflns.*` and
`_reflns_shell.*` for the merging statistics, `_reflns.B_iso_Wilson_estimate` for the Wilson B, and
`_cell.*` / `_diffrn_radiation_wavelength.wavelength` for the geometry.

Anything rugnux reports that has no standard item is written under a **`jfjoch_`** prefix, inside the
standard category it belongs to. That is a deliberate choice: a reader that does not know these items
ignores them, and one that does can find them without guessing.

| item | meaning |
|---|---|
| `_reflns.jfjoch_diffrn_ISa` | Asymptotic I/σ in **XDS's sense**: the whole-range `1/√(a·b)` of the error model, so it can be read directly against a `CORRECT.LP` |
| `_reflns.jfjoch_diffrn_ISa_asymptotic` | The **strong-reflection** tier — the counting-subtracted scatter of well-measured groups. XDS has no equivalent, and it can only ever be the more optimistic of the two. Rotation path only |
| `_reflns.jfjoch_error_model_a`, `_b` | The error model in XDS's convention, `σ² = a(σ₀² + b·I²)`, so the ISa above is re-derivable from the file rather than taken on trust |
| `_reflns.jfjoch_second_moment_I` | Twinning second moment ⟨I²⟩/⟨I⟩² — 2.00 untwinned, 1.50 for a perfect twin |
| `_reflns.jfjoch_L_test_mean_abs_L`, `_L_test_mean_L_squared` | Padilla–Yeates L-test. ⟨\|L\|⟩ is 0.500 untwinned / 0.375 for a perfect twin; ⟨L²⟩ is 0.333 / 0.200. Written only when the test found pairs |
| `_reflns.jfjoch_radiation_damage_relative_B` | Relative *B* from the first to the last rotation batch (Ų); positive is the usual direction, high-resolution intensity fading with dose |
| `_jfjoch_radiation_damage_batch.*` | Per-batch loop: `id`, `rotation_start_deg`, `relative_B` |
| `_diffrn_detector.jfjoch_distance_mm`, `_jfjoch_beam_center_x_pxl`, `_jfjoch_beam_center_y_pxl` | The refined detector geometry actually used, which is not otherwise recoverable from the reflection file |
| `_reflns.pdbx_aniso_B_tensor_eigenvalue_1..3`, `_pdbx_aniso_B_tensor_eigenvector_*` | The anisotropy tensor, eigen-decomposed. Eigenvalues are **relative to the weakest direction** (so the third is 0 and the first is the anisotropic Δ*B*), because only the deviatoric part is determined; eigenvectors are in the PDB orthogonalisation convention. Not written for a cubic Laue class, where symmetry forces Δ*B* to be zero |
| `_reflns.jfjoch_aniso_delta_B`, `_jfjoch_aniso_delta_B_linear` | The anisotropic Δ*B*, and the part of it that actually follows exp(−½ **s**ᵀ*B***s**). The second is what the verdict is gated on |
| `_reflns.jfjoch_aniso_d_min_1..3` | Diffraction limit (Å) along each principal direction. A comment marks a value that is the edge of the measured data rather than the crystal's own limit |
| `_reflns.jfjoch_aniso_shape`, `_jfjoch_aniso_floor`, `_jfjoch_aniso_significance`, `_jfjoch_aniso_verdict` | The resolution signature of the deficit, the data set's own systematic-error floor, Δ*B*<sub>linear</sub> over that floor, and the resulting verdict. Each carries its vocabulary as a comment |

> **Compatibility note.** Before rc.161, `_reflns.jfjoch_diffrn_ISa` carried the *asymptote*, not the
> whole-range value. There is no version marker inside the file, so a number taken from an older
> `.cif` is not comparable with one taken from a newer one.

**SHELX HKLF 4** (`<prefix>.hkl`). Fixed-format `3I4,2F8.2` — `h k l I σ(I)`, one record per
reflection, terminated by a `0 0 0` record — which is what **SHELXC**, **SHELXD** and **ANODE**
expect. Two properties worth knowing before using it:

- **Bijvoet mates are written separately**, `I(+)` at `+hkl` and `I(-)` at `-hkl`, so the anomalous
  differences survive into SHELXC; a reflection with no anomalous split is written once, as its mean.
- **Intensities are rescaled** by a single global factor so the largest value fits the `F8.2` field.
  `I` and `σ(I)` share that factor, so every ratio — and therefore the anomalous signal — is
  untouched, but the absolute scale is not meaningful. This matters only if you intend to compare
  magnitudes with another file; SHELXC and ANODE use ratios alone.

## The unmerged export

`<prefix>_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column
layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics`
read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as
well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an
output prefix (`-o`). It is the largest file a run produces, larger on a dense rotation dataset than
the merged `.mtz`, `.cif` and `.hkl` put together, so a run that only wants the merged numbers — a
regression battery, or a throughput pipeline — turns it off with `--no-export-unmerged`.

Use it to scale the data with a different program, to have pointless give an independent opinion on
the space group, or to compare rugnux's merge against another one on identical input. Each sweep's
file is self-contained, so several of them can be handed to pointless and aimless as separate
`HKLIN`s to merge sweeps rugnux does not combine itself.

> **Trap when combining a wild-carded series.** For an `HKLIN` given with wild-cards, POINTLESS
> accepts the files in order and **terminates acceptance at the first file out of chronological
> order**, then merges what it kept and prints a plausible result. Its own keyword lifts the check —
> `ALLOW OUTOFSEQUENCEFILES` — or name each file as its own `HKLIN`, which is not a series; either
> way, check the file count in its log against the number you meant to give.

**Columns.** `H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG` — POINTLESS's own set —
plus four rugnux extras, `DELPHI` (offset from the centre of the rocking curve), `ZETA` (the Lorentz
geometry of that curve), `BGMEAN` and `BGVAR` (the background that was subtracted, and its variance).
`BATCH` is the image ordinal plus one, and a batch header is written for every batch that carries an
observation. `M/ISYM` records both the symmetry operation and the Friedel hand, so the index as
measured is recoverable from the index as stored.

**Header symmetry and order.** The file's MTZ header carries the **space group the run determined**
(P1 where none was), and the rows are **sorted on `H K L M/ISYM BATCH`** — the order POINTLESS
leaves an unmerged file in, and the order AIMLESS requires of its input — so both programs take the
file directly.

**What has been applied to the intensities, and what has not.** `I` and `SIGI` carry the
**Lorentz-polarization factor and nothing else**; the factor itself is in the `LP` column, so raw
counts are `I/LP`. LP is applied because it is per-observation geometry that varies by more than two
orders of magnitude across a sweep and no reader can reconstruct it. Deliberately *not* applied:
the **partiality is not divided out** (it is reported in `FRACTIONCALC`), and the **per-image scale is
not applied at all** — those programs fit their own scale model, and handing them pre-scaled data
would have them fit a correction to a correction. No resolution cut, outlier rejection or ice-ring
filtering is applied either.

**Partials.** On a rotation run the partials of each reflection are summed into one full, using the
same rule rugnux's own 3D combine uses — consecutive frames no more than two apart — and the full is
written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in
`FRACTIONCALC`. An event that caught less of its rocking curve than `--min-partiality` is not
written, exactly as in the merge. Summing is the default because a downstream program's own partial
handling is far more conservative than rugnux's: given raw partials, aimless accepted a small
fraction of the file and merged at a fraction of the multiplicity; given summed fulls it uses
essentially all of it. `--export-unmerged-partials` writes the unsummed form to
`<prefix>_unmerged_partials.mtz` for a program that would rather sum them itself. Stills have no
rocking events and are the same either way.

**Systematic absences.** Lattice-**centring** absences are not written; **screw and glide** absences
are. Prediction runs in a primitive setting so that the space-group search can test the centring,
but the interstitial reflections that leaves make a reading program take the lattice for primitive
and demote the group. Screw and glide absences are kept because they are the evidence the space
group was chosen on — deleting them would turn a reading program's test into an assumption. XDS and
DIALS draw the line in the same place.

**Scan axis.** The batch headers carry the goniometer axis **negated** relative to the one in the
input file. This is not a correction to the file: rugnux brings an observation made at angle φ back
to zero by rotating it by +φ, so the crystal itself turns by −φ, and an MTZ batch header records the
axis a batch's own increasing `PHI` turns the crystal about. With the sign as exported, pointless's
independently determined orientation matrix agrees with rugnux's to well under a degree.

## Taking the data onward

The reflection files are inputs to other suites, and the handover has a few conventions worth one
line each. These are the minimum commands that get each program running on rugnux output.

**phenix.** The merged files carry both the mean intensity and the Bijvoet pairs, and phenix
refuses to choose between two equally usable intensity arrays — `Sorry: Multiple equally suitable
arrays of observed xray data found.` (from the mmCIF the message at least lists the choices; from
the MTZ it names none, which is where a user is genuinely stuck). It is one behaviour surfacing in
both formats; only the label vocabulary differs, following each format's own column names — the
MTZ label fails on the mmCIF with `Sorry: No matching array`:

```
phenix.xtriage myrun.mtz xray_data.obs_labels=IMEAN,SIGIMEAN
phenix.xtriage myrun.cif xray_data.obs_labels=intensity_meas
phenix.refine  model.pdb myrun.mtz miller_array.labels.name=IMEAN
```

(phenix.refine takes a different parameter name for the same choice.)

**The R-free convention.** `FreeR_flag` is **0 = work, 1 = free** (the phenix/CNS convention;
5 % free by default). phenix.refine detects that on its own. **REFMAC5 does not**: its default
takes flag 0 as the free set and stops with `Error ==> Cannot switch free R flag` — give it the
keyword `FREE 1`:

```
refmac5 XYZIN model.pdb HKLIN myrun.mtz XYZOUT refined.pdb HKLOUT refined.mtz <<eof
LABIN FP=F SIGFP=SIGF FREE=FreeR_flag
FREE 1
NCYC 10
END
eof
```

**POINTLESS / AIMLESS.** `myrun_unmerged.mtz` opens in both directly — it is sorted the way AIMLESS
requires and its header carries the determined space group (see
[The unmerged export](#the-unmerged-export)). Running pointless first remains the safe route, and
its independent space-group opinion is what the file exists for:

```
pointless HKLIN myrun_unmerged.mtz HKLOUT sorted.mtz
aimless   HKLIN sorted.mtz         HKLOUT scaled.mtz
```

Several sweeps of one crystal form go in as separate `HKLIN`s to the same pointless run — that is
how sweeps rugnux does not combine itself are merged.

**careless** wants exactly what the unmerged export is — unmerged, unscaled, LP-only intensities
with the partiality reported and not divided out. Against its published examples, two renames:
`BG`/`SIGBG` are called `BGMEAN`/`BGVAR` here **and `BGVAR` is a variance, not a sigma**; there is
no `QE` column. `Hobs`/`Kobs`/`Lobs` are reconstructed from `M/ISYM` by reciprocalspaceship, and
`dHKL` careless computes from the cell, so the metadata string that names this file's columns is

```
careless mono --anomalous "BATCH,dHKL,Hobs,Kobs,Lobs,XDET,YDET,BGMEAN,BGVAR,LP,FRACTIONCALC" \
    myrun_unmerged.mtz out/myrun
```

**Molecular replacement.** When the report names an enantiomorphic pair (`P 41 or P 43`), let
Phaser try both hands: `SGALTERNATIVE SELECT HAND`. When section 4's candidate table shows a close
runner-up, `SGALTERNATIVE SELECT ALL` searches the alternatives, and `myrun_P1.mtz` is there for
re-merging in a subgroup when the call itself is in doubt.

**`iotbx.merging_statistics myrun_unmerged.mtz`** and **SHELXC on `myrun.hkl`** need no arguments
or label choices at all.

## Comparing the geometry with XDS

Every run logs the detector geometry a second time in **XDS's convention**, so it can be read
straight across against the `IDXREF.LP` / `CORRECT.LP` of an XDS run on the same data:

```
XDS convention: ORGX= 1091.00 ORGY= 1137.00 DETECTOR_DISTANCE= 75.0000
XDS convention: DIRECTION_OF_DETECTOR_X-AXIS= 1.000000 0.000000 0.000000
XDS convention: DIRECTION_OF_DETECTOR_Y-AXIS= 0.000000 1.000000 0.000000
XDS convention: INCIDENT_BEAM_DIRECTION= 0 0 1 X-RAY_WAVELENGTH= 1.000000 QX= QY= 0.075000
XDS convention: ROTATION_AXIS= -1.000000 0.000000 0.000000
```

XDS is never given this geometry — the [XDS plugin](SOFTWARE_INTEGRATION.md) supplies image data
only, and XDS refines its own from `XDS.INP` — which is what makes the comparison worth having. The
two laboratory frames coincide (x along increasing detector column, y along increasing row, z along
the beam), so the numbers are directly comparable, and a tilt appears as the two detector axis
vectors rather than as angles, which is how XDS reports it after refinement. Two things to keep in
mind: **`ORGX`/`ORGY` are 1-based**, because XDS counts pixels from 1 and Jungfraujoch from 0; and
they are the **PONI**, the same quantity Jungfraujoch's beam centre is — so no correction is needed —
but not the direct beam once the detector is tilted (see above).