# Rugnux with other programs What the reflection files promise to a reading program, and the minimum commands that get each downstream suite running on Rugnux output. ```{contents} On this page :local: :depth: 2 ``` ## Reflection-file conventions **mmCIF.** Standard items carry their standard meanings — `_refln.intensity_meas` / `_intensity_sigma`, the `pdbx_I_plus`/`pdbx_I_minus` and `pdbx_F_plus`/`pdbx_F_minus` anomalous pairs, `_reflns.*` and `_reflns_shell.*` for the merging statistics, `_reflns.B_iso_Wilson_estimate` for the Wilson B, and `_cell.*` / `_diffrn_radiation_wavelength.wavelength` for the geometry. Anything Rugnux reports that has no standard item is written under a **`jfjoch_`** prefix, inside the standard category it belongs to. That is a deliberate choice: a reader that does not know these items ignores them, and one that does can find them without guessing. | item | meaning | |---|---| | `_reflns.jfjoch_diffrn_ISa` | Asymptotic I/σ in **XDS's sense**: the whole-range `1/√(a·b)` of the error model, so it can be read directly against a `CORRECT.LP` | | `_reflns.jfjoch_diffrn_ISa_asymptotic` | The **strong-reflection** tier — the counting-subtracted scatter of well-measured groups. XDS has no equivalent, and it can only ever be the more optimistic of the two. Rotation path only | | `_reflns.jfjoch_error_model_a`, `_b` | The error model in XDS's convention, `σ² = a(σ₀² + b·I²)`, so the ISa above is re-derivable from the file rather than taken on trust | | `_reflns.jfjoch_second_moment_I` | Twinning second moment ⟨I²⟩/⟨I⟩² — 2.00 untwinned, 1.50 for a perfect twin | | `_reflns.jfjoch_L_test_mean_abs_L`, `_L_test_mean_L_squared` | Padilla–Yeates L-test. ⟨\|L\|⟩ is 0.500 untwinned / 0.375 for a perfect twin; ⟨L²⟩ is 0.333 / 0.200. Written only when the test found pairs | | `_reflns.jfjoch_radiation_damage_relative_B` | Relative *B* from the first to the last rotation batch (Ų); positive is the usual direction, high-resolution intensity fading with dose | | `_jfjoch_radiation_damage_batch.*` | Per-batch loop: `id`, `rotation_start_deg`, `relative_B` | | `_diffrn_detector.jfjoch_distance_mm`, `_jfjoch_beam_center_x_pxl`, `_jfjoch_beam_center_y_pxl` | The refined detector geometry actually used, which is not otherwise recoverable from the reflection file | | `_reflns.pdbx_aniso_B_tensor_eigenvalue_1..3`, `_pdbx_aniso_B_tensor_eigenvector_*` | The anisotropy tensor, eigen-decomposed. Eigenvalues are **relative to the weakest direction** (so the third is 0 and the first is the anisotropic Δ*B*), because only the deviatoric part is determined; eigenvectors are in the PDB orthogonalisation convention. Not written for a cubic Laue class, where symmetry forces Δ*B* to be zero | | `_reflns.jfjoch_aniso_delta_B`, `_jfjoch_aniso_delta_B_linear` | The anisotropic Δ*B*, and the part of it that actually follows exp(−½ **s**ᵀ*B***s**). The second is what the verdict is gated on | | `_reflns.jfjoch_aniso_d_min_1..3` | Diffraction limit (Å) along each principal direction. A comment marks a value that is the edge of the measured data rather than the crystal's own limit | | `_reflns.jfjoch_aniso_shape`, `_jfjoch_aniso_floor`, `_jfjoch_aniso_significance`, `_jfjoch_aniso_verdict` | The resolution signature of the deficit, the data set's own systematic-error floor, Δ*B*linear over that floor, and the resulting verdict. Each carries its vocabulary as a comment | > **Compatibility note.** Before rc.161, `_reflns.jfjoch_diffrn_ISa` carried the *asymptote*, not the > whole-range value. There is no version marker inside the file, so a number taken from an older > `.cif` is not comparable with one taken from a newer one. **SHELX HKLF 4** (`.hkl`). Fixed-format `3I4,2F8.2` — `h k l I σ(I)`, one record per reflection, terminated by a `0 0 0` record — which is what **SHELXC**, **SHELXD** and **ANODE** expect. Two properties worth knowing before using it: - **Bijvoet mates are written separately**, `I(+)` at `+hkl` and `I(-)` at `-hkl`, so the anomalous differences survive into SHELXC; a reflection with no anomalous split is written once, as its mean. - **Intensities are rescaled** by a single global factor so the largest value fits the `F8.2` field. `I` and `σ(I)` share that factor, so every ratio — and therefore the anomalous signal — is untouched, but the absolute scale is not meaningful. This matters only if you intend to compare magnitudes with another file; SHELXC and ANODE use ratios alone. **MTZ** (`.mtz`, and `_P1.mtz` beside it). The CCP4 anomalous layout, with the column types CCP4 programs dispatch on: ``` H K L IMEAN SIGIMEAN I(+) SIGI(+) I(-) SIGI(-) F SIGF F(+) SIGF(+) F(-) SIGF(-) FreeR_flag H H H J Q K M K M F Q G L G L I ``` `F` is the French–Wilson amplitude. The header carries the determined space group, the refined cell and the wavelength, on a dataset of its own behind the reserved `HKL_base`, which is where the MTZ format puts them. Older Rugnux wrote the data on dataset 0, the id reserved for `HKL_base`, and CCP4's `mtzinfo` then reported its 1.54187 Å (Cu Kα) default instead of the real wavelength — every other reader tried, `mtzdmp`, `truncate`, `ctruncate`, gemmi, iotbx and `phenix.xtriage`, recovered the true value from those files as well, so the effect was confined to that one report. Note that `_unmerged.mtz` still reads 1.54187 under `mtzinfo` and is not wrong: its columns sit on `HKL_base` deliberately, as POINTLESS expects, and the wavelength AIMLESS and POINTLESS read is the per-batch one, which is correct. The Bijvoet columns are present on any rotation merge, with or without `-A`; a stills merge has no Bijvoet split and the file then stops after `F SIGF FreeR_flag`. There is deliberately **no `DANO`/`SIGDANO`** pair, the anomalous difference columns a CCP4 merged file usually carries. They are a restatement rather than a measurement: checked column against column on a `ctruncate` file, `DANO` is `F(+) − F(-)` to the last bit and `SIGDANO` is `√(σ(+)² + σ(−)²)` to the last bit, on every reflection — the quadrature sum is the convention whether or not the two mates came from one scale model, and no correlation correction is applied by anybody. Every program in the phasing routes below reads the Bijvoet columns directly and forms the difference itself, and CCP4's own phasing engines prefer them: `bp3` and `afro` want `F+/SF+/F-/SF-` and tell a user holding `F`/`DANO` to convert *to* that form, and `mtz2sca` ranks `I(+/-)` over `F(+/-)` over `F/DANO`. Where the pair is genuinely wanted — `fft`'s anomalous-difference Fourier takes a `DANO` label and has no other spelling — one command makes it, with the `ISYM` column that belongs beside it: ``` ctruncate -hklin myrun.mtz -hklout myrun_ct.mtz \ -colin '/*/*/[IMEAN,SIGIMEAN]' -colano '/*/*/[I(+),SIGI(+),I(-),SIGI(-)]' ``` ## The unmerged export `_unmerged.mtz` holds every integrated observation, before scaling and merging, in the column layout POINTLESS writes and **aimless**, **pointless**, **careless** and `iotbx.merging_statistics` read. It is written by default, in `--mode mx` and `--mode scale` alike and with `--no-merge` as well, and it replaces nothing — Rugnux still writes its own merged files in the same run. It needs an output prefix (`-o`). It is the largest file a run produces, larger on a dense rotation dataset than the merged `.mtz`, `.cif` and `.hkl` put together, so a run that only wants the merged numbers — a regression battery, or a throughput pipeline — turns it off with `--no-export-unmerged`. Use it to scale the data with a different program, to have pointless give an independent opinion on the space group, or to compare Rugnux's merge against another one on identical input. Each sweep's file is self-contained, so several of them can be handed to pointless and aimless as separate `HKLIN`s to merge sweeps Rugnux does not combine itself. > **Trap when combining a wild-carded series.** For an `HKLIN` given with wild-cards, POINTLESS > accepts the files in order and **terminates acceptance at the first file out of chronological > order**, then merges what it kept and prints a plausible result. Its own keyword lifts the check — > `ALLOW OUTOFSEQUENCEFILES` — or name each file as its own `HKLIN`, which is not a series; either > way, check the file count in its log against the number you meant to give. **Columns.** `H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP QE FLIGHT FLAG` — POINTLESS's own set down to `FLAG`, plus `QE` and `FLIGHT` (the sensor-efficiency and flight-path divisors described above; `QE` is DIALS's column) — then four Rugnux extras, `DELPHI` (offset from the centre of the rocking curve), `ZETA` (the Lorentz geometry of that curve), `BGMEAN` and `BGVAR` (the background that was subtracted, and its variance). `BATCH` is the image ordinal plus one, and a batch header is written for every batch that carries an observation. `M/ISYM` records both the symmetry operation and the Friedel hand, so the index as measured is recoverable from the index as stored. **Header symmetry and order.** The file's MTZ header carries the **space group the run determined** (P1 where none was), and the rows are **sorted on `H K L M/ISYM BATCH`** — the order POINTLESS leaves an unmerged file in, and the order AIMLESS requires of its input — so both programs take the file directly. **What has been applied to the intensities, and what has not.** `I` and `SIGI` carry the **deterministic per-reflection corrections and nothing else**. Three columns record them: `LP` is Lorentz x polarization; `QE` is the sensor's quantum efficiency at the angle the diffracted beam meets the detector; and `FLIGHT` is the attenuation in the medium the reflection crossed on its way there. `QE` and `FLIGHT` are both **divisors** normalised to 1 at normal incidence, so raw counts are `I / LP * QE * FLIGHT`. They are applied because they are per-observation geometry that varies by more than two orders of magnitude across a sweep and no reader can reconstruct them. `QE` is at least 1 and `FLIGHT` at most 1: an oblique reflection crosses more sensor, which makes it read high, and more of the medium, which makes it read low. `FLIGHT` is a column of ones under `--flight-path vacuum`. `QE` is kept out of `LP` because that is what the field means by `LP`: XDS's `RLP` is Lorentz x polarization alone (its own column is flat to 0.1% across a detector over which the efficiency term spans 7%), and DIALS fills `LP` from lorentz/polarization only and writes `QE` as a separate column — a column of ones where it has no correction. Rugnux normalises `QE` to normal incidence where DIALS stores the un-normalised absorbed fraction; the two differ by a per-dataset constant, i.e. by an overall scale. Deliberately *not* applied: the **partiality is not divided out** (it is reported in `FRACTIONCALC`), and the **per-image scale is not applied at all** — those programs fit their own scale model, and handing them pre-scaled data would have them fit a correction to a correction. No resolution cut, outlier rejection or ice-ring filtering is applied either. **Partials.** On a rotation run the partials of each reflection are summed into one full, using the same rule Rugnux's own 3D combine uses — consecutive frames no more than two apart — and the full is written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in `FRACTIONCALC`. An event that caught less of its rocking curve than `--min-captured-fraction` (or `--min-partiality`) is not written, exactly as in the merge. Summing is the default because a downstream program's own partial handling is far more conservative than Rugnux's: given raw partials, aimless accepted a small fraction of the file and merged at a fraction of the multiplicity; given summed fulls it uses essentially all of it. `--export-unmerged-partials` writes the unsummed form to `_unmerged_partials.mtz` for a program that would rather sum them itself. Stills have no rocking events and are the same either way. **Systematic absences.** Lattice-**centring** absences are not written; **screw and glide** absences are. Prediction runs in a primitive setting so that the space-group search can test the centring, but the interstitial reflections that leaves make a reading program take the lattice for primitive and demote the group. Screw and glide absences are kept because they are the evidence the space group was chosen on — deleting them would turn a reading program's test into an assumption. XDS and DIALS draw the line in the same place. **Scan axis.** The batch headers carry the goniometer axis **negated** relative to the one in the input file. This is not a correction to the file: Rugnux brings an observation made at angle φ back to zero by rotating it by +φ, so the crystal itself turns by −φ, and an MTZ batch header records the axis a batch's own increasing `PHI` turns the crystal about. With the sign as exported, pointless's independently determined orientation matrix agrees with Rugnux's to well under a degree. ## Taking the data onward The reflection files are inputs to other suites, and the handover has a few conventions worth one line each. These are the minimum commands that get each program running on Rugnux output. **phenix.** The merged files carry both the mean intensity and the Bijvoet pairs, and a phenix program that has not said which it wants stops on the pair of them — from the MTZ and from the mmCIF alike, each listing its own format's labels: ``` Sorry: Multiple equally suitable arrays of observed xray data found. Possible choices: myrun.mtz:IMEAN,SIGIMEAN myrun.mtz:I(+),SIGI(+),I(-),SIGI(-) ``` Two things are worth knowing before reading that as a fault in the file. The tie is between the two **intensity** arrays and nothing else: iotbx scores `F`/`SIGF` and `F(+)`/`F(-)` below them, so they are never in the running and writing amplitudes as well as intensities is not what causes this. And **ctruncate's own output ties in the same place** — put any merged data through CCP4's truncate step and phenix asks the same question of the result, because a mean intensity array and an anomalous one score equally whenever the calling program has expressed no preference. The only file change that removes the tie is dropping one of the two, and dropping the Bijvoet columns would take the anomalous signal — and the whole [SHELX route](#experimental-phasing-with-shelx) — with it. So the answer is a label. The parameter name differs by program, which is the part that catches people out: ``` phenix.xtriage myrun.mtz xray_data.obs_labels=IMEAN phenix.xtriage myrun.mtz "xray_data.obs_labels=I(+)" # the Bijvoet array instead phenix.refine model.pdb myrun.mtz miller_array.labels.name=IMEAN ``` `IMEAN` on its own is enough — the match is on a substring — and `IMEAN,SIGIMEAN` and the fully-qualified `scaling.input.xray_data.obs_labels=` work equally. Quote the anomalous one: the parentheses are shell syntax otherwise. The same behaviour appears on the mmCIF in that format's own vocabulary, and a label from one format does not work on the other (`Sorry: No matching array`): ``` phenix.xtriage myrun.cif xray_data.obs_labels=intensity_meas phenix.xtriage myrun.cif xray_data.obs_labels=pdbx_I_plus ``` **A program that states a preference needs none of this.** `phenix.hyss`, `phenix.find_peaks_holes`, `phenix.molprobity` and the data import behind `phenix.autosol` ask for anomalous data by preference, which breaks the tie for them. `phenix.hyss myrun.mtz n_sites=6 scattering_type=S` opens the file with no labels given, reports `Miller array info: myrun.mtz:I(+),SIGI(+),I(-),SIGI(-)`, and forms the anomalous differences itself. **The R-free convention.** `FreeR_flag` is **0 = free, 1 = work** — the CCP4 convention the column's own name belongs to (5 % free by default). REFMAC5's default `FREE 0` reads it directly and phenix.refine detects the numbering on its own, so neither needs a keyword: ``` refmac5 XYZIN model.pdb HKLIN myrun.mtz XYZOUT refined.pdb HKLOUT refined.mtz <` should be about 0.80 where there is no anomalous signal. Two sweeps are quoted below, both collected at 5 keV for the sulfur signal: a cubic one that went all the way, and a tetragonal one that did not. The cubic one, 2.5 Å at 95 % completeness and multiplicity 30, reads: ``` Resl. Inf. 13.02 8.01 6.03 4.93 4.22 3.71 3.33 3.04 2.80 2.60 2.43 108.8 91.4 63.0 64.7 70.6 61.8 45.5 34.7 23.7 12.4 5.0 %Complete 96.2 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 99.0 72.7 2.58 5.06 3.97 2.92 2.36 1.68 1.50 1.33 1.48 1.38 1.79 ``` **SHELXD will separate space groups the merged intensities could not.** That sweep's report named a body-centred cubic pair as indistinguishable, so SHELXC and SHELXD were run once per candidate — same reflections, same `FIND`, only `SPAG` different. One gave `CC 37.93 / CC(weak) 14.05 / CFOM 51.98` and the other `CC 46.76 / CC(weak) 22.61 / CFOM 69.37`. The substructure is where the screw axis shows itself, and the second group is the right one. This is the same handover as Phaser's arrived at from the other side, and it is worth doing whenever `SPACE_GROUP_ALTERNATIVES` is not `NONE` — SHELXD takes seconds, and the pair of runs costs less than reprocessing anything. **SHELXE decides the hand**, and says so. Run it twice, `-i` inverting the substructure. `-s` is the solvent fraction, `-h` says the substructure atoms belong to the native structure, as sulfur does, and `-a` turns on autotracing, which is what actually makes the two hands separate. The two runs write `sad.pdb` and `sad_i.pdb`, so they can share a directory: ``` shelxe sad sad_fa -h -s0.62 -m20 -a15 -q shelxe sad sad_fa -h -s0.62 -m20 -a15 -q -i ``` At 63 % solvent the two hands came out at 42.93 % and 15.28 % for the autotrace CC against the native data — pseudo-free CC 66.49 against 37.12, map contrast 0.87 against 0.44, 215 traced atoms — which is a solved structure, from `myrun.hkl` and nothing else. Where the group is one of the 22 that come in enantiomorphic pairs, SHELXE makes the group change itself: the inverted run prints `** Space group converted to enantiomorph **` and writes the changed group into the `CRYST1` of its traced model, so the answer is readable off the output file the same way it is off Phaser's. **A negative result, for calibration.** A tetragonal dataset at the same wavelength with the same kind of substructure, but 87 % complete at multiplicity 20 rather than 95 % at 30, gave a plausible SHELXD `CFOM 47.62` and then failed at the hand: 15.33 % against 15.60 % autotrace CC, map contrast 0.33 either way. That is not a discrimination and it is not a solution. Nothing about the file was the limit — the anomalous signal SHELXC measured on it was real, `` reaching 4.2 — so the reading is that sulfur phasing wants the completeness and the multiplicity, and a `.hkl` from a sweep that does not have them will get this far and no further. ## Comparing the geometry with XDS Every run logs the detector geometry a second time in **XDS's convention**, so it can be read straight across against the `IDXREF.LP` / `CORRECT.LP` of an XDS run on the same data: ``` XDS convention: ORGX= 1091.00 ORGY= 1137.00 DETECTOR_DISTANCE= 75.0000 XDS convention: DIRECTION_OF_DETECTOR_X-AXIS= 1.000000 0.000000 0.000000 XDS convention: DIRECTION_OF_DETECTOR_Y-AXIS= 0.000000 1.000000 0.000000 XDS convention: INCIDENT_BEAM_DIRECTION= 0 0 1 X-RAY_WAVELENGTH= 1.000000 QX= QY= 0.075000 XDS convention: ROTATION_AXIS= -1.000000 0.000000 0.000000 ``` XDS is never given this geometry — the [XDS plugin](SOFTWARE_INTEGRATION.md) supplies image data only, and XDS refines its own from `XDS.INP` — which is what makes the comparison worth having. The two laboratory frames coincide (x along increasing detector column, y along increasing row, z along the beam), so the numbers are directly comparable, and a tilt appears as the two detector axis vectors rather than as angles, which is how XDS reports it after refinement. Two things to keep in mind: **`ORGX`/`ORGY` are 1-based**, because XDS counts pixels from 1 and Jungfraujoch from 0; and they are the **PONI**, the same quantity Jungfraujoch's beam centre is — so no correction is needed — but not the direct beam once the detector is tilted (see above).