Jungfraujoch/docs/RUGNUX_INTEGRATION.md

rugnux with other programs

What the reflection files promise to a reading program, and the minimum commands that get each downstream suite running on rugnux output.

:local:
:depth: 2

Reflection-file conventions

mmCIF. Standard items carry their standard meanings — _refln.intensity_meas / _intensity_sigma, the pdbx_I_plus/pdbx_I_minus and pdbx_F_plus/pdbx_F_minus anomalous pairs, _reflns.* and _reflns_shell.* for the merging statistics, _reflns.B_iso_Wilson_estimate for the Wilson B, and _cell.* / _diffrn_radiation_wavelength.wavelength for the geometry.

Anything rugnux reports that has no standard item is written under a jfjoch_ prefix, inside the standard category it belongs to. That is a deliberate choice: a reader that does not know these items ignores them, and one that does can find them without guessing.

item	meaning
_reflns.jfjoch_diffrn_ISa	Asymptotic I/σ in XDS's sense: the whole-range 1/√(a·b) of the error model, so it can be read directly against a CORRECT.LP
_reflns.jfjoch_diffrn_ISa_asymptotic	The strong-reflection tier — the counting-subtracted scatter of well-measured groups. XDS has no equivalent, and it can only ever be the more optimistic of the two. Rotation path only
_reflns.jfjoch_error_model_a, _b	The error model in XDS's convention, σ² = a(σ₀² + b·I²), so the ISa above is re-derivable from the file rather than taken on trust
_reflns.jfjoch_second_moment_I	Twinning second moment ⟨I²⟩/⟨I⟩² — 2.00 untwinned, 1.50 for a perfect twin
_reflns.jfjoch_L_test_mean_abs_L, _L_test_mean_L_squared	Padilla–Yeates L-test. ⟨|L|⟩ is 0.500 untwinned / 0.375 for a perfect twin; ⟨L²⟩ is 0.333 / 0.200. Written only when the test found pairs
_reflns.jfjoch_radiation_damage_relative_B	Relative B from the first to the last rotation batch (Ų); positive is the usual direction, high-resolution intensity fading with dose
_jfjoch_radiation_damage_batch.*	Per-batch loop: id, rotation_start_deg, relative_B
_diffrn_detector.jfjoch_distance_mm, _jfjoch_beam_center_x_pxl, _jfjoch_beam_center_y_pxl	The refined detector geometry actually used, which is not otherwise recoverable from the reflection file
_reflns.pdbx_aniso_B_tensor_eigenvalue_1..3, _pdbx_aniso_B_tensor_eigenvector_*	The anisotropy tensor, eigen-decomposed. Eigenvalues are relative to the weakest direction (so the third is 0 and the first is the anisotropic ΔB), because only the deviatoric part is determined; eigenvectors are in the PDB orthogonalisation convention. Not written for a cubic Laue class, where symmetry forces ΔB to be zero
_reflns.jfjoch_aniso_delta_B, _jfjoch_aniso_delta_B_linear	The anisotropic ΔB, and the part of it that actually follows exp(−½ sᵀBs). The second is what the verdict is gated on
_reflns.jfjoch_aniso_d_min_1..3	Diffraction limit (Å) along each principal direction. A comment marks a value that is the edge of the measured data rather than the crystal's own limit
_reflns.jfjoch_aniso_shape, _jfjoch_aniso_floor, _jfjoch_aniso_significance, _jfjoch_aniso_verdict	The resolution signature of the deficit, the data set's own systematic-error floor, ΔBlinear over that floor, and the resulting verdict. Each carries its vocabulary as a comment

Compatibility note. Before rc.161, _reflns.jfjoch_diffrn_ISa carried the asymptote, not the whole-range value. There is no version marker inside the file, so a number taken from an older .cif is not comparable with one taken from a newer one.

SHELX HKLF 4 (<prefix>.hkl). Fixed-format 3I4,2F8.2 — h k l I σ(I), one record per reflection, terminated by a 0 0 0 record — which is what SHELXC, SHELXD and ANODE expect. Two properties worth knowing before using it:

• Bijvoet mates are written separately, I(+) at +hkl and I(-) at -hkl, so the anomalous differences survive into SHELXC; a reflection with no anomalous split is written once, as its mean.
• Intensities are rescaled by a single global factor so the largest value fits the F8.2 field. I and σ(I) share that factor, so every ratio — and therefore the anomalous signal — is untouched, but the absolute scale is not meaningful. This matters only if you intend to compare magnitudes with another file; SHELXC and ANODE use ratios alone.

The unmerged export

<prefix>_unmerged.mtz holds every integrated observation, before scaling and merging, in the column layout POINTLESS writes and aimless, pointless, careless and iotbx.merging_statistics read. It is written by default, in --mode mx and --mode scale alike and with --no-merge as well, and it replaces nothing — rugnux still writes its own merged files in the same run. It needs an output prefix (-o). It is the largest file a run produces, larger on a dense rotation dataset than the merged .mtz, .cif and .hkl put together, so a run that only wants the merged numbers — a regression battery, or a throughput pipeline — turns it off with --no-export-unmerged.

Use it to scale the data with a different program, to have pointless give an independent opinion on the space group, or to compare rugnux's merge against another one on identical input. Each sweep's file is self-contained, so several of them can be handed to pointless and aimless as separate HKLINs to merge sweeps rugnux does not combine itself.

Trap when combining a wild-carded series. For an HKLIN given with wild-cards, POINTLESS accepts the files in order and terminates acceptance at the first file out of chronological order, then merges what it kept and prints a plausible result. Its own keyword lifts the check — ALLOW OUTOFSEQUENCEFILES — or name each file as its own HKLIN, which is not a series; either way, check the file count in its log against the number you meant to give.

Columns. H K L M/ISYM BATCH I SIGI FRACTIONCALC XDET YDET ROT LP FLAG — POINTLESS's own set — plus four rugnux extras, DELPHI (offset from the centre of the rocking curve), ZETA (the Lorentz geometry of that curve), BGMEAN and BGVAR (the background that was subtracted, and its variance). BATCH is the image ordinal plus one, and a batch header is written for every batch that carries an observation. M/ISYM records both the symmetry operation and the Friedel hand, so the index as measured is recoverable from the index as stored.

Header symmetry and order. The file's MTZ header carries the space group the run determined (P1 where none was), and the rows are sorted on H K L M/ISYM BATCH — the order POINTLESS leaves an unmerged file in, and the order AIMLESS requires of its input — so both programs take the file directly.

What has been applied to the intensities, and what has not. I and SIGI carry the Lorentz-polarization factor and nothing else; the factor itself is in the LP column, so raw counts are I/LP. LP is applied because it is per-observation geometry that varies by more than two orders of magnitude across a sweep and no reader can reconstruct it. Deliberately not applied: the partiality is not divided out (it is reported in FRACTIONCALC), and the per-image scale is not applied at all — those programs fit their own scale model, and handing them pre-scaled data would have them fit a correction to a correction. No resolution cut, outlier rejection or ice-ring filtering is applied either.

Partials. On a rotation run the partials of each reflection are summed into one full, using the same rule rugnux's own 3D combine uses — consecutive frames no more than two apart — and the full is written at the batch its rocking curve is centred on, with the summed rocking-curve fraction in FRACTIONCALC. An event that caught less of its rocking curve than --min-partiality is not written, exactly as in the merge. Summing is the default because a downstream program's own partial handling is far more conservative than rugnux's: given raw partials, aimless accepted a small fraction of the file and merged at a fraction of the multiplicity; given summed fulls it uses essentially all of it. --export-unmerged-partials writes the unsummed form to <prefix>_unmerged_partials.mtz for a program that would rather sum them itself. Stills have no rocking events and are the same either way.

Systematic absences. Lattice-centring absences are not written; screw and glide absences are. Prediction runs in a primitive setting so that the space-group search can test the centring, but the interstitial reflections that leaves make a reading program take the lattice for primitive and demote the group. Screw and glide absences are kept because they are the evidence the space group was chosen on — deleting them would turn a reading program's test into an assumption. XDS and DIALS draw the line in the same place.

Scan axis. The batch headers carry the goniometer axis negated relative to the one in the input file. This is not a correction to the file: rugnux brings an observation made at angle φ back to zero by rotating it by +φ, so the crystal itself turns by −φ, and an MTZ batch header records the axis a batch's own increasing PHI turns the crystal about. With the sign as exported, pointless's independently determined orientation matrix agrees with rugnux's to well under a degree.

Taking the data onward

The reflection files are inputs to other suites, and the handover has a few conventions worth one line each. These are the minimum commands that get each program running on rugnux output.

phenix. The merged files carry both the mean intensity and the Bijvoet pairs, and phenix refuses to choose between two equally usable intensity arrays — Sorry: Multiple equally suitable arrays of observed xray data found. (from the mmCIF the message at least lists the choices; from the MTZ it names none, which is where a user is genuinely stuck). It is one behaviour surfacing in both formats; only the label vocabulary differs, following each format's own column names — the MTZ label fails on the mmCIF with Sorry: No matching array:

phenix.xtriage myrun.mtz xray_data.obs_labels=IMEAN,SIGIMEAN
phenix.xtriage myrun.cif xray_data.obs_labels=intensity_meas
phenix.refine  model.pdb myrun.mtz miller_array.labels.name=IMEAN

(phenix.refine takes a different parameter name for the same choice.)

The R-free convention. FreeR_flag is 0 = work, 1 = free (the phenix/CNS convention; 5 % free by default). phenix.refine detects that on its own. REFMAC5 does not: its default takes flag 0 as the free set and stops with Error ==> Cannot switch free R flag — give it the keyword FREE 1:

refmac5 XYZIN model.pdb HKLIN myrun.mtz XYZOUT refined.pdb HKLOUT refined.mtz <<eof
LABIN FP=F SIGFP=SIGF FREE=FreeR_flag
FREE 1
NCYC 10
END
eof

POINTLESS / AIMLESS. myrun_unmerged.mtz opens in both directly — it is sorted the way AIMLESS requires and its header carries the determined space group (see The unmerged export). Running pointless first remains the safe route, and its independent space-group opinion is what the file exists for:

pointless HKLIN myrun_unmerged.mtz HKLOUT sorted.mtz
aimless   HKLIN sorted.mtz         HKLOUT scaled.mtz

Several sweeps of one crystal form go in as separate HKLINs to the same pointless run — that is how sweeps rugnux does not combine itself are merged.

careless wants exactly what the unmerged export is — unmerged, unscaled, LP-only intensities with the partiality reported and not divided out. Against its published examples, two renames: BG/SIGBG are called BGMEAN/BGVAR here and BGVAR is a variance, not a sigma; there is no QE column. Hobs/Kobs/Lobs are reconstructed from M/ISYM by reciprocalspaceship, and dHKL careless computes from the cell, so the metadata string that names this file's columns is

careless mono --anomalous "BATCH,dHKL,Hobs,Kobs,Lobs,XDET,YDET,BGMEAN,BGVAR,LP,FRACTIONCALC" \
    myrun_unmerged.mtz out/myrun

Molecular replacement. When the report names an enantiomorphic pair (P 41 or P 43), let Phaser try both hands: SGALTERNATIVE SELECT HAND. When section 4's candidate table shows a close runner-up, SGALTERNATIVE SELECT ALL searches the alternatives, and myrun_P1.mtz is there for re-merging in a subgroup when the call itself is in doubt.

iotbx.merging_statistics myrun_unmerged.mtz and SHELXC on myrun.hkl need no arguments or label choices at all.

Comparing the geometry with XDS

Every run logs the detector geometry a second time in XDS's convention, so it can be read straight across against the IDXREF.LP / CORRECT.LP of an XDS run on the same data:

XDS convention: ORGX= 1091.00 ORGY= 1137.00 DETECTOR_DISTANCE= 75.0000
XDS convention: DIRECTION_OF_DETECTOR_X-AXIS= 1.000000 0.000000 0.000000
XDS convention: DIRECTION_OF_DETECTOR_Y-AXIS= 0.000000 1.000000 0.000000
XDS convention: INCIDENT_BEAM_DIRECTION= 0 0 1 X-RAY_WAVELENGTH= 1.000000 QX= QY= 0.075000
XDS convention: ROTATION_AXIS= -1.000000 0.000000 0.000000

XDS is never given this geometry — the XDS plugin supplies image data only, and XDS refines its own from XDS.INP — which is what makes the comparison worth having. The two laboratory frames coincide (x along increasing detector column, y along increasing row, z along the beam), so the numbers are directly comparable, and a tilt appears as the two detector axis vectors rather than as angles, which is how XDS reports it after refinement. Two things to keep in mind: ORGX/ORGY are 1-based, because XDS counts pixels from 1 and Jungfraujoch from 0; and they are the PONI, the same quantity Jungfraujoch's beam centre is — so no correction is needed — but not the direct beam once the detector is tilted (see above).