Jungfraujoch/docs/RUGNUX.md

Rugnux

rugnux is the offline crystallographic data-analysis tool of Jungfraujoch — the data-processing half of the system (see Naming for where the name comes from). It takes a stored dataset from the diffraction images to merged data — spot finding, indexing, geometry refinement, Bragg integration, scaling and merging, written as .mtz/.cif/.hkl with the unmerged MTZ and the results report — and, with an atomic model, on to a refined structure and maps: the model is placed against the merge, refined, and written with its map coefficients and the difference-density blobs it leaves unexplained. The per-image _process.h5 is written only on request (--write-process-h5), or as the only output with --no-merge. A rotation run that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits as failed.

It runs the same analysis code as the online and interactive tools, driven from the command line over files.

rugnux {<options>} <input.h5> {<input.h5>} [<broker URL>]

Run it with no arguments to print the usage.

Several Jungfraujoch/NXmx masters of stills, given one after another, are read as one run: their images are taken in the order given, indexed and merged together, and the log gives the indexing rate of each input beside the run's. They must be one setup - the same detector, wavelength, distance and beam centre (to 0.1%, 0.1 mm and 1 pixel), since stills geometry refinement refines one geometry for all frames - and a pixel masked in any of them is masked for all. Stills in the other formats are given the same way, as all their files: rugnux run*.h5 (SACLA), rugnux *.cxi, rugnux *.xtc. Inputs are always files, read in the order given (a shell glob gives them sorted by name); a directory is refused. Rotation sweeps are not read together.

Given the URL of a jfjoch_broker (http://host:port/) and the path the master file will have, rugnux follows the broker's current collection while it is being written: the data files are read as the broker reports them closed, processing starts on the first images, and the run ends with the images the collection delivered. A choice made early on a near tie is checked again on every frame once all are in, and the run is made again where that check ends elsewhere, so the answer is the one the whole sweep gives. This is for rotation data and --mode mx; the collection's token is read from JUNGFRAUJOCH_HTTP_TOKEN and sent only to the broker. With the master file already on disk, the URL is ignored.

Note. rugnux is under very active development. This page describes the tool and its options at a high level; the authoritative, always-current list of options is the program's own usage message — run rugnux with no arguments.

:local:
:depth: 2

Quick start

Give rugnux the images, and nothing else. A dataset carrying a goniometer axis that turns is processed as a rotation sweep, one without (or with an axis that stands still) as serial stills; indexing, the space group, the cell, the resolution limit and the error model are all decided from the data, and scaling and merging run by default. -o names the output files (default output).

We expect no need to tune anything else. The defaults are meant to be the answer, and a run that needed a flag to do well is one we want to hear about. The other options exist for unusual cases and for diagnosis; they are listed under Command-line options.

Rotation data

rugnux -o myrun dataset_master.h5          # NXmx / EIGER / Jungfraujoch master file
rugnux -o myrun sweep_00001.cbf            # a CBF, marCCD or SMV sweep: name its first frame

Serial stills

rugnux -o serial *_master.h5               # several NXmx masters of one setup, read as one run
rugnux -o serial run*.h5                   # all the files of one dataset (SACLA); *.cxi, *.xtc alike

Inputs are always files, never a directory; a shell glob hands them over sorted by name, and they are read in that order and merged together. Without a cell the stills are indexed de novo.

Serial stills with a known cell, space group and a CrystFEL geometry file

rugnux -o serial -C 79,79,38,90,90,90 -S P43212 --crystfel-geometry detector.geom run*.h5

-C is the cell (a,b,c,alpha,beta,gamma), -S the space group as a symbol (P43212) or a number (96). With -S the run puts every crystal on one indexing where the space group admits two, from the data alone. --crystfel-geometry places the panels (and takes the masks it names) for HDF5/CXI input that does not place them itself. The wavelength and the detector distance come from the data files or the geometry file; only where neither states them add --wavelength <A> and --detector-distance <mm> (the distance is then held, not refined — without it a run with -C searches and refines it).

With a model

rugnux -o myrun dataset_master.h5 model.cif     # model and images in any order;
rugnux -o myrun --model model.pdb dataset_master.h5   # or with --model

A model is told from the images by its contents (PDB, or mmCIF with atoms), gzipped or not. After merging it is placed — as a rigid body, or by molecular replacement where that does not place it or the model is a prediction (AlphaFold, ModelCIF) — then refined against the merge (restrained, maximum likelihood; seconds on a GPU, typically minutes on the CPU; --no-refine skips it), and the run writes the maps (σ_A-weighted 2mFo−DFc / mFo−DFc), the placed and refined model, and the blobs of difference density the model does not explain. R-work / R-free are in the report.

The model does not change processing decisions: the merge, its space group and resolution are the ones the data give without it (it only names the hand of an enantiomorphic pair, see below). The one exception is serial stills with -C and -S in a space group that admits two indexings: there the model picks which of them the crystals are put on, so the merge comes out in the model's indexing. Without a model the data resolve the ambiguity themselves, consistently but in an arbitrary one of the two. A reference reflection file (-z reference.mtz, also accepted as a positional input) does the same, and steers the processing from the start (see Models and reference data).

What comes out

A rotation run that merges — the default — leaves these files next to each other:

myrun.mtz            merged intensities + French-Wilson amplitudes, for CCP4 / phenix
myrun.cif            the same, as mmCIF - the self-describing format, and what to deposit
myrun.hkl            the scaled observations unmerged, as SHELX HKLF 4 - for SHELXL, SHELXC / ANODE
myrun.ins            the SHELX instructions to go with it: CELL, ZERR (the cell's e.s.d.s), LATT, SYMM
myrun_unmerged.mtz   every observation before scaling, for pointless / aimless / careless -
                     the largest file of the run (--no-export-unmerged skips it)
myrun_report.txt     what the run determined: cell, space group, statistics, warnings
myrun_plot.txt       one row per image, for plotting how the crystal behaved over the sweep
myrun_detector.jpg   the detector with the pixel mask and the beam-stop shadow drawn on it
myrun_log.txt        the full timestamped log of the run (debug level)

With a model, beside them:

myrun_maps.mtz            map coefficients on the placed model (Coot and PyMOL open it as maps)
myrun_model.cif / .pdb    the model as placed, in the cell and space group of myrun.mtz
myrun_refined_model.cif   the refined model (and .pdb)
myrun_refined_maps.mtz    map coefficients on the refined model
myrun_blobs.json          the unexplained difference-density blobs

A stills run writes the same merged files (its .hkl holds the merged reflections) and, on a detector of several modules, serial_modules.h5 with where each module was placed. Output files lists every file a run can write.

Parallelism needs no asking for: a run already uses the machine's threads. -N is there to limit that, or to lift the per-image loop's default ceiling of 16 workers per GPU.

Read myrun_report.txt first: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention.

What the terminal shows

The terminal shows the run as numbered steps, each with one-line results and the time it took, the warnings among them, and a summary at the end; everything else goes to myrun_log.txt. A note's time is how long the step ran since the line above it, and the rule under a step gives its total, so the steps add up to the wall time. Abridged:

Step 3  Indexing
        ... beam centre kept: measured 7.5 px off, same lattice         1.36 s
        ... lattice triclinic P: 28.56 35.34 63.56 106.0 90.3 90.3      0.00 s
        ... 52/60 validation frames, 24% of their spots on it           0.00 s
        ------------------------------------------------------- done in 1.36 s
Step 4  Integration (geometry pre-pass), 1800 images
        ... 1800 images, 836 per second                                 2.15 s
        ------------------------------------------------------- done in 2.16 s
   ...
Step 9  Scaling and merging
        ... merge in P1: 14464 unique                                   0.64 s
        ... space group P 1 21 1 (No. 4)                                0.05 s
        ... merge in P21: 16847 unique                                  0.26 s
        ... resolution 1.57 A (CC1/2 fall-off at 1.64 A)                0.02 s
        ------------------------------------------------------- done in 1.89 s
   ...
==============================================================================
 Result        WARNINGS: Merged to 1.57 A in P 1 21 1. 3 conditions need
               attention: SCALING_NOT_CONVERGED, MULTIPLE_LATTICES, ICE_RINGS.
 Space group   P 1 21 1
 Unit cell     35.337 28.465 62.937 90.000 105.824 90.000
 Resolution    1.57 A written; the CC1/2 fall-off is fitted at 1.64 A
 Signal        I/sigma 6.0   R_meas 17.2 %   CC1/2 0.995   ISa 10.2
 Wall time     11.78 s
==============================================================================
 With any question about this run, send myrun_report.txt and myrun_log.txt.

On a terminal an integration pass draws a progress bar in place; written to a file or a pipe (a batch job's log) every line starts with the time of day instead, there is no bar, and a long pass says how far it has got once a quarter. Warnings are marked !! and errors ** ERROR:; a warning that comes back from a later merge is shown once more on one line and then only counted. -v puts the whole log on the terminal in place of the steps. The log file carries what is needed to answer a question about a run without the data: version, build, GPU, threads, the command line, the header geometry, every decision with its numbers, and a timestamp on every line.

A few things worth knowing before reaching for more flags:

• The written reflections stop where CC1/2 falls through 0.30. Every reflection file is resolution-trimmed automatically (--resolution-cutoff cc-logistic, one shell past the crossing); --resolution-cutoff off keeps the full measured range, --scaling-high-resolution fixes the limit by hand. A Rugnux file reaching less far than another program's on the same data is usually this default at work, not lost data.
• Rotation data are best left de novo. Pinning the cell and space group (-C, -S) is the normal thing to do for serial stills, where the ffbidx indexer needs a cell; on a rotation sweep it tends to degrade low-symmetry cases, so give the images alone and let the run determine both (see Rotation data). -S takes a Hermann-Mauguin symbol (P43212) or a space-group number (96), whichever is to hand.
• Anomalous data are there without -A. A rotation merge always keeps the Bijvoet split: a default run's .mtz carries I(+)/I(-) beside IMEAN, and its .hkl every observation at the index it was measured at — FRIEDELS_LAW= TRUE in the report says how the statistics were counted, not that the signal was averaged away. What -A changes is the counting basis and the error model: each hand becomes a merged observation of its own, so multiplicity, completeness and ⟨I/σ⟩ are counted anomalously and the sigmas are refitted on the Bijvoet-separated merge. Reach for it for anomalous statistics; the signal itself is in the file either way. (Stills merges carry no split by default — there -A is what creates one.)
• A model names the enantiomorph. Where the data accept the model — it is tested against a null of the same model in random orientations, and MODEL_FIT= in the report says the verdict — --model settles which of P4₁2₁2 and P4₃2₁2 the merged reflections are labelled with — a choice no merged intensity can make. It is a label and nothing more: the two groups have the same rotation operations, so no reflection moves, and in particular I(+) and I(-) are left exactly as measured. Whether the model agrees with the data about the hand is then a real question, and the anomalous difference map answers it — a run says so when the density at the model's atoms comes out inverted.
• -z and --model overlap but are not the same. A reference MTZ steers the processing from the start; a model scores the merge and settles the frame it is written in — where the data accept it; a model they reject changes nothing. Either resolves an indexing ambiguity in its own indexing. On serial data the ambiguity decides whether the merged intensities are usable at all; there the space group (-S) alone already lets the run make the crystals consistent from the data.
• Small molecules need no flag either. Short axes, spots wider than the integration disk and glide-plane space groups are handled by the default run, and myrun.hkl goes straight to SHELXT / SHELXL (Small-molecule data).
• --scaling-high-resolution <d>, where the resolution is already known, sharpens both the space-group search and the error model.
• A run wants memory in proportion to what it integrates, not to the detector: 2.5-14 GB of host RAM and 3-7 GB on the card over the datasets measured, both peaking in scaling and merging. Installing Rugnux ▸ Memory has the table and the two flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of GB of host RAM (Very large unit cells).
• When a run fails it exits non-zero and says why on the terminal; VERDICT= and the WARNING: lines of the report say what needs attention on a run that finished (The results report). A rotation run that indexes nothing most often has a wrong header: the beam centre is measured on every run, but the distance and the wavelength are indexed at the values stated — --detector-distance and --wavelength override them (--beam-x/--beam-y the centre); serial stills that index poorly want the cell (-C). A run that ends with Killed ran out of memory (When host memory runs out). With any question about a run, send myrun_report.txt and myrun_log.txt.
• Everything else is in Running Rugnux and the full Command-line options.

The rest of the manual

One page per job, so the answer needed is near the top of a short page:

• What Rugnux does — the pipeline from images to merged data and, with a model, a refined structure, in order. Read this one first.
• Download Rugnux — the archive for Linux, macOS and Windows, and the three commands that run it.
• Installing Rugnux — packages, the release archive, GPU drivers, building from source, hardware.
• What Rugnux reads — will it open your data: NXmx / EIGER masters, PILATUS miniCBF, marCCD and SMV (ADSC, Rigaku d*TREK) sweeps, one sweep per input, and serial XFEL stills from SACLA (Cheetah HDF5), Cheetah CXI single-panel files and LCLS-I XTC.
• Running Rugnux — a first run in detail, rotation, serial and small-molecule data, and every file a run writes.
• Rugnux with other programs — the reflection-file conventions, the unmerged export, and worked command lines for phenix, REFMAC, POINTLESS / AIMLESS, careless, Phaser, SHELXC/D/E and, for small molecules, SHELXT / SHELXL.
• The results report — the KEY= value interface, sweep quality and the anisotropy section.
• Advanced usage — reference data and the indexing ambiguity, model validation, re-merging, and the full command-line option tables.
• Models and reference data — every model and reference option, and decision by decision what a model can and cannot change.
• Detector calibration — the geometry from a calibrant's powder rings (--mode calibration).
• Measuring Rugnux — the validation battery, the REFMAC check, the timing benchmark, and how to take timings on another machine that compare with ours.
• CPU/GPU data analysis — the algorithms behind all of it.

Where it fits among the three analysis tools

Tool	Mode	Driven by	Output
jfjoch_broker	Online, real-time streaming analysis on FPGA + GPU	HTTP/REST + ZeroMQ	Live results and statistics, images streamed to jfjoch_writer
jfjoch_viewer	Interactive, on-screen exploration	Qt desktop application	On screen; a processing job can write the same files as rugnux
rugnux	Offline batch processing of a stored dataset	Command-line interface	.mtz/.cif/.hkl and the report; _process.h5 with --no-merge or --write-process-h5

Use rugnux to re-analyse data after acquisition, to experiment with processing parameters, or to produce merged intensities for downstream structure solution.