Jungfraujoch/docs/RUGNUX.md

Rugnux

rugnux is the offline crystallographic data-analysis tool of Jungfraujoch — the data-processing half of the system (see Naming for where the name comes from). It takes a stored dataset from the diffraction images to merged data — spot finding, indexing, geometry refinement, Bragg integration, scaling and merging, written as .mtz/.cif/.hkl with the unmerged MTZ and the results report — and, with an atomic model, on to a refined structure and maps: the model is placed against the merge, refined, and written with its map coefficients and the difference-density blobs it leaves unexplained. The per-image _process.h5 is written only on request (--write-process-h5), or as the only output with --no-merge. A rotation run that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits as failed.

It runs the same analysis code as the online and interactive tools, driven from the command line over files.

rugnux {<options>} <input.h5> {<input.h5>} [<broker URL>]

Run it with no arguments to print the usage.

Several Jungfraujoch/NXmx masters of stills, given one after another, are read as one run: their images are taken in the order given, indexed and merged together, and the log gives the indexing rate of each input beside the run's. They must be one setup - the same detector, wavelength, distance and beam centre (to 0.1%, 0.1 mm and 1 pixel), since stills geometry refinement refines one geometry for all frames - and a pixel masked in any of them is masked for all. Stills in the other formats are given the same way, as all their files: rugnux run*.h5 (SACLA), rugnux *.cxi, rugnux *.xtc. Inputs are always files, read in the order given (a shell glob gives them sorted by name); a directory is refused. Rotation sweeps are not read together.

Given the URL of a jfjoch_broker (http://host:port/) and the path the master file will have, rugnux follows the broker's current collection while it is being written: the data files are read as the broker reports them closed, processing starts on the first images, and the run ends with the images the collection delivered. A decision made early on a near tie is re-checked on all frames once the collection is complete. If the check gives a different answer, the run is repeated, so the result is that of the whole sweep. This is for rotation data and --mode mx; the collection's token is read from JUNGFRAUJOCH_HTTP_TOKEN and sent only to the broker. With the master file already on disk, the URL is ignored.

Note. rugnux is under very active development. This page describes the tool and its options at a high level. Run rugnux with no arguments for its usage - how to run each mode and the basic options; every option is on Command-line options.

:local:
:depth: 2

Quick start

Rugnux is one executable in an archive on the release page, https://gitea.psi.ch/mx/jungfraujoch/releases. On Linux x86-64 (the other platforms are on Download):

VERSION=1.0.0-rc.184
curl -L -O https://gitea.psi.ch/mx/jungfraujoch/releases/download/$VERSION/rugnux-$VERSION-linux-x86_64-cuda12.tgz
mkdir -p rugnux-$VERSION && tar xzf rugnux-$VERSION-linux-x86_64-cuda12.tgz -C rugnux-$VERSION
export PATH=$PWD/rugnux-$VERSION/bin:$PATH

To process a dataset, give rugnux an output prefix and the dataset: its master file (HDF5), or any one image of a CBF, marCCD, SMV or SFRM sequence. The example is the 9ZLO dataset (10.8 GB, 19 files, CC-BY 4.0 on Zenodo): Proteus mirabilis UreE, 3600 EIGER images collected at the Australian Synchrotron.

mkdir -p demo/9zlo && cd demo
for f in ASP0887_09_0114_master.h5 ASP0887_09_0114_data_0000{01..18}.h5; do curl -L -C - -o 9zlo/$f "https://zenodo.org/records/18652652/files/$f?download=1"; done
rugnux -o 9zlo 9zlo/ASP0887_09_0114_master.h5

No other option is needed. Rugnux tells rotation data from serial data by the associated metadata (goniometer axis), determines the unit cell, space group and resolution cutoff from the data, and scales and merges by default. The results are written next to each other under the prefix: 9zlo.mtz (merged intensities and amplitudes), 9zlo.cif, 9zlo.hkl with 9zlo.ins, 9zlo_unmerged.mtz, 9zlo_report.txt (statistics and decisions, written for a person), 9zlo_plot.txt, 9zlo_detector.jpg and 9zlo_log.txt. Without -o, the prefix is output. On a desktop with a 16-core CPU and an NVIDIA RTX 5080 the run takes 17 s with release 1.0.0-rc.183 (minutes with --no-gpu). 9zlo_report.txt then says: space group P 21 21 21, cell 38.363 89.746 106.871 90 90 90, resolution 68.7-1.57 A, completeness 99.7 %, multiplicity 12.7, R_meas 0.120, CC1/2 0.9994, I/sigma 11.6, ISa 23.6. A second run, another GPU or the CPU repeats the space group, the cell to 0.1 % and the resolution to one shell. The last digit of a statistic can differ.

With a model, the run goes on to place and refine it. The AlphaFold prediction of UreE is in this repository under tools/battery/models/, two files; the PAE file is read when it sits beside the model:

curl -L -O https://gitea.psi.ch/mx/jungfraujoch/raw/branch/main/tools/battery/models/AF-P17090-F1-model_v6.cif
curl -L -O https://gitea.psi.ch/mx/jungfraujoch/raw/branch/main/tools/battery/models/AF-P17090-F1-predicted_aligned_error_v6.json
rugnux -o 9zlo --model AF-P17090-F1-model_v6.cif 9zlo/ASP0887_09_0114_master.h5

Molecular replacement places the two copies (MR_DECISION= AGREES), refinement brings them to R-work / R-free 0.271 / 0.297 (the refined R factors can differ in the second decimal between builds), and 9zlo_refined_model.cif and 9zlo_refined_maps.mtz are written beside the merged data; the run takes 22 s. The paper's times do not include the refinement; a current run's do. Every file the run writes is listed under Output files and in A first run in detail; the other example datasets are in the repository's top-level README.

The remaining options are for unusual data and for diagnosis. They are all listed under Command-line options. If a dataset only processes well with one of them, please report it and send the run's _report.txt and _log.txt.

Rotation data

rugnux -o myrun dataset_master.h5          # NXmx / EIGER / Jungfraujoch master file
rugnux -o myrun sweep_00001.cbf            # a CBF, marCCD or SMV sweep: name one of its frames

Serial stills

rugnux -o serial *_master.h5               # several NXmx masters of one setup, read as one run
rugnux -o serial run*.h5                   # all the files of one dataset (SACLA); *.cxi, *.xtc alike

Inputs are always files, never a directory; a shell glob hands them over sorted by name, and they are read in that order and merged together. Without a cell the stills are indexed de novo.

Serial stills with a known cell, space group and a CrystFEL geometry file

rugnux -o serial -C 79,79,38,90,90,90 -S P43212 --crystfel-geometry detector.geom run*.h5

-C is the cell (a,b,c,alpha,beta,gamma), -S the space group as a symbol (P43212) or a number (96). With -S the run puts every crystal on one indexing where the space group admits two, from the data alone. --crystfel-geometry places the panels (and takes the masks it names) for HDF5/CXI input that does not place them itself. The wavelength and the detector distance come from the data files or the geometry file; only where neither states them add --wavelength <A> and --detector-distance <mm> (the distance is then held, not refined — without it a run with -C searches and refines it).

With a model

rugnux -o myrun dataset_master.h5 model.cif     # model and images in any order;
rugnux -o myrun --model model.pdb dataset_master.h5   # or with --model

A model is told from the images by its contents (PDB, or mmCIF with atoms), gzipped or not. After merging it is placed — as a rigid body, or by molecular replacement where that does not place it or the model is a prediction (AlphaFold, ModelCIF) — then refined against the merge (restrained, maximum likelihood; seconds on a GPU, typically minutes on the CPU; --no-refine skips it) with its solvent as given; --refine-solvent (off by default, also with --mode refine) also rebuilds the solvent - waters added at difference peaks and removed from negative density, ions relabelled where their surroundings say so, each site audited - and the run writes the maps (σ_A-weighted 2mFo−DFc / mFo−DFc), the placed and refined model, and the blobs of difference density the model does not explain. R-work / R-free are in the report.

The model does not change the processing: the intensities, the cell and the resolution are the ones the data give without it. What it can change is the space group the files are labelled with: it names the hand of an enantiomorphic pair, which merged intensities cannot tell apart (see below). A different space group the molecular replacement prefers is never taken into the answer; it is written beside it as <prefix>_mr.mtz. The one further change is on serial stills with -C and -S in a space group that admits two indexings: there the model picks which of them the crystals are put on, so the merge comes out in the model's indexing. Without a model the data resolve the ambiguity themselves, consistently but in an arbitrary one of the two. A reference reflection file (-z reference.mtz, also accepted as a positional input) does the same, and steers the processing from the start (see Models and reference data).

Output files

A rotation run that merges — the default — leaves these files next to each other:

myrun.mtz            merged intensities + French-Wilson amplitudes, for CCP4 / phenix
myrun.cif            the same, as mmCIF - the self-describing format, and what to deposit
myrun.hkl            the scaled observations unmerged, as SHELX HKLF 4 - for SHELXL, SHELXC / ANODE
myrun.ins            the SHELX instructions to go with it: CELL, ZERR (the cell's e.s.d.s), LATT, SYMM
myrun_unmerged.mtz   the observations unmerged, scaled by the merge, for pointless / aimless -
                     the largest file of the run (--no-export-unmerged skips it)
myrun_report.txt     what the run determined: cell, space group, statistics, warnings
myrun_plot.txt       one row per image, for plotting how the crystal behaved over the sweep
myrun_detector.jpg   the detector with the pixel mask and the beam-stop shadow drawn on it
myrun_log.txt        the full timestamped log of the run (debug level)

With a model, beside them:

myrun_maps.mtz            map coefficients on the placed model (Coot and PyMOL open it as maps)
myrun_model.cif / .pdb    the model as placed, in the cell and space group of myrun.mtz
myrun_refined_model.cif   the refined model (and .pdb)
myrun_refined_maps.mtz    map coefficients on the refined model
myrun_blobs.json          the unexplained difference-density blobs
myrun_solvent.json        with --refine-solvent: the refined model's waters and ions, each scored by its surroundings, and the waters the build added and removed

A stills run writes the same merged files (its .hkl holds the merged reflections) and, on a detector of several modules, serial_module_placement.json with where each module was placed (and serial_modules.h5 where the refinement moved some). Output files lists every file a run can write.

A run uses all the machine's threads by default. -N limits the thread count, or raises the per-image loop's default ceiling of 16 workers per GPU.

Read myrun_report.txt first: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention.

Terminal output

The terminal shows the run as numbered steps, each with one-line results and the time it took, the warnings among them, and a summary at the end; everything else goes to myrun_log.txt. A note's time is how long the step ran since the line above it, and the rule under a step gives its total, so the steps add up to the wall time. Abridged:

Step 3  Indexing
        ... beam centre kept: measured 7.5 px off, same lattice         1.36 s
        ... lattice triclinic P: 28.56 35.34 63.56 106.0 90.3 90.3      0.00 s
        ... 52/60 validation frames, 24% of their spots on it           0.00 s
        ------------------------------------------------------- done in 1.36 s
Step 4  Integration (geometry pre-pass), 1800 images
        ... 1800 images, 836 per second                                 2.15 s
        ------------------------------------------------------- done in 2.16 s
   ...
Step 9  Scaling and merging
        ... merge in P1: 14464 unique                                   0.64 s
        ... space group P 1 21 1 (No. 4)                                0.05 s
        ... merge in P21: 16847 unique                                  0.26 s
        ... resolution 1.57 A (CC1/2 fall-off at 1.64 A)                0.02 s
        ------------------------------------------------------- done in 1.89 s
   ...
==============================================================================
 Result        WARNINGS: Merged to 1.57 A in P 1 21 1. 3 conditions need
               attention: SCALING_NOT_CONVERGED, MULTIPLE_LATTICES, ICE_RINGS.
 Space group   P 1 21 1
 Unit cell     35.337 28.465 62.937 90.000 105.824 90.000
 Resolution    1.57 A written; the CC1/2 fall-off is fitted at 1.64 A
 Signal        I/sigma 6.0   R_meas 17.2 %   CC1/2 0.995   ISa 10.2
 Wall time     11.78 s
==============================================================================
 With any question about this run, send myrun_report.txt and myrun_log.txt.

On a terminal an integration pass draws a progress bar in place; written to a file or a pipe (a batch job's log) every line starts with the time of day instead, there is no bar, and a long pass says how far it has got once a quarter. Warnings are marked !! and errors ** ERROR:; a warning that comes back from a later merge is shown once more on one line and then only counted. -v puts the whole log on the terminal in place of the steps. The log file carries what is needed to answer a question about a run without the data: version, build, GPU, threads, the command line, the header geometry, every decision with its numbers, and a timestamp on every line.

Notes on the defaults:

• The reflection files are cut at the resolution where CC1/2 drops below 0.30. Every reflection file is resolution-trimmed automatically (--resolution-cutoff cc-logistic, one shell past the crossing); --resolution-cutoff off keeps the full measured range, --scaling-high-resolution fixes the limit by hand. If a Rugnux file reaches lower resolution than another program's on the same data, this cutoff is usually the reason, not missing data.
• Do not give -C or -S for rotation data. A known cell and space group are useful for serial data, where the ffbidx indexer needs a cell. On rotation data they often make low-symmetry cases worse. Let the cell and space group be determined from the data (see Rotation data). -S takes a Hermann-Mauguin symbol (P43212) or a space-group number (96), whichever is to hand.
• -A is not needed for the anomalous signal. A rotation merge always keeps the Bijvoet split: a default run's .mtz carries I(+)/I(-) beside IMEAN, its .hkl every observation at the index it was measured at, and the report gives SIGANO, CC_ANOM and the ANOM_* table (the same merge counted per hand, as an anomalous XDS run counts) on every run. On rotation data -A changes nothing in the processing, the files or the statistics; every decision is taken with Friedel mates merged, -A or not, and the report then only says FRIEDELS_LAW= FALSE. Serial stills are different: their merge keeps no split by default, and there -A is what creates one, merging each Bijvoet hand as a reflection of its own.
• A model names the enantiomorph. Where the data accept the model — it is tested against a null of the same model in random orientations, and MODEL_FIT= in the report says the verdict — --model settles which of P4₁2₁2 and P4₃2₁2 the merged reflections are labelled with — a choice no merged intensity can make. It is a label and nothing more: the two groups have the same rotation operations, so no reflection moves, and in particular I(+) and I(-) are left exactly as measured. Whether the model agrees with the data about the hand is then a real question, and the anomalous difference map answers it — a run says so when the density at the model's atoms comes out inverted.
• -z and --model overlap but are not the same. A reference MTZ steers the processing from the start; a model scores the merge and settles the frame it is written in — where the data accept it; a model they reject changes nothing. Either resolves an indexing ambiguity in its own indexing. On serial data the ambiguity decides whether the merged intensities are usable at all; there the space group (-S) alone already lets the run make the crystals consistent from the data.
• Small-molecule data need no extra options. Short axes, spots wider than the integration disk and glide-plane space groups are handled by default. myrun.hkl can be given directly to SHELXT / SHELXL (Small-molecule data).
• --scaling-high-resolution <d>, where the resolution is already known, sharpens both the space-group search and the error model.
• A run wants memory in proportion to what it integrates, not to the detector: up to about 14 GB of host RAM and 7 GB on the card for an ordinary sweep, both peaking in scaling and merging. Installing Rugnux ▸ Memory has the table and the two flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of GB of host RAM (Very large unit cells).
• When a run fails it exits non-zero and says why on the terminal; VERDICT= and the WARNING: lines of the report say what needs attention on a run that finished (The results report). A rotation run that indexes nothing most often has a wrong header: the beam centre is measured on every run, but the distance and the wavelength are indexed at the values stated — --detector-distance and --wavelength override them (--beam-x/--beam-y the centre); serial stills that index poorly want the cell (-C). A run that ends with Killed ran out of memory (Out of host memory). With any question about a run, send myrun_report.txt and myrun_log.txt.
• Everything else is in Running Rugnux and the full Command-line options.

Manual pages

The manual is split into one page per topic:

• Overview — the pipeline from images to merged data and, with a model, a refined structure, in order. Read this one first.
• Download Rugnux — the archive for Linux, macOS and Windows, and the three commands that run it.
• Installing Rugnux — packages, the release archive, GPU drivers, building from source, hardware.
• Input formats — will it open your data: NXmx / EIGER masters, PILATUS miniCBF, marCCD and SMV (ADSC, Rigaku d*TREK) sweeps, one sweep per input, and serial XFEL stills from SACLA (Cheetah HDF5), Cheetah CXI single-panel files and LCLS-I XTC.
• Running Rugnux — a first run in detail, rotation, serial and small-molecule data, and every file a run writes.
• Rugnux with other programs — the reflection-file conventions, the unmerged export, and worked command lines for phenix, REFMAC, POINTLESS / AIMLESS, careless, Phaser, SHELXC/D/E and, for small molecules, SHELXT / SHELXL.
• The results report — the KEY= value interface, sweep quality and the anisotropy section.
• Command-line options — how to run each mode, and every option with what it does.
• Advanced usage — reference data and the indexing ambiguity, model validation and re-merging.
• Models and reference data — every model and reference option, and decision by decision what a model can and cannot change.
• Detector calibration — the geometry from a calibrant's powder rings (--mode calibration).
• Measuring Rugnux — the validation battery, the REFMAC check, the timing benchmark, and how to take timings on another machine that compare with ours.
• CPU/GPU data analysis — the algorithms behind all of it.

The three analysis tools

Tool	Mode	Driven by	Output
jfjoch_broker	Online, real-time streaming analysis on FPGA + GPU	HTTP/REST + ZeroMQ	Live results and statistics, images streamed to jfjoch_writer
jfjoch_viewer	Interactive, on-screen exploration	Qt desktop application	On screen; a processing job can write the same files as rugnux
rugnux	Offline batch processing of a stored dataset	Command-line interface	.mtz/.cif/.hkl and the report; _process.h5 with --no-merge or --write-process-h5

Use rugnux to re-analyse data after acquisition, to experiment with processing parameters, or to produce merged intensities for downstream structure solution.

How to cite

If you find Rugnux useful, please cite F. Leonarski and M. Wang, "Rugnux: from diffraction images to structure at synchrotrons and XFELs in one program" (2026), bioRxiv doi:10.64898/2026.10.24.761522. Rugnux prints this at startup and at the head of its report; the other papers are on the Publications page.