diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index aff15700b..07aea9754 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -1,14 +1,15 @@ # Rugnux `rugnux` is the **offline** crystallographic data-analysis tool of Jungfraujoch — the -data-processing half of the system (see [Naming](NAMING.md) for where the name comes from). -It takes a stored dataset, runs the full analysis pipeline — spot finding, indexing, geometry -refinement, Bragg integration, scaling and merging — and writes the merged reflections -(`.mtz`/`.cif`/`.hkl`), the unmerged MTZ and the results report; the per-image `_process.h5` is -written only on request (`--write-process-h5`), or as the only output with `--no-merge`. Given an -atomic model, it also validates the merge against it and refines the model it placed. A rotation run -that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits -as failed. +data-processing half of the system (see [Naming](NAMING.md) for where the name comes from). It takes +a stored dataset from the diffraction images to merged data — spot finding, indexing, geometry +refinement, Bragg integration, scaling and merging, written as `.mtz`/`.cif`/`.hkl` with the +unmerged MTZ and the results report — and, with an atomic model, on to a refined structure and maps: +the model is placed against the merge, refined, and written with its map coefficients and the +difference-density blobs it leaves unexplained. The per-image `_process.h5` is written only on +request (`--write-process-h5`), or as the only output with `--no-merge`. A rotation run that finds +no lattice still spot-finds every image and writes their per-image analysis there, then exits as +failed. It runs the *same* analysis code as the online and interactive tools, driven from the command line over files. @@ -48,35 +49,72 @@ is ignored. ## Quick start -Four commands cover most of what people ask of `rugnux`. Each takes the **master** file of the -dataset — one written by Jungfraujoch, a DECTRIS EIGER master, or an NXmx master written by another -facility's toolchain; a PILATUS miniCBF, marCCD or SMV sweep works too (see [What Rugnux reads](RUGNUX_FORMATS.md)) -— and names its output files from `-o`: +Give `rugnux` the images, and nothing else. A dataset carrying a **goniometer axis** that turns is +processed as a rotation sweep, one without (or with an axis that stands still) as **serial stills**; +indexing, the space group, the cell, the resolution limit and the error model are all decided from +the data, and scaling and merging run by default. `-o` names the output files (default `output`). + +**We expect no need to tune anything else.** The defaults are meant to be the answer, and a run that +needed a flag to do well is one we want to hear about. The other options exist for unusual cases and +for diagnosis; they are listed under [Command-line options](RUGNUX_ADVANCED.md#command-line-options). + +### Rotation data ``` -# 1. everything from the data - index, integrate, scale and merge with the defaults -rugnux -o myrun dataset_master.h5 - -# 2. with a reference dataset of the same crystal form: it fixes the space group and the cell, -# resolves the indexing ambiguity, and hands over its R-free set -rugnux -o myrun -z reference.mtz dataset_master.h5 - -# 3. with a known structure: R-work / R-free, sigma_A-weighted 2mFo-DFc / mFo-DFc maps and the -# refined model (a predicted model, or one the rigid body cannot place, goes to molecular -# replacement); `rugnux -o myrun dataset_master.h5 model.pdb` does the same -rugnux -o myrun --model model.pdb dataset_master.h5 - -# 4. with the space group and the cell pinned (-S takes either spelling: P43212 or 96) -rugnux -o myrun -S P43212 -C 79,79,38,90,90,90 dataset_master.h5 +rugnux -o myrun dataset_master.h5 # NXmx / EIGER / Jungfraujoch master file +rugnux -o myrun sweep_00001.cbf # a CBF, marCCD or SMV sweep: name its first frame ``` -Parallelism needs no asking for: a run already uses the machine's threads. `-N` is there to *limit* -that, or to lift the per-image loop's default ceiling of 16 workers per GPU. +### Serial stills -Nothing more is needed to pick the workflow: a dataset carrying a **goniometer axis** that turns is -processed as a rotation sweep, one without (or with an axis that stands still) as **independent -stills**, and scaling and merging run by default -in both. A rotation run that merges — the default — leaves nine files next to each other: +``` +rugnux -o serial *_master.h5 # several NXmx masters of one setup, read as one run +rugnux -o serial run*.h5 # all the files of one dataset (SACLA); *.cxi, *.xtc alike +``` + +Inputs are always files, never a directory; a shell glob hands them over sorted by name, and they are +read in that order and merged together. Without a cell the stills are indexed de novo. + +### Serial stills with a known cell, space group and a CrystFEL geometry file + +``` +rugnux -o serial -C 79,79,38,90,90,90 -S P43212 --crystfel-geometry detector.geom run*.h5 +``` + +`-C` is the cell (`a,b,c,alpha,beta,gamma`), `-S` the space group as a symbol (`P43212`) or a number +(`96`). With `-S` the run puts every crystal on one indexing where the space group admits two, from +the data alone. `--crystfel-geometry` places the panels (and takes the masks it names) for HDF5/CXI +input that does not place them itself. The wavelength and the detector distance come from the data +files or the geometry file; only where neither states them add `--wavelength ` and +`--detector-distance ` (the distance is then held, not refined — without it a run with `-C` +searches and refines it). + +### With a model + +``` +rugnux -o myrun dataset_master.h5 model.cif # model and images in any order; +rugnux -o myrun --model model.pdb dataset_master.h5 # or with --model +``` + +A model is told from the images by its contents (PDB, or mmCIF with atoms), gzipped or not. After +merging it is **placed** — as a rigid body, or by molecular replacement where that does not place it or +the model is a prediction (AlphaFold, ModelCIF) — then **refined** against the merge (restrained, +maximum likelihood; seconds on a GPU, typically minutes on the CPU; `--no-refine` skips it), and the +run writes the maps (σA-weighted 2mFo−DFc / mFo−DFc), the placed and refined model, and +the blobs of difference density the model does not explain. R-work / R-free are in the report. + +The model **does not change processing decisions**: the merge, its space group and resolution are +the ones the data give without it (it only names the hand of an enantiomorphic pair, see below). +The one exception is serial stills with `-C` and `-S` in a space +group that admits two indexings: there the model picks which of them the crystals are put on, so the +merge comes out in the model's indexing. Without a model the data resolve the ambiguity themselves, +consistently but in an arbitrary one of the two. A reference reflection file (`-z reference.mtz`, +also accepted as a positional input) does the same, and steers the processing from the start (see +[Models and reference data](RUGNUX_MODEL_AND_REFERENCE.md)). + +### What comes out + +A rotation run that merges — the default — leaves these files next to each other: ``` myrun.mtz merged intensities + French-Wilson amplitudes, for CCP4 / phenix @@ -91,10 +129,22 @@ myrun_detector.jpg the detector with the pixel mask and the beam-stop shadow d myrun_log.txt the full timestamped log of the run (debug level) ``` -The unmerged MTZ is most of the bytes a run writes — worth knowing when sizing a scratch -directory for a campaign; `--no-export-unmerged` skips it. With a model, the maps, the placed model -and the refined model are written beside these -([Output files](RUGNUX_TUTORIAL.md#output-files) lists every file a run can write). +With a model, beside them: + +``` +myrun_maps.mtz map coefficients on the placed model (Coot and PyMOL open it as maps) +myrun_model.cif / .pdb the model as placed, in the cell and space group of myrun.mtz +myrun_refined_model.cif the refined model (and .pdb) +myrun_refined_maps.mtz map coefficients on the refined model +myrun_blobs.json the unexplained difference-density blobs +``` + +A stills run writes the same merged files (its `.hkl` holds the merged reflections) and, on a +detector of several modules, `serial_modules.h5` with where each module was placed. +[Output files](RUGNUX_TUTORIAL.md#output-files) lists every file a run can write. + +Parallelism needs no asking for: a run already uses the machine's threads. `-N` is there to *limit* +that, or to lift the per-image loop's default ceiling of 16 workers per GPU. Read `myrun_report.txt` first: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention. @@ -150,9 +200,9 @@ A few things worth knowing before reaching for more flags: `--resolution-cutoff off` keeps the full measured range, `--scaling-high-resolution` fixes the limit by hand. A Rugnux file reaching less far than another program's on the same data is usually this default at work, not lost data. -- **Rotation data are best left de novo.** Pinning the cell and space group (recipe 4) is the +- **Rotation data are best left de novo.** Pinning the cell and space group (`-C`, `-S`) is the normal thing to do for **serial stills**, where the `ffbidx` indexer needs a cell; on a rotation - sweep it tends to *degrade* low-symmetry cases, so prefer recipe 1 and let the run determine both + sweep it tends to *degrade* low-symmetry cases, so give the images alone and let the run determine both (see [Rotation data](RUGNUX_TUTORIAL.md#rotation-data)). `-S` takes a Hermann-Mauguin symbol (`P43212`) or a space-group number (`96`), whichever is to hand. - **Anomalous data are there without `-A`.** A rotation merge always keeps the Bijvoet split: a @@ -203,7 +253,7 @@ A few things worth knowing before reaching for more flags: One page per job, so the answer needed is near the top of a short page: -- [What Rugnux does](RUGNUX_OVERVIEW.md) — the pipeline from images to merged reflections, +- [What Rugnux does](RUGNUX_OVERVIEW.md) — the pipeline from images to merged data and, with a model, a refined structure, in order. Read this one first. - [Download Rugnux](RUGNUX_DOWNLOAD.md) — the archive for Linux, macOS and Windows, and the three commands that run it. diff --git a/docs/RUGNUX_TUTORIAL.md b/docs/RUGNUX_TUTORIAL.md index b7aa76ed0..2ec7e3be5 100644 --- a/docs/RUGNUX_TUTORIAL.md +++ b/docs/RUGNUX_TUTORIAL.md @@ -20,7 +20,8 @@ machine. Nothing is assumed about the crystal — the goniometer axis in the file tells Rugnux this is a rotation sweep, the unit cell comes from indexing the data, the space group from its systematic absences, and the resolution limit from where CC1/2 falls off. Progress, statistics and timing go to the terminal, and the output files land -next to each other. +next to each other. Serial stills and a run with a model start the same way, with the stills' files +or the model added to the inputs ([Quick start](RUGNUX.md#quick-start)). `myrun_report.txt` is written for a person, top to bottom: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention. To pull one number out of @@ -163,31 +164,39 @@ disturbance of the sweep, not dose, and the report says so and points at the swe ## Still / serial data A dataset with **no goniometer axis**, or one whose axis does not turn (e.g. a serial grid scan), is -processed as **independent stills automatically** — no flag needed. Known-cell indexing with the fast-feedback indexer, -then merge against a reference structure: +processed as **independent stills automatically** — no flag needed. A run spread over several files +is given as all of them at once — several NXmx masters of one setup, or the files an XFEL pipeline +writes ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)): ``` -rugnux serial_master.h5 \ - -o serial_run \ - -X ffbidx -C 79,79,38,90,90,90 -S 96 \ - -z reference.mtz \ - --scaling-high-resolution 1.8 +rugnux -o serial *_master.h5 # several NXmx masters, read and merged as one run +rugnux -o serial run*.h5 # all the files of one stills dataset; *.cxi, *.xtc alike ``` +Where the cell and space group are known, give them; where the detector's panels are described by a +CrystFEL geometry file, give that too: + +``` +rugnux -o serial -C 79,79,38,90,90,90 -S P43212 --crystfel-geometry detector.geom run*.h5 +``` + +The wavelength and distance are read from the data or the geometry file; `--wavelength` and +`--detector-distance` are only for files that state neither (a distance given there is held, not +refined). A reference reflection file of the same crystal form (`-z reference.mtz`) or a model +(`model.pdb` as a further input, or `--model`) can be added to any of these. + A crystal form with an [indexing ambiguity](RUGNUX_ADVANCED.md#the-indexing-ambiguity) — P3, P4, P6 and their relatives — needs the space group on the run that integrates: every crystal is indexed in one of the alternative indexings at random, and they are averaged together in the merge unless the crystals are put into one first. With `-S` given, the run does that from the data alone, making the crystals consistent -with each other (`INDEXING_AMBIGUITY_RESOLVED= data`); a reference (`-z` above), or `--model model.pdb` +with each other (`INDEXING_AMBIGUITY_RESOLVED= data`); a reference (`-z`), or `--model model.pdb` together with `-C` and `-S`, puts each image into the reference's indexing instead, so that the merge also comes out in it. Without `-S` and without either, the mixed merge can look more symmetric than the crystal is, and the run warns. `ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills; the -default `-X Auto` picks it for stills whenever a cell is given, so `-X ffbidx` above only makes that -explicit. A run spread over several files is given as all of them at once - `rugnux *_master.h5`, or -the files an XFEL pipeline writes ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). The -self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for +default `-X Auto` picks it for stills whenever a cell is given, so there is no `-X` to set; without a +cell the stills are indexed de novo. The self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this combination raises the indexing rate and typically extends resolution over a fixed threshold and fixed min-pix, at equal or better CC1/2. (You can still pin a fixed threshold with `--spot-sigma` / diff --git a/docs/documentation.rst b/docs/documentation.rst index 7c43af867..f0dab2b51 100644 --- a/docs/documentation.rst +++ b/docs/documentation.rst @@ -14,11 +14,13 @@ Documentation

Data processing

Rugnux

Processes a stored macromolecular diffraction dataset, a rotation - sweep or serial stills, from the images to merged reflections: spot finding, indexing, - integration, scaling and merging, with the space group, cell and resolution decided from the - data and a report that says on what evidence; given an atomic model, it also places and refines - it against the merge. One command, no parameters.

- 
rugnux <diffraction image file>
+ sweep or serial stills, from the images to merged data: spot finding, indexing, integration, + scaling and merging, with the space group, cell and resolution decided from the data and a + report that says on what evidence. Given an atomic model it goes on to a refined structure and + maps: the model is placed (rigid body or molecular replacement) and refined against the merge. + One command, no parameters.

+ 
rugnux <diffraction image file>
+   rugnux <diffraction image file> <CIF/PDB model file>

Linux · macOS · Windows · fast on an NVIDIA GPU, works without one

Get started diff --git a/docs/landing/index.html b/docs/landing/index.html index 8d03dae29..f9c4f6807 100644 --- a/docs/landing/index.html +++ b/docs/landing/index.html @@ -7,8 +7,8 @@ -Jungfraujoch and Rugnux: X-ray diffraction data, from the detector to the merged reflections - +Jungfraujoch and Rugnux: X-ray diffraction data, from the detector to merged data and refined structures +