diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 28cc8328f..aff15700b 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -5,7 +5,8 @@ data-processing half of the system (see [Naming](NAMING.md) for where the name c It takes a stored dataset, runs the full analysis pipeline — spot finding, indexing, geometry refinement, Bragg integration, scaling and merging — and writes the merged reflections (`.mtz`/`.cif`/`.hkl`), the unmerged MTZ and the results report; the per-image `_process.h5` is -written only on request (`--write-process-h5`), or as the only output with `--no-merge`. A rotation run +written only on request (`--write-process-h5`), or as the only output with `--no-merge`. Given an +atomic model, it also validates the merge against it and refines the model it placed. A rotation run that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits as failed. @@ -60,8 +61,9 @@ rugnux -o myrun dataset_master.h5 # resolves the indexing ambiguity, and hands over its R-free set rugnux -o myrun -z reference.mtz dataset_master.h5 -# 3. with a known structure: R-work / R-free and sigma_A-weighted 2mFo-DFc / mFo-DFc maps -# (a predicted model, or one the rigid body cannot place, goes to molecular replacement) +# 3. with a known structure: R-work / R-free, sigma_A-weighted 2mFo-DFc / mFo-DFc maps and the +# refined model (a predicted model, or one the rigid body cannot place, goes to molecular +# replacement); `rugnux -o myrun dataset_master.h5 model.pdb` does the same rugnux -o myrun --model model.pdb dataset_master.h5 # 4. with the space group and the cell pinned (-S takes either spelling: P43212 or 96) @@ -89,8 +91,10 @@ myrun_detector.jpg the detector with the pixel mask and the beam-stop shadow d myrun_log.txt the full timestamped log of the run (debug level) ``` -The two MTZ extras are most of the bytes a run writes — worth knowing when sizing a scratch -directory for a campaign, and both have off switches. +The unmerged MTZ is most of the bytes a run writes — worth knowing when sizing a scratch +directory for a campaign; `--no-export-unmerged` skips it. With a model, the maps, the placed model +and the refined model are written beside these +([Output files](RUGNUX_TUTORIAL.md#output-files) lists every file a run can write). Read `myrun_report.txt` first: it says which space group was chosen and on what evidence, how far the data go, and anything that needs attention. @@ -183,6 +187,14 @@ A few things worth knowing before reaching for more flags: merging. [Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory) has the table and the two flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of GB of host RAM ([Very large unit cells](RUGNUX_INSTALL.md#very-large-unit-cells)). +- **When a run fails** it exits non-zero and says why on the terminal; `VERDICT=` and the `WARNING:` + lines of the report say what needs attention on a run that finished + ([The results report](RUGNUX_REPORT.md#the-summary-and-what-the-run-decided)). A rotation run that indexes + nothing most often has a wrong header: the beam centre is measured on every run, but the distance + and the wavelength are indexed at the values stated — `--detector-distance` and `--wavelength` override them + (`--beam-x`/`--beam-y` the centre); serial stills that index poorly want the cell (`-C`). A run that ends + with `Killed` ran out of memory ([When host memory runs out](RUGNUX_INSTALL.md#when-host-memory-runs-out)). + With any question about a run, send `myrun_report.txt` and `myrun_log.txt`. - Everything else is in [Running Rugnux](RUGNUX_TUTORIAL.md#running-rugnux) and the full [Command-line options](RUGNUX_ADVANCED.md#command-line-options). @@ -223,7 +235,7 @@ One page per job, so the answer needed is near the top of a short page: | --- | --- | --- | --- | | [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) | | [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` | -| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** | +| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`.mtz`/`.cif`/`.hkl` and the report; `_process.h5` with `--no-merge` or `--write-process-h5`** | Use `rugnux` to re-analyse data after acquisition, to experiment with processing parameters, or to produce merged intensities for downstream structure solution. diff --git a/docs/RUGNUX_CALIBRATION.md b/docs/RUGNUX_CALIBRATION.md index f6ec498bb..95fef8562 100644 --- a/docs/RUGNUX_CALIBRATION.md +++ b/docs/RUGNUX_CALIBRATION.md @@ -41,6 +41,10 @@ else, and has two members: **`ice`** calibrates a real experiment against its own ice rings — no calibrant exposure needed — and is the reason a calibrant is a list of ring positions rather than a unit cell: hexagonal ice is P63/mmc, so rings enumerated from its cell would include systematically absent ones. +A standard that is not on the list is given by its unit cell instead: with `-C` the cell **is** the +calibrant, its rings enumerated from the cell, with the absences of `-S` where that is given (taken as +primitive where it is not). `-S` supplies symmetry absences only, so a standard whose extinctions come +from its atomic basis — silicon's 222 — is better named with `--calibrant`. `--calibration` picks how the rings are measured, and **both aggregate over every processed image** — `-s`/`-e`/`-t` select which images: diff --git a/docs/RUGNUX_FORMATS.md b/docs/RUGNUX_FORMATS.md index a93f06f8a..75048fb6d 100644 --- a/docs/RUGNUX_FORMATS.md +++ b/docs/RUGNUX_FORMATS.md @@ -1,8 +1,10 @@ # What Rugnux reads Which data `rugnux` opens, before anything is typed. The short answer: an HDF5 master (NXmx or -DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d\*TREK) — nothing else -is read, so any other format has to be converted to one of these first. +DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d\*TREK), and +for serial stills the files of the XFEL pipelines ([below](#serial-xfel-inputs)) — nothing else is +read, so any other format has to be converted to one of these first. An atomic model and a reference +reflection file may be given among the inputs too, told from the images by their contents. **Input** is files, never a directory: an HDF5 master file, or one frame of a PILATUS miniCBF, marCCD or SMV sweep. For rotation data one input is **one sweep of one crystal** — Rugnux does not combine sweeps or crystals in a run; process each diff --git a/docs/RUGNUX_INSTALL.md b/docs/RUGNUX_INSTALL.md index 512c57d29..ef8d06326 100644 --- a/docs/RUGNUX_INSTALL.md +++ b/docs/RUGNUX_INSTALL.md @@ -78,8 +78,9 @@ The released Linux and Windows archives are CUDA builds (the macOS one is CPU-on an NVIDIA **driver** on the host — 525.60.13 or newer for the CUDA 12 archive, 580.65.06 or newer for the CUDA 13 ones — and no CUDA toolkit, because everything CUDA is linked statically. With no GPU or no driver, `rugnux` reports zero CUDA -devices and falls back to the CPU path, which works but is far slower and offers only the `fftw` -indexer. A **V100 needs the CUDA 12 archive**; which generations each build covers is in +devices and falls back to the CPU path, which works but is far slower: the FFT search runs on FFTW +and `ffbidx` on its CPU implementation, and the refinement of a placed model takes minutes rather than +seconds. A **V100 needs the CUDA 12 archive**; which generations each build covers is in [Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver). ## Building from source @@ -116,8 +117,8 @@ preprocessing, azimuthal integration, spot finding, prediction and Bragg integra pre-scan (beam-stop mask, background beam-centre fit, spot widths, defective pixels; one accumulator per card), rotation scaling and merging from the ingest of the reflections onwards (sorting and the resolution cut included) with its correction surfaces, and `--model`: the structure factors, scale -fits and rigid-body placement of the validation and the rotation and translation functions of -molecular replacement. The CPU implementations are the path where there is no GPU, and the two agree +fits and rigid-body placement of the validation, the rotation and translation functions of +molecular replacement, and the refinement of the placed model. The CPU implementations are the path where there is no GPU, and the two agree to rounding rather than bit for bit (parts of the scaling and the model fits round differently on the card), so a GPU and a CPU run can differ in the last digits of the merged data. The choice is automatic; a CUDA build runs on the CPU when told to (`rugnux --no-gpu ...`). diff --git a/docs/RUGNUX_MODEL_AND_REFERENCE.md b/docs/RUGNUX_MODEL_AND_REFERENCE.md index 03daac042..8c6ce9b20 100644 --- a/docs/RUGNUX_MODEL_AND_REFERENCE.md +++ b/docs/RUGNUX_MODEL_AND_REFERENCE.md @@ -30,7 +30,7 @@ the rest walks through a run step by step. `_mr.mtz`/`.cif`, with the model as placed there. `.mtz` stays the data's own answer. - **A placed model is then refined**, after every decision has been taken, and written beside the - answer as `_refined_model.cif` with its maps ([Refinement](#refinement)); `--no-refine` + answer as `_refined_model.cif`/`.pdb` with its maps ([Refinement](#refinement)); `--no-refine` skips it. The refinement changes nothing else the run writes. - **A reference dataset (`-z`) and a space group or cell you give (`-S`, `-C`) are different:** they are taken as facts about the crystal. They fix the space group and the cell before processing, @@ -51,7 +51,7 @@ the rest walks through a run step by step. | Option | What you give | What Rugnux uses it for | Can it change `.mtz`? | |---|---|---|---| | *(none)* | — | Everything from the data | — | -| `--model ` | An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then [refinement](#refinement) of the placed model (`REFINE_` keys, `_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json`), which decides nothing | Labels only (hand, indexing, axes), and only where the model fits. Other space groups → `_mr.*` | +| `--model ` | An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then [refinement](#refinement) of the placed model (`REFINE_` keys, `_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json`), which decides nothing | Labels only (hand, indexing, axes), and only where the model fits. Other space groups → `_mr.*` | | `--mr ` | The same kinds of model | As `--model`, but molecular replacement straight away, without the rigid body. `--mode mx` only, not together with `--model` | Hand label only. Other space groups → `_mr.*` | | PAE file beside a prediction | `AF--F1-predicted_aligned_error_v.json` next to `AF--F1-model_v.cif` (read automatically, no option) | Cuts the prediction into rigid domains for molecular replacement | No (a domain alone can only confirm the answer's group) | | `-z, --reference ` (`--reference-mtz`) | Merged data of the same crystal form: an MTZ, or a PDB structure-factor mmCIF (`-sf.cif`) | Fixes the space group (in its own setting) and the cell (softly), puts the merge on its axes, resolves the indexing ambiguity, hands over its R-free set, reports CCref on stills. Never a scale anchor | **Yes**: the space group, the axes, the indexing and the free set are the reference's | @@ -176,7 +176,7 @@ these letters. `_maps.mtz` (map coefficients), `_model.cif`/`.pdb` — or the `_mr_*` equivalents where the choice was `_mr`. A rejected model gets its maps too: that is the case where looking at it in the density is most useful. A placed model is then - [refined](#refinement) (`_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json`). + [refined](#refinement) (`_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json`). 9. **The reflection files** — `.mtz`, `.cif`, `.hkl`, `_unmerged.mtz` — are written last, carrying only the class-(b) labels from step 5 and the hand from step 7. @@ -336,7 +336,7 @@ map coefficients such as FC, FWT, DELFWT are passed over) where the file has the I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from a FreeR_flag-like column (flag 0 is the test set; where 0 marks the majority, as phenix and CNS write it, the minority value is); a file without one gets rugnux's own 5 % set. The refinement is the one run after a merge on a placed model, and -writes the same files - `_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json` - with +writes the same files - `_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json` - with the `REFINE_` keys in `_report.txt`, beside `REFINE_FRENCH_WILSON=` and `REFINE_FREE_FLAGS= FILE|GENERATED` saying what had to be made of the input. @@ -419,7 +419,7 @@ the data, a weak shell says nothing about them, and nothing acts on it. |---|---|---| | `.mtz` / `.cif` / `.hkl` | The answer: the data's space group, with only the class-(b) labels from the model | Refinement, deposition, everything by default | | `_model.cif` / `.pdb`, `_maps.mtz` | The model as placed, in the frame of `.mtz`, and its map coefficients (Coot/PyMOL open them as maps) | Starting refinement (REFMAC5, phenix.refine), PanDDA, Coot | -| `_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json` | The placed model refined by rugnux, its map coefficients, and the unexplained blobs of its difference map | A first look at the refined density and at what the model leaves unexplained; continuing refinement | +| `_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json` | The placed model refined by rugnux, its map coefficients, and the unexplained blobs of its difference map | A first look at the refined density and at what the model leaves unexplained; continuing refinement | | `_twin_.mtz` / `.cif` | The same data merged in a subgroup with a twin law — written from the data, model or not | Twin refinement, where you suspect the higher group absorbed a twin law | | `_mr.mtz` / `.cif`, `_mr_model.pdb` / `.cif`, `_mr_*` maps | The merge in the space group the model chose (`MODEL_CHOICE_SPACE_GROUP=`), and the model placed there | Refinement *if* you accept the model's choice; compare R-free against the answer's | diff --git a/docs/RUGNUX_TUTORIAL.md b/docs/RUGNUX_TUTORIAL.md index 34b8a7255..b7aa76ed0 100644 --- a/docs/RUGNUX_TUTORIAL.md +++ b/docs/RUGNUX_TUTORIAL.md @@ -63,7 +63,8 @@ rugnux -o myrun --write-process-h5 dataset_master.h5 # skip the unmerged MTZ (written by default), when only the merged data is wanted rugnux -o myrun --no-export-unmerged dataset_master.h5 -# check the merged data against a known structure: R-work / R-free and maps +# check the merged data against a known structure: R-work / R-free, maps and the +# refined model (the model may also be given as a second input, without --model) rugnux -o myrun --model model.pdb dataset_master.h5 ``` @@ -182,7 +183,10 @@ together with `-C` and `-S`, puts each image into the reference's indexing inste also comes out in it. Without `-S` and without either, the mixed merge can look more symmetric than the crystal is, and the run warns. -`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The +`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills; the +default `-X Auto` picks it for stills whenever a cell is given, so `-X ffbidx` above only makes that +explicit. A run spread over several files is given as all of them at once - `rugnux *_master.h5`, or +the files an XFEL pipeline writes ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). The self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this combination raises the indexing rate and typically extends resolution over a fixed threshold and @@ -270,6 +274,9 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke stills runs write the merged reflections (Bijvoet mates at `+hkl` and `-hkl`). Intensities are put on a common scale so the largest value fits the fixed-width field, and the file ends with the `0 0 0` terminator record. + - `.ins` — the SHELX instructions to go with the `.hkl`: `CELL` at the wavelength, `ZERR` + with the cell's e.s.d.s, `LATT` and `SYMM` for the space group, and a placeholder `SFAC`/`UNIT` for the + user to replace with the formula (see [Small-molecule structures with SHELXT and SHELXL](RUGNUX_INTEGRATION.md#small-molecule-structures-with-shelxt-and-shelxl)). All three carry the **refined unit cell** (from rotation indexing) and the **space group determined from systematic absences** (constrained to the indexed lattice symmetry). @@ -289,6 +296,10 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for a cell the PDB format cannot hold. See [Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model). +- `_refined_model.cif` / `.pdb`, `_refined_maps.mtz` and `_blobs.json` — + where the model was placed, the model as rugnux refined it, its map coefficients and the blobs of + density its difference map leaves unexplained; `--no-refine` skips them. See + [Refinement](RUGNUX_MODEL_AND_REFERENCE.md#refinement). - `_twin_.mtz` / `.cif` — only where the chosen space group may have absorbed a twin law: the same data merged in a subgroup ``, with the twin law and its fraction, same resolution and free set as `.mtz`, for refinement with that law. With `--model`, the maps come from @@ -300,8 +311,13 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke the model chose, beside the run's own answer, which is left as it was. See [Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model). - `_report.txt` — the **results report**: what the run determined, in a form both a person and - a beamline script can read. Always written, next to the files above. See + a beamline script can read. Written by every `mx` and `scale` run, next to the files above (an empty prefix, `-o ""`, writes nothing). See [The results report](RUGNUX_REPORT.md#the-results-report). +- `_log.txt` — the full timestamped log of the run, at debug level, written by every run with + an output prefix; send it with the report when asking about a run. +- `_modules.h5` — serial stills on a detector of several modules: where the geometry + refinement placed each module, readable back with `--module-geometry` (see + [Module geometry](RUGNUX_FORMATS.md#module-geometry)). - `_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated, with a single `#` legend line over its own columns so gnuplot plots it as it stands (`plot '_plot.txt' using 1:4`). The columns are: