docs: rugnux getting-started pages - refinement, the full output list, a troubleshooting note
RUGNUX: a model run also refines the model it placed; the model can be given as an input; only the unmerged MTZ is written by default; the _process.h5 is not a default output; a short "when a run fails" note. TUTORIAL: the .ins, the refined model/maps/blobs, the log and _modules.h5 in the output list; the report is not written with -o ""; stills: -X Auto picks ffbidx with a cell, a run over several files. FORMATS: the XFEL stills formats and model/reference inputs told by content. CALIBRATION: -C and -S as the calibrant. INSTALL: the CPU path runs every indexer and the refinement. MODEL_AND_REFERENCE: the refined model is written as .pdb too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015CeEGgpKdfkCTJb3YNMJcu (cherry picked from commit 0062d72b95e996438ddc65dc343b526c84daa7fb)
This commit is contained in:
1 parent
8edd0e5f3e
commit
c3c5a32a0d
6 files changed
+55
-20
No files matched your search
+18
-6
@@ -5,7 +5,8 @@ data-processing half of the system (see [Naming](NAMING.md) for where the name c
|
||||
It takes a stored dataset, runs the full analysis pipeline — spot finding, indexing, geometry
|
||||
refinement, Bragg integration, scaling and merging — and writes the merged reflections
|
||||
(`.mtz`/`.cif`/`.hkl`), the unmerged MTZ and the results report; the per-image `_process.h5` is
|
||||
written only on request (`--write-process-h5`), or as the only output with `--no-merge`. A rotation run
|
||||
written only on request (`--write-process-h5`), or as the only output with `--no-merge`. Given an
|
||||
atomic model, it also validates the merge against it and refines the model it placed. A rotation run
|
||||
that finds no lattice still spot-finds every image and writes their per-image analysis there, then exits
|
||||
as failed.
|
||||
|
||||
@@ -60,8 +61,9 @@ rugnux -o myrun dataset_master.h5
|
||||
# resolves the indexing ambiguity, and hands over its R-free set
|
||||
rugnux -o myrun -z reference.mtz dataset_master.h5
|
||||
|
||||
# 3. with a known structure: R-work / R-free and sigma_A-weighted 2mFo-DFc / mFo-DFc maps
|
||||
# (a predicted model, or one the rigid body cannot place, goes to molecular replacement)
|
||||
# 3. with a known structure: R-work / R-free, sigma_A-weighted 2mFo-DFc / mFo-DFc maps and the
|
||||
# refined model (a predicted model, or one the rigid body cannot place, goes to molecular
|
||||
# replacement); `rugnux -o myrun dataset_master.h5 model.pdb` does the same
|
||||
rugnux -o myrun --model model.pdb dataset_master.h5
|
||||
|
||||
# 4. with the space group and the cell pinned (-S takes either spelling: P43212 or 96)
|
||||
@@ -89,8 +91,10 @@ myrun_detector.jpg the detector with the pixel mask and the beam-stop shadow d
|
||||
myrun_log.txt the full timestamped log of the run (debug level)
|
||||
```
|
||||
|
||||
The two MTZ extras are most of the bytes a run writes — worth knowing when sizing a scratch
|
||||
directory for a campaign, and both have off switches.
|
||||
The unmerged MTZ is most of the bytes a run writes — worth knowing when sizing a scratch
|
||||
directory for a campaign; `--no-export-unmerged` skips it. With a model, the maps, the placed model
|
||||
and the refined model are written beside these
|
||||
([Output files](RUGNUX_TUTORIAL.md#output-files) lists every file a run can write).
|
||||
|
||||
Read `myrun_report.txt` first: it says which space group was chosen and on what evidence, how far
|
||||
the data go, and anything that needs attention.
|
||||
@@ -183,6 +187,14 @@ A few things worth knowing before reaching for more flags:
|
||||
merging. [Installing Rugnux ▸ Memory](RUGNUX_INSTALL.md#memory) has the table and the two
|
||||
flags that lower it. A very large cell needs far more, and a CPU-only build most of all — tens of
|
||||
GB of host RAM ([Very large unit cells](RUGNUX_INSTALL.md#very-large-unit-cells)).
|
||||
- **When a run fails** it exits non-zero and says why on the terminal; `VERDICT=` and the `WARNING:`
|
||||
lines of the report say what needs attention on a run that finished
|
||||
([The results report](RUGNUX_REPORT.md#the-summary-and-what-the-run-decided)). A rotation run that indexes
|
||||
nothing most often has a wrong header: the beam centre is measured on every run, but the distance
|
||||
and the wavelength are indexed at the values stated — `--detector-distance` and `--wavelength` override them
|
||||
(`--beam-x`/`--beam-y` the centre); serial stills that index poorly want the cell (`-C`). A run that ends
|
||||
with `Killed` ran out of memory ([When host memory runs out](RUGNUX_INSTALL.md#when-host-memory-runs-out)).
|
||||
With any question about a run, send `myrun_report.txt` and `myrun_log.txt`.
|
||||
- Everything else is in [Running Rugnux](RUGNUX_TUTORIAL.md#running-rugnux) and the full
|
||||
[Command-line options](RUGNUX_ADVANCED.md#command-line-options).
|
||||
|
||||
@@ -223,7 +235,7 @@ One page per job, so the answer needed is near the top of a short page:
|
||||
| --- | --- | --- | --- |
|
||||
| [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) |
|
||||
| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` |
|
||||
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** |
|
||||
| **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`.mtz`/`.cif`/`.hkl` and the report; `_process.h5` with `--no-merge` or `--write-process-h5`** |
|
||||
|
||||
Use `rugnux` to re-analyse data after acquisition, to experiment with processing
|
||||
parameters, or to produce merged intensities for downstream structure solution.
|
||||
|
||||
@@ -41,6 +41,10 @@ else, and has two members:
|
||||
**`ice`** calibrates a real experiment against its own ice rings — no calibrant exposure needed —
|
||||
and is the reason a calibrant is a list of ring positions rather than a unit cell: hexagonal ice
|
||||
is P6<sub>3</sub>/mmc, so rings enumerated from its cell would include systematically absent ones.
|
||||
A standard that is not on the list is given by its unit cell instead: with `-C` the cell **is** the
|
||||
calibrant, its rings enumerated from the cell, with the absences of `-S` where that is given (taken as
|
||||
primitive where it is not). `-S` supplies symmetry absences only, so a standard whose extinctions come
|
||||
from its atomic basis — silicon's 222 — is better named with `--calibrant`.
|
||||
|
||||
`--calibration` picks how the rings are measured, and **both aggregate over every processed image** — `-s`/`-e`/`-t`
|
||||
select which images:
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
# What Rugnux reads
|
||||
|
||||
Which data `rugnux` opens, before anything is typed. The short answer: an HDF5 master (NXmx or
|
||||
DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d\*TREK) — nothing else
|
||||
is read, so any other format has to be converted to one of these first.
|
||||
DECTRIS, from any facility), a PILATUS miniCBF sweep, a marCCD sweep or an SMV sweep (ADSC or Rigaku d\*TREK), and
|
||||
for serial stills the files of the XFEL pipelines ([below](#serial-xfel-inputs)) — nothing else is
|
||||
read, so any other format has to be converted to one of these first. An atomic model and a reference
|
||||
reflection file may be given among the inputs too, told from the images by their contents.
|
||||
|
||||
**Input** is files, never a directory: an HDF5 master file, or one frame of a PILATUS miniCBF, marCCD
|
||||
or SMV sweep. For rotation data one input is **one sweep of one crystal** — Rugnux does not combine sweeps or crystals in a run; process each
|
||||
|
||||
@@ -78,8 +78,9 @@ The released Linux and Windows archives are CUDA builds (the macOS one is CPU-on
|
||||
an NVIDIA **driver** on the host — 525.60.13
|
||||
or newer for the CUDA 12 archive, 580.65.06 or newer for the CUDA 13 ones — and no CUDA toolkit,
|
||||
because everything CUDA is linked statically. With no GPU or no driver, `rugnux` reports zero CUDA
|
||||
devices and falls back to the CPU path, which works but is far slower and offers only the `fftw`
|
||||
indexer. A **V100 needs the CUDA 12 archive**; which generations each build covers is in
|
||||
devices and falls back to the CPU path, which works but is far slower: the FFT search runs on FFTW
|
||||
and `ffbidx` on its CPU implementation, and the refinement of a placed model takes minutes rather than
|
||||
seconds. A **V100 needs the CUDA 12 archive**; which generations each build covers is in
|
||||
[Release contents ▸ GPU generations and the NVIDIA driver](RELEASE_CONTENTS.md#gpu-generations-and-the-nvidia-driver).
|
||||
|
||||
## Building from source
|
||||
@@ -116,8 +117,8 @@ preprocessing, azimuthal integration, spot finding, prediction and Bragg integra
|
||||
pre-scan (beam-stop mask, background beam-centre fit, spot widths, defective pixels; one accumulator
|
||||
per card), rotation scaling and merging from the ingest of the reflections onwards (sorting and the
|
||||
resolution cut included) with its correction surfaces, and `--model`: the structure factors, scale
|
||||
fits and rigid-body placement of the validation and the rotation and translation functions of
|
||||
molecular replacement. The CPU implementations are the path where there is no GPU, and the two agree
|
||||
fits and rigid-body placement of the validation, the rotation and translation functions of
|
||||
molecular replacement, and the refinement of the placed model. The CPU implementations are the path where there is no GPU, and the two agree
|
||||
to rounding rather than bit for bit (parts of the scaling and the model fits round differently
|
||||
on the card), so a GPU and a CPU run can differ in the last digits of the merged data. The choice is automatic;
|
||||
a CUDA build runs on the CPU when told to (`rugnux --no-gpu ...`).
|
||||
|
||||
@@ -30,7 +30,7 @@ the rest walks through a run step by step.
|
||||
`<prefix>_mr.mtz`/`.cif`, with the model as placed there. `<prefix>.mtz` stays the data's own
|
||||
answer.
|
||||
- **A placed model is then refined**, after every decision has been taken, and written beside the
|
||||
answer as `<prefix>_refined_model.cif` with its maps ([Refinement](#refinement)); `--no-refine`
|
||||
answer as `<prefix>_refined_model.cif`/`.pdb` with its maps ([Refinement](#refinement)); `--no-refine`
|
||||
skips it. The refinement changes nothing else the run writes.
|
||||
- **A reference dataset (`-z`) and a space group or cell you give (`-S`, `-C`) are different:** they
|
||||
are taken as facts about the crystal. They fix the space group and the cell before processing,
|
||||
@@ -51,7 +51,7 @@ the rest walks through a run step by step.
|
||||
| Option | What you give | What Rugnux uses it for | Can it change `<prefix>.mtz`? |
|
||||
|---|---|---|---|
|
||||
| *(none)* | — | Everything from the data | — |
|
||||
| `--model <file>` | An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then [refinement](#refinement) of the placed model (`REFINE_` keys, `<prefix>_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json`), which decides nothing | Labels only (hand, indexing, axes), and only where the model fits. Other space groups → `<prefix>_mr.*` |
|
||||
| `--model <file>` | An atomic model, PDB or mmCIF, gzipped or not, recognised by content: an experimental structure or a prediction | Validation after the merge (R-work/R-free, CC(model, data), maps, the placed model); a rigid-body placement checked against a null, with molecular replacement where the rigid body does not place it, and straight away for a prediction; then [refinement](#refinement) of the placed model (`REFINE_` keys, `<prefix>_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json`), which decides nothing | Labels only (hand, indexing, axes), and only where the model fits. Other space groups → `<prefix>_mr.*` |
|
||||
| `--mr <file>` | The same kinds of model | As `--model`, but molecular replacement straight away, without the rigid body. `--mode mx` only, not together with `--model` | Hand label only. Other space groups → `<prefix>_mr.*` |
|
||||
| PAE file beside a prediction | `AF-<id>-F1-predicted_aligned_error_v<n>.json` next to `AF-<id>-F1-model_v<n>.cif` (read automatically, no option) | Cuts the prediction into rigid domains for molecular replacement | No (a domain alone can only confirm the answer's group) |
|
||||
| `-z, --reference <file>` (`--reference-mtz`) | Merged data of the same crystal form: an MTZ, or a PDB structure-factor mmCIF (`-sf.cif`) | Fixes the space group (in its own setting) and the cell (softly), puts the merge on its axes, resolves the indexing ambiguity, hands over its R-free set, reports CC<sub>ref</sub> on stills. Never a scale anchor | **Yes**: the space group, the axes, the indexing and the free set are the reference's |
|
||||
@@ -176,7 +176,7 @@ these letters.
|
||||
`<prefix>_maps.mtz` (map coefficients), `_model.cif`/`.pdb` — or the
|
||||
`<prefix>_mr_*` equivalents where the choice was `<prefix>_mr`. A rejected model gets its maps
|
||||
too: that is the case where looking at it in the density is most useful. A placed model is then
|
||||
[refined](#refinement) (`<prefix>_refined_model.cif`, `_refined_maps.mtz`, `_blobs.json`).
|
||||
[refined](#refinement) (`<prefix>_refined_model.cif`/`.pdb`, `_refined_maps.mtz`, `_blobs.json`).
|
||||
9. **The reflection files** — `<prefix>.mtz`, `.cif`, `.hkl`, `_unmerged.mtz` — are written last,
|
||||
carrying only the class-(b) labels from step 5 and the hand from step 7.
|
||||
|
||||
@@ -336,7 +336,7 @@ map coefficients such as FC, FWT, DELFWT are passed over) where the file has the
|
||||
I/SIGI, ...) made into amplitudes by French-Wilson, else the mean of the anomalous pairs. The free set is read from
|
||||
a FreeR_flag-like column (flag 0 is the test set; where 0 marks the majority, as phenix and CNS write it, the
|
||||
minority value is); a file without one gets rugnux's own 5 % set. The refinement is the one run after a merge on a placed model, and
|
||||
writes the same files - `<prefix>_refined_model.cif`, `<prefix>_refined_maps.mtz`, `<prefix>_blobs.json` - with
|
||||
writes the same files - `<prefix>_refined_model.cif`/`.pdb`, `<prefix>_refined_maps.mtz`, `<prefix>_blobs.json` - with
|
||||
the `REFINE_` keys in `<prefix>_report.txt`, beside `REFINE_FRENCH_WILSON=` and `REFINE_FREE_FLAGS= FILE|GENERATED`
|
||||
saying what had to be made of the input.
|
||||
|
||||
@@ -419,7 +419,7 @@ the data, a weak shell says nothing about them, and nothing acts on it.
|
||||
|---|---|---|
|
||||
| `<prefix>.mtz` / `.cif` / `.hkl` | The answer: the data's space group, with only the class-(b) labels from the model | Refinement, deposition, everything by default |
|
||||
| `<prefix>_model.cif` / `.pdb`, `<prefix>_maps.mtz` | The model as placed, in the frame of `<prefix>.mtz`, and its map coefficients (Coot/PyMOL open them as maps) | Starting refinement (REFMAC5, phenix.refine), PanDDA, Coot |
|
||||
| `<prefix>_refined_model.cif`, `<prefix>_refined_maps.mtz`, `<prefix>_blobs.json` | The placed model refined by rugnux, its map coefficients, and the unexplained blobs of its difference map | A first look at the refined density and at what the model leaves unexplained; continuing refinement |
|
||||
| `<prefix>_refined_model.cif`/`.pdb`, `<prefix>_refined_maps.mtz`, `<prefix>_blobs.json` | The placed model refined by rugnux, its map coefficients, and the unexplained blobs of its difference map | A first look at the refined density and at what the model leaves unexplained; continuing refinement |
|
||||
| `<prefix>_twin_<SG>.mtz` / `.cif` | The same data merged in a subgroup with a twin law — written from the data, model or not | Twin refinement, where you suspect the higher group absorbed a twin law |
|
||||
| `<prefix>_mr.mtz` / `.cif`, `<prefix>_mr_model.pdb` / `.cif`, `<prefix>_mr_*` maps | The merge in the space group the model chose (`MODEL_CHOICE_SPACE_GROUP=`), and the model placed there | Refinement *if* you accept the model's choice; compare R-free against the answer's |
|
||||
|
||||
|
||||
+19
-3
@@ -63,7 +63,8 @@ rugnux -o myrun --write-process-h5 dataset_master.h5
|
||||
# skip the unmerged MTZ (written by default), when only the merged data is wanted
|
||||
rugnux -o myrun --no-export-unmerged dataset_master.h5
|
||||
|
||||
# check the merged data against a known structure: R-work / R-free and maps
|
||||
# check the merged data against a known structure: R-work / R-free, maps and the
|
||||
# refined model (the model may also be given as a second input, without --model)
|
||||
rugnux -o myrun --model model.pdb dataset_master.h5
|
||||
```
|
||||
|
||||
@@ -182,7 +183,10 @@ together with `-C` and `-S`, puts each image into the reference's indexing inste
|
||||
also comes out in it. Without `-S` and without either, the mixed merge can look more symmetric than the
|
||||
crystal is, and the run warns.
|
||||
|
||||
`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills. The
|
||||
`ffbidx` requires a known cell (`-C`) and is the indexer of choice for sparse serial stills; the
|
||||
default `-X Auto` picks it for stills whenever a cell is given, so `-X ffbidx` above only makes that
|
||||
explicit. A run spread over several files is given as all of them at once - `rugnux *_master.h5`, or
|
||||
the files an XFEL pipeline writes ([Serial XFEL inputs](RUGNUX_FORMATS.md#serial-xfel-inputs)). The
|
||||
self-calibrating spot finder is on by default for both workflows (`--no-adaptive-spots` turns it off), and for
|
||||
serial stills leave `--min-pix-per-spot` **unset** so it is chosen per image — across the still-target battery this
|
||||
combination raises the indexing rate and typically extends resolution over a fixed threshold and
|
||||
@@ -270,6 +274,9 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke
|
||||
stills runs write the merged reflections (Bijvoet mates at `+hkl` and `-hkl`). Intensities are put
|
||||
on a common scale so the largest value fits the fixed-width field, and the file ends with the
|
||||
`0 0 0` terminator record.
|
||||
- `<prefix>.ins` — the SHELX instructions to go with the `.hkl`: `CELL` at the wavelength, `ZERR`
|
||||
with the cell's e.s.d.s, `LATT` and `SYMM` for the space group, and a placeholder `SFAC`/`UNIT` for the
|
||||
user to replace with the formula (see [Small-molecule structures with SHELXT and SHELXL](RUGNUX_INTEGRATION.md#small-molecule-structures-with-shelxt-and-shelxl)).
|
||||
|
||||
All three carry the **refined unit cell** (from rotation indexing) and the **space group determined
|
||||
from systematic absences** (constrained to the indexed lattice symmetry).
|
||||
@@ -289,6 +296,10 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke
|
||||
per-dataset input layout, and the pair dimple produces — and it is skipped, with a log line, for
|
||||
a cell the PDB format cannot hold. See
|
||||
[Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model).
|
||||
- `<prefix>_refined_model.cif` / `.pdb`, `<prefix>_refined_maps.mtz` and `<prefix>_blobs.json` —
|
||||
where the model was placed, the model as rugnux refined it, its map coefficients and the blobs of
|
||||
density its difference map leaves unexplained; `--no-refine` skips them. See
|
||||
[Refinement](RUGNUX_MODEL_AND_REFERENCE.md#refinement).
|
||||
- `<prefix>_twin_<SG>.mtz` / `.cif` — only where the chosen space group may have absorbed a twin law:
|
||||
the same data merged in a subgroup `<SG>`, with the twin law and its fraction, same resolution and
|
||||
free set as `<prefix>.mtz`, for refinement with that law. With `--model`, the maps come from
|
||||
@@ -300,8 +311,13 @@ Absorption is corrected only by the empirical surface a rotation run fits and ke
|
||||
the model chose, beside the run's own answer, which is left as it was. See
|
||||
[Validating against a model](RUGNUX_ADVANCED.md#validating-against-a-model-rugnux---model).
|
||||
- `<prefix>_report.txt` — the **results report**: what the run determined, in a form both a person and
|
||||
a beamline script can read. Always written, next to the files above. See
|
||||
a beamline script can read. Written by every `mx` and `scale` run, next to the files above (an empty prefix, `-o ""`, writes nothing). See
|
||||
[The results report](RUGNUX_REPORT.md#the-results-report).
|
||||
- `<prefix>_log.txt` — the full timestamped log of the run, at debug level, written by every run with
|
||||
an output prefix; send it with the report when asking about a run.
|
||||
- `<prefix>_modules.h5` — serial stills on a detector of several modules: where the geometry
|
||||
refinement placed each module, readable back with `--module-geometry` (see
|
||||
[Module geometry](RUGNUX_FORMATS.md#module-geometry)).
|
||||
- `<prefix>_plot.txt` — **the per-image table**, one row per processed image, whitespace-separated,
|
||||
with a single `#` legend line over its own columns so gnuplot plots it as it stands
|
||||
(`plot '<prefix>_plot.txt' using 1:4`). The columns are:
|
||||
|
||||
Reference in new issue
Block a user