mx/Jungfraujoch
3
2 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 92 Wiki Activity

README: reviewer corrections, plainer sentences, the .ins in the Files lines

Browse Source 
Short sentences throughout; the FPGA cards serve JUNGFRAU and the PSI
EIGER, DECTRIS detectors need none; requirements as one paragraph with
the tested OS and the install time; the command block before the format
list under "Your own data"; the build example without -march flags; the
paper-timing remark as one sentence here and in the Quick start. The
terminal summaries list the .ins beside the .hkl, as the program writes
them, in the README and in the tutorial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-10 19:40:46 +01:00
1 parent 7a60cb6a2f
commit 2ec7b959e8
3 files changed
+77 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+72 -70
View File
@@ -1,9 +1,9 @@
# Jungfraujoch
Jungfraujoch is open-source software from the Paul Scherrer Institute for X-ray diffraction data:
the data-acquisition and real-time analysis system for the PSI **JUNGFRAU** detector and, through
the DECTRIS detector control unit, for **EIGER** and other DECTRIS detectors, and **Rugnux**, its
program for processing diffraction data from the images to merged reflections and a refined
Jungfraujoch is open-source software from the Paul Scherrer Institute for X-ray diffraction data.
It is the data-acquisition and real-time analysis system for the PSI **JUNGFRAU** detector and,
through their detector control unit, for **DECTRIS** detectors, and it includes **Rugnux**, the
program that processes diffraction data from the images to merged reflections and a refined
structure.
Repository: **https://gitea.psi.ch/mx/jungfraujoch**. Documentation:
@@ -16,9 +16,9 @@ Repository: **https://gitea.psi.ch/mx/jungfraujoch**. Documentation:
writer, which writes NXmx HDF5. Linux servers only. See [Jungfraujoch](docs/JUNGFRAUJOCH.md) and
[Deployment](docs/DEPLOYMENT.md).
- **Rugnux** (`rugnux`) - offline processing of a stored dataset, rotation sweeps and serial
stills, from the images to scaled and merged reflections and, with an atomic model, to a refined
structure with maps. The unit cell, space group and resolution are decided from the data. Linux,
Windows and macOS. See [Rugnux](docs/RUGNUX.md).
stills. It takes the images to scaled and merged reflections and, with an atomic model, on to a
refined structure with maps. The unit cell, space group and resolution are decided from the data.
Linux, Windows and macOS. See [Rugnux](docs/RUGNUX.md).
- **Jungfraujoch Viewer** (`jfjoch_viewer`) - the Qt desktop application: shows the images with
their spots and predictions, runs the Rugnux pipeline from a settings panel and follows a live
collection. See [Using the viewer](docs/JFJOCH_VIEWER.md).
@@ -28,19 +28,17 @@ acquisition system needs a beamline and is covered by its own pages.
## Requirements
Rugnux runs on Linux x86-64 (glibc 2.28 or newer, a CPU with AVX2), Linux aarch64 (GH200, DGX
Spark), Windows 11 and macOS 13 or newer on Apple Silicon. An NVIDIA GPU is optional and
recommended: every stage has a CPU implementation, but the demo below takes seconds on a GPU and
minutes without one. The CUDA 12 archive needs an NVIDIA driver 525.60.13 or newer and no CUDA
toolkit. The demo needs about 10 GB of host memory and 9 GB of GPU memory (15 GB of host memory on
the CPU alone). Details in [Installing Rugnux](docs/RUGNUX_INSTALL.md).
Rugnux runs on Linux x86-64 and aarch64, Windows 11 and macOS 13 or newer on Apple Silicon. An
NVIDIA GPU is optional and recommended: the demo below takes seconds on a GPU and minutes without
one. The CUDA archive needs only the NVIDIA driver, no CUDA toolkit. The demo needs about 10 GB of
host memory and 9 GB of GPU memory. Details in [Installing Rugnux](docs/RUGNUX_INSTALL.md).
The acquisition system needs a server with AMD Alveo U55C FPGA cards for JUNGFRAU and for the PSI
EIGER (DECTRIS detectors are read through their detector control unit and need no FPGA), NVIDIA
GPUs for live indexing and a 100G switch ([Hardware requirements](docs/HARDWARE.md)).
The acquisition system needs a server with AMD Alveo U55C FPGA cards for JUNGFRAU and the PSI
EIGER, NVIDIA GPUs for live indexing and a 100G switch ([Hardware requirements](docs/HARDWARE.md)).
DECTRIS detectors are read through their detector control unit and need no FPGA.
The times in this file were measured with release 1.0.0-rc.183 on a desktop with a 16-core CPU and
an NVIDIA RTX 5080, the data on a local SSD.
The times in this file were measured with release 1.0.0-rc.183 on a Rocky Linux 8 desktop with a
16-core CPU and an NVIDIA RTX 5080, the data on a local SSD.
## Installation
@@ -56,17 +54,17 @@ export PATH=$PWD/rugnux-$VERSION/bin:$PATH
rugnux # prints the version and the usage
```
The archive is 268 MB and nothing else is installed. The same page has the CPU-only Linux archive,
the aarch64, Windows (unzip, run `bin\rugnux.exe`) and macOS archives, the viewer and the XDS
plugin. On RHEL / Rocky 8 and 9 and Ubuntu 22.04 / 24.04 Rugnux, the viewer and the server stack are
also packages (`dnf install rugnux`, `apt install rugnux`; [package repositories](docs/REPOSITORIES.md)).
The archive is 268 MB, installs in under a minute, and nothing else is installed. The same page has
the CPU-only, aarch64, Windows and macOS archives, the viewer and the XDS plugin. On RHEL / Rocky
and Ubuntu, Rugnux is also a package (`dnf install rugnux`, `apt install rugnux`;
[package repositories](docs/REPOSITORIES.md)).
## Demo on public data
The demo is the 9ZLO dataset ([PDB 9ZLO](https://www.rcsb.org/structure/9ZLO)): 3600 EIGER images
of *Proteus mirabilis* UreE collected at the Australian Synchrotron, 10.8 GB in 19 files on Zenodo
([10.5281/zenodo.18652652](https://doi.org/10.5281/zenodo.18652652), CC-BY 4.0; cite it if you use
the data). The download takes about 30 minutes at 50 Mbit/s.
([10.5281/zenodo.18652652](https://doi.org/10.5281/zenodo.18652652), CC-BY 4.0). Cite the record if
you use the data. The download takes about 30 minutes at 50 Mbit/s.
```
mkdir -p demo/9zlo && cd demo
@@ -91,13 +89,13 @@ The run prints numbered steps and takes **17 s** on the GPU (minutes with `--no-
Completeness 100.0 % to 1.66 A; 99.7 % over the full written range
Signal I/sigma 11.6 R_meas 12.0 % CC1/2 0.999 ISa 23.6 (23.1-24.0)
Anomalous no usable signal (CC_anom -0.16)
Files 9zlo.cif 9zlo.hkl 9zlo.mtz 9zlo_detector.jpg 9zlo_log.txt
9zlo_plot.txt 9zlo_report.txt 9zlo_unmerged.mtz
Files 9zlo.cif 9zlo.hkl 9zlo.ins 9zlo.mtz 9zlo_detector.jpg
9zlo_log.txt 9zlo_plot.txt 9zlo_report.txt 9zlo_unmerged.mtz
Wall time 17.30 s
```
Another run, GPU or CPU, gives the same space group, the cell to 0.1 % and the resolution to one
shell; the last digit of a statistic can differ. The deposited structure has the same space group,
shell. The last digit of a statistic can differ. The deposited structure has the same space group,
a cell within 0.5 % of this one and 2.0 A as its resolution.
### Output files
@@ -108,7 +106,9 @@ a cell within 0.5 % of this one and 2.0 A as its resolution.
- `9zlo_unmerged.mtz` - the scaled observations, for POINTLESS and AIMLESS.
- `9zlo_report.txt` - the statistics and decisions. Read this one first; every value is also a
`KEY= value` line (`grep '^R_MEAS= ' 9zlo_report.txt`).
- `9zlo_plot.txt` (one row per image), `9zlo_detector.jpg`, `9zlo_log.txt`.
- `9zlo_plot.txt` - one row per image, for plotting how the crystal behaved over the sweep.
- `9zlo_detector.jpg` - the detector with the pixel mask and the beam-stop shadow drawn on it.
- `9zlo_log.txt` - the full log of the run, to send with any question about it.
### With a model
@@ -123,33 +123,36 @@ rugnux -o 9zlo --model AF-P17090-F1-model_v6.cif 9zlo/ASP0887_09_0114_master.h5
The model does not change the processing. Molecular replacement places the two copies
(`MR_DECISION= AGREES` in the report), refinement brings them to R-work / R-free 0.27 / 0.30, and the
run takes **22 s**. It adds `9zlo_model.cif`/`.pdb` and `9zlo_maps.mtz` (the model as placed),
`9zlo_refined_model.cif`/`.pdb` and `9zlo_refined_maps.mtz` (as refined, with map coefficients that
Coot opens) and `9zlo_blobs.json` (difference density the model does not explain).
run takes **22 s**. It adds these files:
- `9zlo_model.cif`, `9zlo_model.pdb`, `9zlo_maps.mtz` - the model as placed, with its maps.
- `9zlo_refined_model.cif`, `9zlo_refined_model.pdb`, `9zlo_refined_maps.mtz` - the refined model,
with map coefficients that Coot opens.
- `9zlo_blobs.json` - difference density the model does not explain.
The same walk-through is in the documentation: [Quick start](docs/RUGNUX.md#quick-start) and
[A first run in detail](docs/RUGNUX_TUTORIAL.md#a-first-run-in-detail).
### Further examples
The paper's timing benchmark runs the three rotation sets below with 9ZLO; the serial set is one
run of a tape-drive experiment at SACLA. All are public. A rotation set is run as 9ZLO is, with the
master file of its archive (any one frame for 4NWV) and its model: the AlphaFold predictions for
9S02 and 9GJX are in [tools/battery/models/](tools/battery/models/), the 4NWV model is the
deposited structure (`https://files.rcsb.org/download/4NWV.cif`).
The paper's timing benchmark runs 9ZLO and the three rotation sets below. All are public. A
rotation set is run as 9ZLO is, with the master file of its archive (any one frame for 4NWV) and its
model. The AlphaFold predictions for 9S02 and 9GJX are in
[tools/battery/models/](tools/battery/models/); the 4NWV model is the deposited structure
(`https://files.rcsb.org/download/4NWV.cif`).
| Set | Source (cite this) | Data | Run time with a model |
| --- | --- | --- | --- |
| [9S02](https://www.rcsb.org/structure/9S02) | MX-RDR [10.60884/NRNGS4](https://doi.org/10.60884/NRNGS4) | EIGER 16M, NXmx HDF5, 11.7 GB | 45 s |
| [9GJX](https://www.rcsb.org/structure/9GJX) | IRRMC [10.18430/M39GJX](https://doi.org/10.18430/M39GJX) | EIGER 16M, HDF5, 20.2 GB archive | 43 s |
| [4NWV](https://www.rcsb.org/structure/4NWV) | SBGrid Data Bank [10.15785/sbgrid/9](https://doi.org/10.15785/sbgrid/9) | marCCD frames, 16.8 GB; a 400 A cell | 390 s |
| [9V3D](https://www.rcsb.org/structure/9V3D) | CXIDB [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | MPCCD, Cheetah-SACLA HDF5; files `run157084-0..2.h5`, 10.1 GB | 35 s |
| [9V3D](https://www.rcsb.org/structure/9V3D) | CXIDB [236](https://cxidb.org/id-236.html), [10.11577/3014306](https://doi.org/10.11577/3014306) | MPCCD, Cheetah-SACLA HDF5, 10.1 GB | 35 s |
### Serial stills: lysozyme at SACLA
Serial data go through the same program. A dataset without a goniometer axis is taken as stills;
give it the known cell and space group. The example is one run of a tape-drive experiment at SACLA
BL2 (CXIDB 236): three of its hit files. The files carry the detector's panel layout, so no geometry
Serial data go through the same program: a dataset without a goniometer axis is taken as stills.
The example is three hit files of one run of a tape-drive experiment at SACLA BL2 (CXIDB 236). It
gives the known cell and space group. The files carry the detector's panel layout, so no geometry
file is needed.
```
@@ -171,10 +174,11 @@ It takes about 35 s and ends with
Resolution 1.69 A written; the CC1/2 fall-off is fitted at 1.72 A
Completeness 100.0 % to 1.79 A; 98.4 % over the full written range
Signal I/sigma 12.8 CC1/2 0.978
Files sacla236.cif sacla236.hkl sacla236.mtz sacla236_blobs.json
sacla236_log.txt sacla236_maps.mtz sacla236_model.cif
sacla236_model.pdb sacla236_module_placement.json
sacla236_modules.h5 sacla236_plot.txt sacla236_refined_maps.mtz
Files sacla236.cif sacla236.hkl sacla236.ins sacla236.mtz
sacla236_blobs.json sacla236_log.txt sacla236_maps.mtz
sacla236_model.cif sacla236_model.pdb
sacla236_module_placement.json sacla236_modules.h5
sacla236_plot.txt sacla236_refined_maps.mtz
sacla236_refined_model.cif sacla236_refined_model.pdb
sacla236_report.txt sacla236_unmerged.mtz
Wall time 34.64 s
@@ -184,40 +188,38 @@ Both warnings are expected here. The deposited model was refined against these r
free set drawn by this run is not truly free. A few percent of the reflections hold a saturated
pixel. About 95 % of the 2424 frames are indexed and the refined model has an R-free of about 0.17.
For serial data `sacla236_report.txt` gives `R_SPLIT` and `CC_HALF_SPLIT` (one half of the frames
against the other) in place of R_meas. `sacla236_module_placement.json` records where each panel
was placed.
against the other) in place of R_meas. `sacla236_module_placement.json` holds the refined panel
positions.
## Your own data
Rugnux reads, with no conversion ([Input formats](docs/RUGNUX_FORMATS.md)): HDF5 masters written by
Jungfraujoch, by a DECTRIS detector or by another NXmx toolchain; PILATUS miniCBF, marCCD, SMV and
Bruker SFRM sequences; and for serial stills Cheetah-SACLA HDF5, Cheetah CXI, LCLS-I XTC and NXmx
stills files.
```
rugnux -o myrun dataset_master.h5 # rotation: cell, space group, resolution from the data
rugnux -o myrun -A dataset_master.h5 # anomalous statistics and error model
rugnux -o myrun --model model.cif dataset_master.h5 # with a model: placement, refinement, maps
rugnux -o serial -C "a,b,c,alpha,beta,gamma" -S P43212 run*.h5 # serial stills with the known cell
rugnux -o serial -C ... -S ... --crystfel-geometry det.geom *.cxi # XFEL files that do not place their own panels
rugnux -o myrun --no-gpu dataset_master.h5 # CPU only; -N limits the threads
```
Do not give `-C` or `-S` for rotation data. Every option is listed in
[Command-line options](docs/RUGNUX_CLI.md); [Running Rugnux](docs/RUGNUX_TUTORIAL.md) is the manual
Rugnux reads these without conversion ([Input formats](docs/RUGNUX_FORMATS.md)): NXmx HDF5 from
Jungfraujoch, DECTRIS detectors or another toolchain, PILATUS miniCBF, marCCD, SMV and Bruker SFRM
sequences, and the serial formats Cheetah-SACLA HDF5, Cheetah CXI and LCLS-I XTC.
Rotation data need neither `-C` nor `-S`. Every option is listed in
[Command-line options](docs/RUGNUX_CLI.md). [Running Rugnux](docs/RUGNUX_TUTORIAL.md) is the manual,
and [Rugnux with other programs](docs/RUGNUX_INTEGRATION.md) takes the files on to Phenix, REFMAC,
SHELX and the CCP4 tools. The same datasets open in `jfjoch_viewer`.
## Reproducing the paper's results
The results of the Rugnux paper come from the validation battery: the rotation datasets of
[External test data](docs/EXTERNAL_TEST_DATA.md) scored against their depositions, the serial
datasets of [External test data: serial](docs/EXTERNAL_TEST_DATA_STILLS.md) scored against their
deposited models, and the timing benchmark on the four rotation sets above. Every file the runs
wrote, the manifests and scripts as used, and the source code of the versions used are on Zenodo:
*Rugnux: Results of synchrotron and XFEL tests*,
[doi:10.5281/zenodo.23596195](https://doi.org/10.5281/zenodo.23596195) (CC-BY-SA 4.0). The images
come from the repositories listed on those two pages.
The results of the Rugnux paper come from the validation battery. The rotation datasets of
[External test data](docs/EXTERNAL_TEST_DATA.md) are scored against their depositions. The serial
datasets of [External test data: serial](docs/EXTERNAL_TEST_DATA_STILLS.md) are scored against
their deposited models. The timing benchmark runs the four rotation sets above. The results are on
Zenodo: *Rugnux: Results of synchrotron and XFEL tests*,
[doi:10.5281/zenodo.23596195](https://doi.org/10.5281/zenodo.23596195) (CC-BY-SA 4.0). The record
holds every file the runs wrote, the manifests and scripts as used, and the source code of the
versions used. The images come from the repositories listed on those two pages.
To rerun, copy the datasets under one data root with the paths of
[tools/battery/open.json](tools/battery/open.json) and
@@ -227,23 +229,23 @@ To rerun, copy the datasets under one data root with the paths of
```
tools/battery/battery.py run --rugnux <binary> # the battery (hours); --only <set> or --tier smoke for a subset
tools/battery/battery.py timing --rugnux <binary> # the timing benchmark (minutes)
tools/battery/battery.py compare <run> <run> # two runs side by side; the record's directory can be one of them
tools/battery/battery.py compare <run> <run> # two runs side by side; the record's untarred synchrotron_result.tar can be one
```
The paper's timings were taken with a release that placed the model but did not refine it; a
current run includes the refinement. The protocol and the scoring rules are in
The paper's times do not include the refinement; a current run's do. The protocol and the
scoring rules are in
[tools/battery/README.md](tools/battery/README.md) and [Measuring Rugnux](docs/RUGNUX_BENCHMARKING.md).
## Building from source
For developers and for platforms without a released binary. CMake 3.26 or newer, a C++20 compiler
(GCC 11 or newer, Clang 14 or newer, MSVC or Xcode), `git-lfs` for the clone, and network access on
the first configure, which downloads every third-party library. Optional: CUDA 12.8 or newer for
the GPU path, Qt 6 for the viewer.
Build from source for development or for a platform without a released binary. It needs CMake 3.26
or newer, a C++20 compiler (GCC 11 or newer, Clang 14 or newer, MSVC or Xcode), `git-lfs` for the
clone, and network access on the first configure, which downloads every third-party library.
Optional: CUDA 12.8 or newer for the GPU path, Qt 6 for the viewer.
```
git clone https://gitea.psi.ch/mx/jungfraujoch.git && cd jungfraujoch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=x86-64-v3" -DCMAKE_C_FLAGS="-march=x86-64-v3"
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target rugnux # build/rugnux/rugnux
cmake --build build -j$(nproc) --target jfjoch_broker # the acquisition service
```
docs/RUGNUX.md
+3 -3
View File
@@ -80,7 +80,7 @@ intensities and amplitudes), `9zlo.cif`, `9zlo.hkl` with `9zlo.ins`, `9zlo_unmer
with `--no-gpu`). `9zlo_report.txt` then says: space group P 21 21 21, cell 38.363 89.746 106.871
90 90 90, resolution 68.7-1.57 A, completeness 99.7 %, multiplicity 12.7, R_meas 0.120, CC1/2
0.9994, I/sigma 11.6, ISa 23.6. A second run, another GPU or the CPU repeats the space group, the
cell to 0.1 % and the resolution to one shell; the last digit of a statistic can differ.
cell to 0.1 % and the resolution to one shell. The last digit of a statistic can differ.
With a model, the run goes on to place and refine it. The AlphaFold prediction of UreE is in this
repository under `tools/battery/models/`, two files; the PAE file is read when it sits beside the
@@ -95,8 +95,8 @@ rugnux -o 9zlo --model AF-P17090-F1-model_v6.cif 9zlo/ASP0887_09_0114_master.h5
Molecular replacement places the two copies (`MR_DECISION= AGREES`), refinement brings them to
R-work / R-free 0.271 / 0.297 (the refined R factors can differ in the second decimal between
builds), and `9zlo_refined_model.cif` and `9zlo_refined_maps.mtz` are written beside the merged
data; the run takes 22 s (the paper's Extended Data Table 1 times this dataset with an earlier
release, which placed the model but did not refine it). Every file the run writes is listed under
data; the run takes 22 s. The paper's times do not include the refinement; a current run's do.
Every file the run writes is listed under
[Output files](#output-files) and in [A first run in detail](RUGNUX_TUTORIAL.md#a-first-run-in-detail);
the other example datasets are in the repository's top-level README.
docs/RUGNUX_TUTORIAL.md
+2 -2
View File
@@ -35,8 +35,8 @@ with a 16-core CPU and an NVIDIA RTX 5080, the files on a local SSD; the CPU alo
Completeness 100.0 % to 1.66 A; 99.7 % over the full written range
Signal I/sigma 11.6 R_meas 12.0 % CC1/2 0.999 ISa 23.6 (23.1-24.0)
Anomalous no usable signal (CC_anom -0.16)
Files 9zlo.cif 9zlo.hkl 9zlo.mtz 9zlo_detector.jpg 9zlo_log.txt
9zlo_plot.txt 9zlo_report.txt 9zlo_unmerged.mtz
Files 9zlo.cif 9zlo.hkl 9zlo.ins 9zlo.mtz 9zlo_detector.jpg
9zlo_log.txt 9zlo_plot.txt 9zlo_report.txt 9zlo_unmerged.mtz
Wall time 17.30 s
```