From cfcb84aece31cad85b78956408040f7a031802ee Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Mon, 3 Aug 2026 20:53:37 +0200 Subject: [PATCH] docs: bring the tool pages back in line with the programs rugnux gained --model - R-free and 2Fo-Fc/Fo-Fc maps against an atomic model, and with it the resolution of the enantiomorph and of a merohedral indexing ambiguity - without the page ever mentioning it. It was the only option missing; the two lists now agree in both directions, checked against the usage the binary prints. The viewer page still said results are never saved and that no Windows package exists. Both have been false for a while: the Processing panel runs full rugnux jobs on the open dataset, writes _process.h5 and the merged reflections, registers each run as a selectable view so runs can be compared, and can hand out the equivalent command line for a cluster; and the installer is published with every release. The mask menu also loads TIFFs now, and the View menu has layout presets. The writer page documented -R for the root directory, which is the back-compatibility alias for -d, and an HTTP status interface that no longer exists - status reaches the broker over the writer notification socket, and a writer is stopped with a signal. The test page pointed at .gitlab-ci.yml and at jfjoch_offline_process, which is not a binary any more; the CrystFEL fixture pointed at HDF5DatasetWriteTest, which is not either. The broker page linked ../broker/redoc-static.html, which MyST resolved by copying the 700 kB file into _downloads/ rather than using the copy already in _static. Co-Authored-By: Claude Opus 5 (1M context) --- docs/JFJOCH_BROKER.md | 2 +- docs/JFJOCH_VIEWER.md | 68 ++++++++++++++++++++++------------------ docs/JFJOCH_WRITER.md | 33 ++++++++++++------- docs/RUGNUX.md | 19 ++++++++++- docs/TESTS.md | 33 +++++++++++++++---- docs/conf.py | 2 ++ tests/crystfel/README.md | 10 +++--- 7 files changed, 113 insertions(+), 54 deletions(-) diff --git a/docs/JFJOCH_BROKER.md b/docs/JFJOCH_BROKER.md index 76e3f2b7..833f8868 100644 --- a/docs/JFJOCH_BROKER.md +++ b/docs/JFJOCH_BROKER.md @@ -17,7 +17,7 @@ processing. See details [here](IMAGE_STREAM.md#image-stream). **Metadata stream** ZeroMQ PUB socket, contains metadata for all the images, with bundling. See details [here](IMAGE_STREAM.md#metadata-stream). **Configuration, status and results interface** HTTP/REST interface described in the OpenAPI format. -Description of the API is presented in the [OpenAPI description](../broker/redoc-static.html). +Description of the API is presented in the [OpenAPI specification](OPENAPI_SPECS.rst). ## Broker configuration `jfjoch_broker` requires JSON configuration files. The file is described by OpenAPI structure `jfjoch_settings` defined in `jfjoch_api.yaml` file. diff --git a/docs/JFJOCH_VIEWER.md b/docs/JFJOCH_VIEWER.md index c36b4464..8d881078 100644 --- a/docs/JFJOCH_VIEWER.md +++ b/docs/JFJOCH_VIEWER.md @@ -5,15 +5,17 @@ datasets, displays each image together with the analysis overlay (spots, predict integration, per-image statistics), and can follow a live data collection by syncing with a running [`jfjoch_broker`](JFJOCH_BROKER.md) over its HTTP interface. -It is a standalone Qt 6 application, distributed pre-built on the Gitea release page and in the -Jungfraujoch RPM/APT repositories (see [Deployment](DEPLOYMENT.md)). +It is a standalone Qt 6 application, distributed pre-built for **Linux and Windows** on the Gitea +release page and in the Jungfraujoch RPM/APT repositories — see [Release contents](RELEASE_CONTENTS.md) +for what each package contains and what it requires, and [Deployment](DEPLOYMENT.md) for how to +install it. ## Where it fits among the three analysis tools | Tool | Mode | Driven by | Output | | --- | --- | --- | --- | | [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) | -| **`jfjoch_viewer`** | **Interactive, on-screen exploration** | **Qt desktop application** | **Displayed on screen (results not saved to disk)** | +| **`jfjoch_viewer`** | **Interactive, on-screen exploration** | **Qt desktop application** | **On screen; a processing job can write the same files as `rugnux`** | | [`rugnux`](RUGNUX.md) | Offline batch processing of a stored dataset | Command-line interface | `_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging | ## Functionality @@ -22,18 +24,33 @@ Jungfraujoch RPM/APT repositories (see [Deployment](DEPLOYMENT.md)). `*_process.h5` files produced by [`rugnux`](RUGNUX.md). It also opens NXmx files written by DECTRIS detectors, though that path has had only limited testing. - Runs an **embedded data-processing pipeline** — the same analysis code as the rest of - Jungfraujoch — performing spot finding, indexing and integration on the displayed images. - Results are shown on screen but are **not** saved to disk. -- Auxiliary windows and panels: image list, image metadata, spot list, reflection list, - per-region-of-interest statistics, the azimuthal-integration profile, and dataset-info charts. -- User-mask editing: build a user mask interactively, clear it, save it as TIFF, or upload it to a - connected server. + Jungfraujoch — performing spot finding, indexing and integration on the displayed image, with the + result drawn over it. This interactive analysis is not written anywhere. +- Runs **full processing jobs** on the open dataset from the *Processing* panel, on the same + [`rugnux`](RUGNUX.md) engine and off the GUI thread: full analysis or azimuthal integration only, + over a chosen image range, optionally writing `_process.h5` and the merged `.mtz`/`.cif`. A + finished run becomes a selectable view of the dataset, so several processing runs can be compared + against each other, and its merging statistics open in their own window. The equivalent `rugnux` + command line can also be copied out to run the same job on a cluster instead. +- **Settings** panel for the geometry, unit cell, spot finding, indexing, azimuthal integration, + Bragg integration, scaling, powder calibration and a reference dataset — the same settings the + CLI takes. +- Auxiliary windows: image list, dataset metadata, spot list, reflection list, reciprocal-space + viewer, 2D azimuthal-integration image, calibration-image viewer and a magnifier; plus the + *Inspector* (per-image statistics, image features, resolution rings, ROI statistics), the + *Image strip* thumbnail feed and dataset-info charts. +- User-mask editing: build a user mask interactively, load one from TIFF (replacing or adding to the + current one), save it as TIFF, clear it, or upload it to a connected server. +- Layout presets (*View ▸ Image layout / Processing layout / Reset layout*) rearrange the docks for + looking at images or at processing results. ## Hardware As with the rest of Jungfraujoch, **serious performance requires an NVIDIA GPU**. On systems with a -GPU, use the CUDA build (provided as separate RPM/APT repositories) for the embedded indexing and -integration; the non-CUDA build runs the same pipeline on the CPU at much lower throughput. +GPU, use the CUDA build (a separate package variant everywhere: RPM/APT repository, `.tgz` and +Windows installer) for the embedded indexing and integration; the non-CUDA build runs the same +pipeline on the CPU at much lower throughput. The CUDA build also runs on a machine without a GPU — +see [Release contents ▸ CUDA and non-CUDA builds](RELEASE_CONTENTS.md#cuda-and-non-cuda-builds). ## Opening data @@ -60,12 +77,12 @@ can drive it: `jfjoch_viewer` is the one Jungfraujoch component that is cross-platform: it builds on Windows 11 with MSVC and the full CUDA GPU path. (The rest of Jungfraujoch — broker, receiver, FPGA host — is -Linux-only.) There is no pre-built Windows package yet, so build it from source. On Windows the -build is automatically restricted to the viewer and the libraries it needs (`JFJOCH_VIEWER_ONLY` is -forced on), and the remaining dependencies are fetched and built automatically (the first configure -needs network access). +Linux-only.) A pre-built installer is published with every release, so building from source is only +needed to develop or to change the build options. On Windows the build is automatically restricted +to the viewer and the libraries it needs (`JFJOCH_VIEWER_ONLY` is forced on), and the remaining +dependencies are fetched and built automatically (the first configure needs network access). -Verified toolchain: +Verified toolchain — the same one the released installer is built with: - Windows 11 - Visual Studio 2026 with the C++ (MSVC) toolset — required; CUDA on Windows builds through MSVC @@ -111,23 +128,14 @@ Notes: pipeline on the CPU (FFTW indexer) at lower throughput. To produce a self-contained installer (bundles the Qt runtime via `windeployqt`, the analysis CLIs, -and — on the CUDA build — the cuFFT runtime DLL, so the target host needs no Qt and no CUDA toolkit, -only an NVIDIA GPU driver), with NSIS installed: +and — on the CUDA build — the cuFFT runtime DLL, so the target host needs neither Qt nor a CUDA +toolkit), with NSIS installed: ``` cd build-win cpack ``` -The NSIS generator is selected automatically on Windows (no `-G` needed). The installer filename and -the Add/Remove Programs entry mark the CUDA/CPU variant, while the install folder and Start Menu -group stay plain `Jungfraujoch` (the two variants share an install location and replace each other — -CUDA is a strict superset): - -| Build | Installer file | Add/Remove Programs | -| --- | --- | --- | -| CUDA (default) | `jfjoch--win64-cuda.exe` | `Jungfraujoch (CUDA)` | -| `-DJFJOCH_USE_CUDA=OFF` | `jfjoch--win64-cpu.exe` | `Jungfraujoch (CPU)` | - -`` is the CUDA toolkit major version (e.g. `cuda13`). The cuFFT DLL is ~256 MB, so the CUDA -installer is correspondingly larger — hence the variant tag in the filename. +The NSIS generator is selected automatically on Windows (no `-G` needed). What comes out, and how +the CUDA and CPU variants are named and told apart, is described in +[Release contents ▸ Windows installer](RELEASE_CONTENTS.md#windows-installer). diff --git a/docs/JFJOCH_WRITER.md b/docs/JFJOCH_WRITER.md index 47b81faa..30452e56 100644 --- a/docs/JFJOCH_WRITER.md +++ b/docs/JFJOCH_WRITER.md @@ -17,22 +17,33 @@ Writer needs to be started as a background service, with the following command: jfjoch_writer {options}
Options: --R | --root_dir= Root directory for file writing --H | --http_port= HTTP port for statistics --r | --zmq_repub_port= ZeroMQ port for PUSH socket to republish images --f | --zmq_file_port= ZeroMQ port for PUB socket for notifications on finalized files --w | --rcv_watermark= Receiving ZeroMQ socket watermark (default = 100) --W | --repub_watermark= Republish ZeroMQ socket watermark (default = 1000) +-T | --tcp Use raw TCP/IP instead of ZeroMQ +-j | --nproc= Number of forks (only with -T) +-d | --root_dir= Root directory for file writing (-R is a deprecated alias) +-r | --zmq_repub_port= ZeroMQ port for PUSH socket to republish images +-f | --zmq_file_port= ZeroMQ port for PUB socket for notifications on finalized files +-w | --rcv_watermark= Receiving ZeroMQ socket watermark (default = 100) +-W | --repub_watermark= Republish ZeroMQ socket watermark (default = 1000) +-v | --verbose Verbose output +-h This message ``` for example: ``` -jfjoch_writer -H5234 tcp://dcu-address:5400 +jfjoch_writer -d /data tcp://dcu-address:5400 ``` -## HTTP interface -Writer has dedicated status interface via HTTP. It allows for two operations: -* ***check state of the writer*** to check if the writer is properly synchronized with DCU (e.g., that `file_prefix` agrees with what was set on the DCU) and monitor progress. -* ***cancel writing*** this will close all the HDF5 files being written and restart writer - the option should be used only if DCU process was terminated or disconnected, it SHOULD NOT be used as standard cancellation procedure (when DCU received cancel command it should properly finish writing as well) +## Status and cancellation +When a data collection is finalized, each writer reports its outcome back to `jfjoch_broker` over +the writer notification socket — a ZeroMQ address the broker passes in the START message +(`writer_notification_socket` in the broker configuration) — as a JSON message with the socket +number, run name and number, processed image count, throughput, and on failure an error string. +That is how the broker learns that a writer could not write. On the TCP/IP image stream, failures +additionally come back in-band as negative acknowledgements +(see [Data streams](IMAGE_STREAM.md#tcpip-image-stream)). + +To stop a writer, send it `SIGINT`, `SIGQUIT`, `SIGTERM` or `SIGHUP`: it closes the HDF5 files it is +writing and exits. This is only for the case where the broker was terminated or disconnected — it is +not the normal way to end a data collection, which the broker finishes on its own. ## Republish Republish creates a PULL socket on the writer, where all the messages are republished for further use by data analysis pipeline. diff --git a/docs/RUGNUX.md b/docs/RUGNUX.md index 11307256..9e66da96 100644 --- a/docs/RUGNUX.md +++ b/docs/RUGNUX.md @@ -19,7 +19,7 @@ command line over a file rather than a live detector stream. | Tool | Mode | Driven by | Output | | --- | --- | --- | --- | | [`jfjoch_broker`](JFJOCH_BROKER.md) | Online, real-time streaming analysis on FPGA + GPU | HTTP/REST + ZeroMQ | Live results and statistics, images streamed to [`jfjoch_writer`](JFJOCH_WRITER.md) | -| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | Displayed on screen (results not saved to disk) | +| [`jfjoch_viewer`](JFJOCH_VIEWER.md) | Interactive, on-screen exploration | Qt desktop application | On screen; a processing job can write the same files as `rugnux` | | **`rugnux`** | **Offline batch processing of a stored dataset** | **Command-line interface** | **`_process.h5`, and `.mtz`/`.cif`/`.hkl` when merging** | Use `rugnux` to re-analyse data after acquisition, to experiment with processing @@ -65,6 +65,22 @@ console. By default the written resolution is trimmed automatically where CC1/2 (`--resolution-cutoff cc-logistic`, CC1/2 target 0.30); set `--scaling-high-resolution` to fix the limit by hand, or `--resolution-cutoff off` to keep the full range. +## Validating against a model (`rugnux --model`) + +Given a PDB atomic model of the same structure, `--model model.pdb` scales the model structure +factors to the merged amplitudes — fitting a flat bulk-solvent contribution and an overall +anisotropic *B* — and reports **R-work / R-free** and the mean 2Fo-Fc density at the atom centres. +It also writes `_2fofc.ccp4`, `_fofc.ccp4` and `_maps.mtz` next to the +merged reflections. Nothing about the model is refined; it is only re-fractionalized into the data +cell, so a deposited model with a slightly different cell still lines up. + +It is a *data-quality lens*, independent of the internal statistics: R-free measures the merged +intensities against external truth, where CC1/2 and Rmeas only measure them against +themselves. It also settles the two things merged intensities alone cannot: the enantiomorph (data +merged in P41212 against a P43212 model are reindexed +into the model's hand), and — when no reference MTZ has already fixed it — a merohedral indexing +ambiguity, by keeping the candidate reindexing with the lowest R-free. + ## Re-scaling and re-merging (`rugnux --scale`) The `--scale` mode re-scales and merges the *already-integrated* reflections stored in a @@ -259,6 +275,7 @@ Scaling and merging: | `--scaling-iterations ` | Scaling iterations with no reference data (default: 3) | | `-z, --reference-mtz ` | Reference MTZ (enables reference-driven scaling) | | `--reference-column