Files
Jungfraujoch/docs/SOFTWARE_INTEGRATION.md
leonarski_f 7f2ebd8209
Build Packages / Build documentation (push) Successful in 46s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m25s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m46s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 13m25s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 16m41s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m46s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m37s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 24m31s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m9s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m47s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 26m21s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 27m36s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 29m27s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m46s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 25m22s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 30m36s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 31m49s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 23m57s
Build Packages / Unit tests (push) Successful in 1h47m52s
Build Packages / Validation Battery (push) Successful in 44m55s
v1.0.0-rc.184 (#95)
* Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner.
* Rugnux: More reliable rotation processing and indexing, including small molecules and supercells.
* Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models.
* jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run.
* jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes.
* jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored.
* jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line.
* jfjoch_viewer: Build and display fixes; the README is rewritten.

Reviewed-on: #95
Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
2026-11-10 22:06:51 +01:00

128 lines
6.5 KiB
Markdown

# Integration with MX data processing software
Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
program specifically needs another.**
| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|---|---|---|---|
| Jungfraujoch XDS plugin | yes | yes | yes |
| Durin (PSI or Global Phasing build) | yes | yes | yes |
| Durin (Diamond, original) | yes | known bugs | known bugs |
| Neggia | yes | **no** — no virtual-dataset support | not tested |
| DIALS / xia2 | **only one data file** | yes | yes |
| CrystFEL | yes | yes | yes |
`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
single data file — see the DIALS section below.
## XDS
XDS reads HDF5 through a plugin, named in `XDS.INP`:
```
LIB=/opt/xds/durin-plugin.so
```
**We recommend the Durin plugin**, from either of two sources:
* the PSI build — [gitea.psi.ch/mx/durin/releases](https://gitea.psi.ch/mx/durin/releases);
* the Global Phasing fork by Clemens Vonrhein, distributed with autoPROC —
[github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).
Prefer either over the original Durin from Diamond Light Source, which has known bugs with
non-DECTRIS files (virtual datasets and the single-file layout). Durin is the only third-party plugin
that reads **signed** Jungfraujoch images correctly. Be aware that it does not act on every mask bit
— in particular not on the user mask, the beam-stop shadow or module edges (see
[Which pixels are masked](#which-pixels-are-masked)); where those matter, exclude the regions in
`XDS.INP` (`UNTRUSTED_RECTANGLE=` and friends).
The other options:
* **Jungfraujoch XDS plugin** — released for Linux (x86_64, built on RHEL 8) as
`jfjoch_xds_plugin-<version>-linux-x86_64.tgz` on the Gitea release page, and also shipped inside
the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages. Extract the archive into a directory of
its own; it holds the library (`lib/libjfjoch_xds_plugin.so.<version>`) and its license notices
(`share/doc/`). It acts on every mask bit Jungfraujoch writes.
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
ordinary PSI case — do not use Neggia for it.
### `OVERLOAD` must be set by hand
No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
it *exceeds* `OVERLOAD`.
### Signed images
`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
There is no header field that changes this: if XDS is the target, consider collecting unsigned.
### Which pixels are masked
The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels.
Measured with the Durin and Neggia builds Jungfraujoch is tested against:
| mask bit | meaning | jfjoch plugin | Durin | Neggia | DIALS |
|---|---|---|---|---|---|
| 0 | module gap | yes | yes | yes | yes |
| 1–4 | error, noisy | yes | yes | yes | yes |
| 8, 9, 10, 11 | user mask, beam stop, defective, inactive | yes | **no** | **no** | yes |
| 30 | module edge | yes | **no** | **no** | yes |
| 31 | chip edge | yes | yes | **no** | yes |
DIALS masks a pixel whenever *any* `pixel_mask` bit is set. Bits 30 and 31 mark pixels that are not
defective but do not measure like the rest: a module-edge pixel is larger and reads high, and a
chip-edge pixel is split from a larger one and at low counts reads low. On a JUNGFRAU with the default
edge masking they are of order 2% of the detector, so a dataset processed through Neggia or Durin
will not have used the same pixels as one processed through the Jungfraujoch plugin or DIALS.
A saturated pixel is passed to XDS with its value by all three plugins, so XDS flags it against
`OVERLOAD`; only the error marker becomes an untrusted (negative) pixel.
## DIALS
Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
crystallography.
* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
reports a correspondingly short image count, without an error; if a goniometer is present it then
fails on the frames past the first file. A legacy run that fits in one data file is read
correctly — set `images_per_file` to cover the whole run.
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
signed data those land inside the trusted range.
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
`saturation_value`.
## pyFAI
`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
[Detector geometry](DETECTOR_GEOMETRY.md) and [Rugnux](RUGNUX.md).
* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
and places the beam centre wrongly along the slow axis.
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
count. Pass the marker and the mask at integration time:
```python
ai = pyFAI.load("calibration.poni")
res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
```
with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.
## CrystFEL
Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
exceeds the value — so set it from `saturation_value`.