Jungfraujoch/docs/SOFTWARE_INTEGRATION.md

# Integration with MX data processing software

Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
program specifically needs another.**

| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|---|---|---|---|
| Jungfraujoch XDS plugin | yes | yes | yes |
| Durin (PSI or Global Phasing build) | yes | yes | yes |
| Durin (Diamond, original) | yes | known bugs | known bugs |
| Neggia | yes | **no** — no virtual-dataset support | not tested |
| DIALS / xia2 | **only one data file** | yes | yes |
| CrystFEL | yes | yes | yes |

`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
single data file — see the DIALS section below.

## XDS

XDS reads HDF5 through a plugin, named in `XDS.INP`:

```
LIB=/opt/xds/durin-plugin.so
```

**We recommend the Durin plugin**, from either of two sources:

* the PSI build — [gitea.psi.ch/mx/durin/releases](https://gitea.psi.ch/mx/durin/releases);
* the Global Phasing fork by Clemens Vonrhein, distributed with autoPROC —
  [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).

Prefer either over the original Durin from Diamond Light Source, which has known bugs with
non-DECTRIS files (virtual datasets and the single-file layout). Durin is the only third-party plugin
that reads **signed** Jungfraujoch images correctly. Be aware that it does not act on every mask bit
— in particular not on the user mask, the beam-stop shadow or module edges (see
[Which pixels are masked](#which-pixels-are-masked)); where those matter, exclude the regions in
`XDS.INP` (`UNTRUSTED_RECTANGLE=` and friends).

The other options:

* **Jungfraujoch XDS plugin** — released for Linux (x86_64, built on RHEL 8) as
  `jfjoch_xds_plugin-<version>-linux-x86_64.tgz` on the Gitea release page, and also shipped inside
  the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages. Extract the archive into a directory of
  its own; it holds the library (`lib/libjfjoch_xds_plugin.so.<version>`) and its license notices
  (`share/doc/`). It acts on every mask bit Jungfraujoch writes.
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
  support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
  dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
  reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
  Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
  ordinary PSI case — do not use Neggia for it.

### `OVERLOAD` must be set by hand

No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
it *exceeds* `OVERLOAD`.

### Signed images

`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
There is no header field that changes this: if XDS is the target, consider collecting unsigned.

### Which pixels are masked

The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels.
Measured with the Durin and Neggia builds used in Jungfraujoch CI:

| mask bit | meaning | jfjoch plugin | Durin | Neggia | DIALS |
|---|---|---|---|---|---|
| 0 | module gap | yes | yes | yes | yes |
| 1–4 | error, noisy | yes | yes | yes | yes |
| 8, 9, 10 | user mask, beam stop, defective | yes | **no** | **no** | yes |
| 30 | module edge | yes | **no** | **no** | yes |
| 31 | chip edge | yes | yes | **no** | yes |

DIALS masks a pixel whenever *any* `pixel_mask` bit is set. Bits 30 and 31 mark pixels that are not
defective but do not measure like the rest: a module-edge pixel is larger and reads high, and a
chip-edge pixel is split from a larger one and at low counts reads low. On a JUNGFRAU with the default
edge masking they are of order 2% of the detector, so a dataset processed through Neggia or Durin
will not have used the same pixels as one processed through the Jungfraujoch plugin or DIALS.

A saturated pixel is passed to XDS with its value by all three plugins, so XDS flags it against
`OVERLOAD`; only the error marker becomes an untrusted (negative) pixel.

## DIALS

Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
crystallography.

* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
  reports a correspondingly short image count, without an error; if a goniometer is present it then
  fails on the frames past the first file. A legacy run that fits in one data file is read
  correctly — set `images_per_file` to cover the whole run.
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
  the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
  signed data those land inside the trusted range.
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
  `saturation_value`.

## pyFAI

`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
[Detector geometry](DETECTOR_GEOMETRY.md) and [Rugnux](RUGNUX.md).

* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
  and places the beam centre wrongly along the slow axis.
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
  marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
  count. Pass the marker and the mask at integration time:

  ```python
  ai = pyFAI.load("calibration.poni")
  res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
  ```

  with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
  type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.

## CrystFEL

Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
exceeds the value — so set it from `saturation_value`.