Jungfraujoch/docs/SOFTWARE_INTEGRATION.md

# Integration with MX data processing software

Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
program specifically needs another.**

| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|---|---|---|---|
| Jungfraujoch XDS plugin | yes | yes | yes |
| Durin (Global Phasing) | yes | yes | yes |
| Durin (Diamond, original) | yes | known bugs | known bugs |
| Neggia | yes | **no** — no virtual-dataset support | not tested |
| DIALS / xia2 | **only one data file** | yes | yes |
| CrystFEL | yes | yes | yes |

`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
single data file — see the DIALS section below.

## XDS

XDS reads HDF5 through a plugin, named in `XDS.INP`:

```
LIB="/opt/xds/libjfjoch_xds_plugin.so.1.0.0"
```

**Use the Jungfraujoch plugin.** It is Linux-only, is downloadable from the Gitea release directory
(built on RHEL 8), and also ships inside the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages.
The three numbers are the plugin version and change over time.

The alternatives, in order of preference:

* **Durin, Global Phasing build** — [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).
  Prefer it over the original from Diamond Light Source, which has known bugs with non-DECTRIS files
  (virtual datasets and the single-file layout). It is the only third-party plugin that reads
  **signed** Jungfraujoch images correctly.
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
  support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
  dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
  reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
  Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
  ordinary PSI case — do not use Neggia for it.

### `OVERLOAD` must be set by hand

No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
it *exceeds* `OVERLOAD`.

### Signed images

`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
There is no header field that changes this: if XDS is the target, consider collecting unsigned.

### Which pixels are masked

The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels:

| mask bit | meaning | jfjoch plugin | Durin / Neggia | DIALS |
|---|---|---|---|---|
| 0 | module gap | yes | yes | yes |
| 1, 4, 8, 9 | error, noisy, user mask, beam stop | yes | **no** | yes |
| 30 | module edge | yes | **no** | yes |
| 31 | chip gap | **no** | **no** | yes |

DIALS masks a pixel whenever *any* `pixel_mask` bit is set; Durin and Neggia look only at the low
byte and at bits 1–4. On a JUNGFRAU with the default edge masking this is a difference of order 2%
of the detector. Bits 30 and 31 mark pixels that are larger than normal rather than bad, which is
why the Jungfraujoch plugin passes chip-gap pixels through — but be aware that a dataset processed
by XDS and by DIALS will not have used exactly the same pixels.

## DIALS

Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
crystallography.

* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
  reports a correspondingly short image count, without an error; if a goniometer is present it then
  fails on the frames past the first file. A legacy run that fits in one data file is read
  correctly — set `images_per_file` to cover the whole run.
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
  the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
  signed data those land inside the trusted range.
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
  `saturation_value`.

## pyFAI

`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
[Detector geometry](DETECTOR_GEOMETRY.md) and [rugnux](RUGNUX.md).

* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
  and places the beam centre wrongly along the slow axis.
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
  marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
  count. Pass the marker and the mask at integration time:

  ```python
  ai = pyFAI.load("calibration.poni")
  res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
  ```

  with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
  type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.

## CrystFEL

Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
exceeds the value — so set it from `saturation_value`.