Jungfraujoch/docs/SOFTWARE_INTEGRATION.md

Integration with MX data processing software

Jungfraujoch writes NXmx HDF5 in three layouts (see HDF5 / NeXus data format), and not every downstream program reads all three. NXmxVDS is the default and the one to use unless a program specifically needs another.

	NXmxLegacy	NXmxVDS (default)	NXmxIntegrated
Jungfraujoch XDS plugin	yes	yes	yes
Durin (PSI or Global Phasing build)	yes	yes	yes
Durin (Diamond, original)	yes	known bugs	known bugs
Neggia	yes	no — no virtual-dataset support	not tested
DIALS / xia2	only one data file	yes	yes
CrystFEL	yes	yes	yes

NXmxLegacy joins the data files to the master with external links, which is what DECTRIS's filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a single data file — see the DIALS section below.

XDS

XDS reads HDF5 through a plugin, named in XDS.INP:

LIB=/opt/xds/durin-plugin.so

We recommend the Durin plugin, from either of two sources:

• the PSI build — gitea.psi.ch/mx/durin/releases;
• the Global Phasing fork by Clemens Vonrhein, distributed with autoPROC — github.com/CV-GPhL/durin.

Prefer either over the original Durin from Diamond Light Source, which has known bugs with non-DECTRIS files (virtual datasets and the single-file layout). Durin is the only third-party plugin that reads signed Jungfraujoch images correctly. Be aware that it does not act on every mask bit — in particular not on the user mask, the beam-stop shadow or module edges (see Which pixels are masked); where those matter, exclude the regions in XDS.INP (UNTRUSTED_RECTANGLE= and friends).

The other options:

• Jungfraujoch XDS plugin — released for Linux (x86_64, built on RHEL 8) as jfjoch_xds_plugin-<version>-linux-x86_64.tgz on the Gitea release page, and also shipped inside the jfjoch_viewer RPM/APT packages. Extract the archive into a directory of its own; it holds the library (lib/libjfjoch_xds_plugin.so.<version>) and its license notices (share/doc/). It acts on every mask bit Jungfraujoch writes.
• Neggia — github.com/dectris/neggia. No virtual-dataset support, so it cannot read the default layout. It also mis-reads signed 16-bit images: it dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of -2 reaches XDS as 65534 and the -32768 error marker as 32768. Signed 32-bit degrades safely. Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the ordinary PSI case — do not use Neggia for it.

OVERLOAD must be set by hand

No XDS plugin — ours, Durin or Neggia — reads saturation_value from the file. XDS therefore takes its overload from OVERLOAD= in XDS.INP, and you must set it to the master file's /entry/instrument/detector/saturation_value. XDS treats it inclusively: a pixel is overloaded when it exceeds OVERLOAD.

Signed images

MINIMUM_VALID_PIXEL_VALUE= may not be negative in current XDS, so a genuinely negative photon count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0. There is no header field that changes this: if XDS is the target, consider collecting unsigned.

Which pixels are masked

The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels. Measured with the Durin and Neggia builds Jungfraujoch is tested against:

mask bit	meaning	jfjoch plugin	Durin	Neggia	DIALS
0	module gap	yes	yes	yes	yes
1–4	error, noisy	yes	yes	yes	yes
8, 9, 10, 11	user mask, beam stop, defective, inactive	yes	no	no	yes
30	module edge	yes	no	no	yes
31	chip edge	yes	yes	no	yes

DIALS masks a pixel whenever any pixel_mask bit is set. Bits 30 and 31 mark pixels that are not defective but do not measure like the rest: a module-edge pixel is larger and reads high, and a chip-edge pixel is split from a larger one and at low counts reads low. On a JUNGFRAU with the default edge masking they are of order 2% of the detector, so a dataset processed through Neggia or Durin will not have used the same pixels as one processed through the Jungfraujoch plugin or DIALS.

A saturated pixel is passed to XDS with its value by all three plugins, so XDS flags it against OVERLOAD; only the error marker becomes an untrusted (negative) pixel.

DIALS

Tested regularly against DIALS (currently 3.27.0), including the xia2.ssx pipeline for serial crystallography.

• Use NXmxVDS or NXmxIntegrated. With NXmxLegacy, DIALS reads only the first data file and reports a correspondingly short image count, without an error; if a goniometer is present it then fails on the frames past the first file. A legacy run that fits in one data file is read correctly — set images_per_file to cover the whole run.
• Unsigned 32-bit images require bit_depth_readout, which Jungfraujoch writes. For signed images the field is deliberately omitted: DIALS remaps the top two codes of 2^bit_depth_readout, and on signed data those land inside the trusted range.
• trusted_range is inclusive at both ends, and is taken from underload_value and saturation_value.

pyFAI

rugnux --mode calibration writes a .poni file describing the detector geometry — see Detector geometry and Rugnux.

• It declares orientation, so it needs pyFAI 2024.01 or newer. An older pyFAI ignores the key and places the beam centre wrongly along the slow axis.

• A .poni file carries geometry only. pyFAI does not learn the saturation value, the error marker or the pixel mask from it, and will happily integrate a masked pixel at UINTx_MAX as a count. Pass the marker and the mask at integration time:

  ai = pyFAI.load("calibration.poni")
  res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
  

  with dummy set to the master file's /entry/instrument/detector/error_value for the stored pixel type, and pixel_mask read from /entry/instrument/detector/pixel_mask.

CrystFEL

Jungfraujoch files are compatible with CrystFEL. max_adu is inclusive — a pixel is bad when it exceeds the value — so set it from saturation_value.