diff --git a/docs/HDF5.md b/docs/HDF5.md index 7e16bb8f..88a6e336 100644 --- a/docs/HDF5.md +++ b/docs/HDF5.md @@ -91,6 +91,14 @@ Images are stored chunked (one image per chunk) and compressed with bitshuffle + bitshuffle + Zstd; signed integer image datasets use `INTx_MIN` as the HDF5 fill value (the "masked / no-data" sentinel), unsigned use `UINTx_MAX`. +> **Signed images and the Neggia XDS plugin.** Neggia dispatches on the size of the pixel in bytes +> and always casts to an unsigned type, consulting the signedness of the data only for the pixel +> mask. A **signed 16-bit** image is therefore read wrongly: a count of `-2` reaches XDS as `65534`, +> and the `-32768` error marker as `32768`. Signed 32-bit happens to degrade safely, because every +> negative value ends up above `INT32_MAX` and is mapped to `-1`. Use the Jungfraujoch XDS plugin, +> or the Global Phasing build of Durin, for signed data — see +> [Integration with MX data processing software](SOFTWARE_INTEGRATION.md). + ### Reprocessing output: `_process.h5` The offline reprocessing tool [`rugnux`](TOOLS.md) (`rugnux/rugnux_cli.cpp`) re-runs the diff --git a/docs/SOFTWARE_INTEGRATION.md b/docs/SOFTWARE_INTEGRATION.md index c8db92b1..bd2bc92c 100644 --- a/docs/SOFTWARE_INTEGRATION.md +++ b/docs/SOFTWARE_INTEGRATION.md @@ -1,24 +1,112 @@ # Integration with MX data processing software +Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not +every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a +program specifically needs another.** + +| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` | +|---|---|---|---| +| Jungfraujoch XDS plugin | yes | yes | yes | +| Durin (Global Phasing) | yes | yes | yes | +| Durin (Diamond, original) | yes | known bugs | known bugs | +| Neggia | yes | **no** — no virtual-dataset support | not tested | +| DIALS / xia2 | **only one data file** | yes | yes | +| CrystFEL | yes | yes | yes | + +`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's +filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a +single data file — see the DIALS section below. + ## XDS -Jungfraujoch files are compatible with XDS, but there is a need of a dedicated plugin. -First we recommend to use Jungfraujoch own XDS plugin. It is available for Linux only and can -be downloaded from Gitea release directory (compiled on RHEL 8), it is also distributed in [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages. -To use the plugin, download the file `libjfjoch_xds_plugin.so.1.0.0` (three numbers at the end represent version of the plugin, and can differ later in time), -save it to common directory (e.g., `/opt/xds`) and add the following line in the XDS.INP file: + +XDS reads HDF5 through a plugin, named in `XDS.INP`: + ``` LIB="/opt/xds/libjfjoch_xds_plugin.so.1.0.0" ``` -We are also testing XDS with Durin and Neggia plugins, though they don't have full functionality: -* Neggia plugin doesn't support HDF5 virtual data sets. It can be downloaded from [github.com/dectris/neggia](https://github.com/dectris/neggia/). -* Durin has known bugs with handling non-DECTRIS files (so with virtual data sets or single format HDF5 file format). -We recommend Durin plugin prepared by the Global Phasing consortium: [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin), rather than original from the Diamond Light Source. +**Use the Jungfraujoch plugin.** It is Linux-only, is downloadable from the Gitea release directory +(built on RHEL 8), and also ships inside the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages. +The three numbers are the plugin version and change over time. + +The alternatives, in order of preference: + +* **Durin, Global Phasing build** — [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin). + Prefer it over the original from Diamond Light Source, which has known bugs with non-DECTRIS files + (virtual datasets and the single-file layout). It is the only third-party plugin that reads + **signed** Jungfraujoch images correctly. +* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset + support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it + dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2` + reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely. + Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the + ordinary PSI case — do not use Neggia for it. + +### `OVERLOAD` must be set by hand + +No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes +its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's +`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when +it *exceeds* `OVERLOAD`. + +### Signed images + +`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon +count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0. +There is no header field that changes this: if XDS is the target, consider collecting unsigned. + +### Which pixels are masked + +The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels: + +| mask bit | meaning | jfjoch plugin | Durin / Neggia | DIALS | +|---|---|---|---|---| +| 0 | module gap | yes | yes | yes | +| 1, 4, 8, 9 | error, noisy, user mask, beam stop | yes | **no** | yes | +| 30 | module edge | yes | **no** | yes | +| 31 | chip gap | **no** | **no** | yes | + +DIALS masks a pixel whenever *any* `pixel_mask` bit is set; Durin and Neggia look only at the low +byte and at bits 1–4. On a JUNGFRAU with the default edge masking this is a difference of order 2% +of the detector. Bits 30 and 31 mark pixels that are larger than normal rather than bad, which is +why the Jungfraujoch plugin passes chip-gap pixels through — but be aware that a dataset processed +by XDS and by DIALS will not have used exactly the same pixels. ## DIALS -Jungfraujoch files are tested regularly with DIALS (currently v. 3.27.0) `xia2.ssx` pipeline for serial crystallography. -There is one known limitation: files generated with NXmxLegacy format (mimicking DECTRIS filewriter1 format) -are not handled properly with DIALS. VDS based HDF5 format (NXmxVDS) is recommended, when using DIALS. + +Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial +crystallography. + +* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and + reports a correspondingly short image count, without an error; if a goniometer is present it then + fails on the frames past the first file. A legacy run that fits in one data file is read + correctly — set `images_per_file` to cover the whole run. +* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images + the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on + signed data those land inside the trusted range. +* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and + `saturation_value`. + +## pyFAI + +`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see +[Detector geometry](DETECTOR_GEOMETRY.md) and [rugnux](RUGNUX.md). + +* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key + and places the beam centre wrongly along the slow axis. +* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error + marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a + count. Pass the marker and the mask at integration time: + + ```python + ai = pyFAI.load("calibration.poni") + res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0) + ``` + + with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel + type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`. ## CrystFEL -Jungfraujoch files are compatible with CrystFEL. \ No newline at end of file + +Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it +exceeds the value — so set it from `saturation_value`. diff --git a/docs/index.rst b/docs/index.rst index 20b8256d..f0504f35 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -23,6 +23,7 @@ Jungfraujoch is distributed under the GPLv3 license. SOFTWARE VERSIONING DEPLOYMENT + SECURITY RELEASE_CONTENTS REPOSITORIES NAMING