Build Packages / Build documentation (push) Successful in 46s
Build Packages / Create release (push) Successful in 8s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m25s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 9m46s
Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 13m25s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 16m41s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m46s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 25m37s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 24m31s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m9s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 20m47s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 26m21s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 27m36s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 29m27s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 31m46s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 25m22s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 30m36s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 31m49s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 23m57s
Build Packages / Unit tests (push) Successful in 1h47m52s
Build Packages / Validation Battery (push) Successful in 44m55s
* Rugnux: Rotation processing recovers most of the speed lost since 1.0.0-rc.180; live processing in jfjoch_broker finishes sooner. * Rugnux: More reliable rotation processing and indexing, including small molecules and supercells. * Rugnux: More robust serial data and model refinement; molecular replacement handles multi-chain models. * jfjoch_broker (experimental): Live serial processing keeps the detector geometry fixed for the whole run. * jfjoch_broker: Pixel sum without an FPGA, a mask bit for inactive pixels, and several stability fixes. * jfjoch_broker: Rotation indexing and quick integration moved to rugnux live mode; their API settings are deprecated and ignored. * jfjoch_broker, Rugnux: Azimuthal integration can use logarithmic (log10) Q bins or a given number of bins, set from the API, the web interface and the Rugnux command line. * jfjoch_viewer: Build and display fixes; the README is rewritten. Reviewed-on: #95 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
128 lines
6.5 KiB
Markdown
128 lines
6.5 KiB
Markdown
# Integration with MX data processing software
|
|
|
|
Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
|
|
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
|
|
program specifically needs another.**
|
|
|
|
| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|
|
|---|---|---|---|
|
|
| Jungfraujoch XDS plugin | yes | yes | yes |
|
|
| Durin (PSI or Global Phasing build) | yes | yes | yes |
|
|
| Durin (Diamond, original) | yes | known bugs | known bugs |
|
|
| Neggia | yes | **no** — no virtual-dataset support | not tested |
|
|
| DIALS / xia2 | **only one data file** | yes | yes |
|
|
| CrystFEL | yes | yes | yes |
|
|
|
|
`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
|
|
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
|
|
single data file — see the DIALS section below.
|
|
|
|
## XDS
|
|
|
|
XDS reads HDF5 through a plugin, named in `XDS.INP`:
|
|
|
|
```
|
|
LIB=/opt/xds/durin-plugin.so
|
|
```
|
|
|
|
**We recommend the Durin plugin**, from either of two sources:
|
|
|
|
* the PSI build — [gitea.psi.ch/mx/durin/releases](https://gitea.psi.ch/mx/durin/releases);
|
|
* the Global Phasing fork by Clemens Vonrhein, distributed with autoPROC —
|
|
[github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).
|
|
|
|
Prefer either over the original Durin from Diamond Light Source, which has known bugs with
|
|
non-DECTRIS files (virtual datasets and the single-file layout). Durin is the only third-party plugin
|
|
that reads **signed** Jungfraujoch images correctly. Be aware that it does not act on every mask bit
|
|
— in particular not on the user mask, the beam-stop shadow or module edges (see
|
|
[Which pixels are masked](#which-pixels-are-masked)); where those matter, exclude the regions in
|
|
`XDS.INP` (`UNTRUSTED_RECTANGLE=` and friends).
|
|
|
|
The other options:
|
|
|
|
* **Jungfraujoch XDS plugin** — released for Linux (x86_64, built on RHEL 8) as
|
|
`jfjoch_xds_plugin-<version>-linux-x86_64.tgz` on the Gitea release page, and also shipped inside
|
|
the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages. Extract the archive into a directory of
|
|
its own; it holds the library (`lib/libjfjoch_xds_plugin.so.<version>`) and its license notices
|
|
(`share/doc/`). It acts on every mask bit Jungfraujoch writes.
|
|
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
|
|
support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
|
|
dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
|
|
reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
|
|
Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
|
|
ordinary PSI case — do not use Neggia for it.
|
|
|
|
### `OVERLOAD` must be set by hand
|
|
|
|
No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
|
|
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
|
|
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
|
|
it *exceeds* `OVERLOAD`.
|
|
|
|
### Signed images
|
|
|
|
`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
|
|
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
|
|
There is no header field that changes this: if XDS is the target, consider collecting unsigned.
|
|
|
|
### Which pixels are masked
|
|
|
|
The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels.
|
|
Measured with the Durin and Neggia builds Jungfraujoch is tested against:
|
|
|
|
| mask bit | meaning | jfjoch plugin | Durin | Neggia | DIALS |
|
|
|---|---|---|---|---|---|
|
|
| 0 | module gap | yes | yes | yes | yes |
|
|
| 1–4 | error, noisy | yes | yes | yes | yes |
|
|
| 8, 9, 10, 11 | user mask, beam stop, defective, inactive | yes | **no** | **no** | yes |
|
|
| 30 | module edge | yes | **no** | **no** | yes |
|
|
| 31 | chip edge | yes | yes | **no** | yes |
|
|
|
|
DIALS masks a pixel whenever *any* `pixel_mask` bit is set. Bits 30 and 31 mark pixels that are not
|
|
defective but do not measure like the rest: a module-edge pixel is larger and reads high, and a
|
|
chip-edge pixel is split from a larger one and at low counts reads low. On a JUNGFRAU with the default
|
|
edge masking they are of order 2% of the detector, so a dataset processed through Neggia or Durin
|
|
will not have used the same pixels as one processed through the Jungfraujoch plugin or DIALS.
|
|
|
|
A saturated pixel is passed to XDS with its value by all three plugins, so XDS flags it against
|
|
`OVERLOAD`; only the error marker becomes an untrusted (negative) pixel.
|
|
|
|
## DIALS
|
|
|
|
Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
|
|
crystallography.
|
|
|
|
* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
|
|
reports a correspondingly short image count, without an error; if a goniometer is present it then
|
|
fails on the frames past the first file. A legacy run that fits in one data file is read
|
|
correctly — set `images_per_file` to cover the whole run.
|
|
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
|
|
the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
|
|
signed data those land inside the trusted range.
|
|
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
|
|
`saturation_value`.
|
|
|
|
## pyFAI
|
|
|
|
`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
|
|
[Detector geometry](DETECTOR_GEOMETRY.md) and [Rugnux](RUGNUX.md).
|
|
|
|
* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
|
|
and places the beam centre wrongly along the slow axis.
|
|
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
|
|
marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
|
|
count. Pass the marker and the mask at integration time:
|
|
|
|
```python
|
|
ai = pyFAI.load("calibration.poni")
|
|
res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
|
|
```
|
|
|
|
with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
|
|
type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.
|
|
|
|
## CrystFEL
|
|
|
|
Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
|
|
exceeds the value — so set it from `saturation_value`.
|