leonarski_fandClaude Fable 5
70c4d871b6
- Capitalize Rugnux as a proper noun throughout the prose; the command stays lowercase `rugnux` in code font. RUGNUX_OVERVIEW.md is retitled "What Rugnux does". - ACKNOWLEDGEMENT.md cites the raw-data repositories only: dataset counts and DOI prefixes moved out (EXTERNAL_TEST_DATA.md owns them), the ESRF data portal gains its citation (Dimper et al. 2019), and MXRDR remains name + link - it has no canonical citation paper. - RUGNUX_FORMATS.md: the CCD formats (marCCD, SMV) are supported as-is with very limited scope, and per-panel XFEL data is not read. - Fix wrong facts a reader would act on: nonexistent `make jfjoch` targets, invalid udev rules, PUSH sockets documented as PULL, swapped writer width/height, underload semantics, the transposed pixel-mask numpy example (the server checks width and height separately), the Durin/Neggia mask-bit table, FPGA threshold register addresses and the mailbox bit field, the I2C core's document number (PG090), an inverted MODEL_FIT_SIGMA formula, a self-inconsistent worked report example, and 11 cross-page anchors whose slugs carry MyST section numbers. - Unify CC1/2 spelling in prose; math notation and report keys unchanged. - Sweep grammar, typos and editing residue across the FPGA, deployment, streaming and analysis pages, including historical CHANGELOG typos. - rugnux_cli.cpp: the -S usage/error examples pair 96 with P43212; 92 names a different group. - Root THIRD_PARTY_NOTICES.md: scope the GPL-compatibility claim (CUDA EULA) and the vendored-table intro (traccc); the docs copy regenerates via update_version.sh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
114 lines
5.5 KiB
Markdown
114 lines
5.5 KiB
Markdown
# Integration with MX data processing software
|
|
|
|
Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
|
|
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
|
|
program specifically needs another.**
|
|
|
|
| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|
|
|---|---|---|---|
|
|
| Jungfraujoch XDS plugin | yes | yes | yes |
|
|
| Durin (Global Phasing) | yes | yes | yes |
|
|
| Durin (Diamond, original) | yes | known bugs | known bugs |
|
|
| Neggia | yes | **no** — no virtual-dataset support | not tested |
|
|
| DIALS / xia2 | **only one data file** | yes | yes |
|
|
| CrystFEL | yes | yes | yes |
|
|
|
|
`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
|
|
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
|
|
single data file — see the DIALS section below.
|
|
|
|
## XDS
|
|
|
|
XDS reads HDF5 through a plugin, named in `XDS.INP`:
|
|
|
|
```
|
|
LIB="/opt/xds/libjfjoch_xds_plugin.so.1.0.0"
|
|
```
|
|
|
|
**Use the Jungfraujoch plugin.** It is Linux-only, is downloadable from the Gitea release directory
|
|
(built on RHEL 8), and also ships inside the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages.
|
|
The three numbers are the plugin version and change over time.
|
|
|
|
The alternatives, in order of preference:
|
|
|
|
* **Durin, Global Phasing build** — [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).
|
|
Prefer it over the original from Diamond Light Source, which has known bugs with non-DECTRIS files
|
|
(virtual datasets and the single-file layout). It is the only third-party plugin that reads
|
|
**signed** Jungfraujoch images correctly.
|
|
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
|
|
support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
|
|
dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
|
|
reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
|
|
Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
|
|
ordinary PSI case — do not use Neggia for it.
|
|
|
|
### `OVERLOAD` must be set by hand
|
|
|
|
No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
|
|
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
|
|
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
|
|
it *exceeds* `OVERLOAD`.
|
|
|
|
### Signed images
|
|
|
|
`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
|
|
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
|
|
There is no header field that changes this: if XDS is the target, consider collecting unsigned.
|
|
|
|
### Which pixels are masked
|
|
|
|
The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels:
|
|
|
|
| mask bit | meaning | jfjoch plugin | Durin / Neggia | DIALS |
|
|
|---|---|---|---|---|
|
|
| 0 | module gap | yes | yes | yes |
|
|
| 1, 4 | error, noisy | yes | yes | yes |
|
|
| 8, 9 | user mask, beam stop | yes | **no** | yes |
|
|
| 30 | module edge | yes | **no** | yes |
|
|
| 31 | chip gap | **no** | **no** | yes |
|
|
|
|
DIALS masks a pixel whenever *any* `pixel_mask` bit is set; Durin and Neggia look only at
|
|
bits 0–4. On a JUNGFRAU with the default edge masking this is a difference of order 2%
|
|
of the detector. Bits 30 and 31 mark pixels that are larger than normal rather than bad, which is
|
|
why the Jungfraujoch plugin passes chip-gap pixels through — but be aware that a dataset processed
|
|
by XDS and by DIALS will not have used exactly the same pixels.
|
|
|
|
## DIALS
|
|
|
|
Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
|
|
crystallography.
|
|
|
|
* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
|
|
reports a correspondingly short image count, without an error; if a goniometer is present it then
|
|
fails on the frames past the first file. A legacy run that fits in one data file is read
|
|
correctly — set `images_per_file` to cover the whole run.
|
|
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
|
|
the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
|
|
signed data those land inside the trusted range.
|
|
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
|
|
`saturation_value`.
|
|
|
|
## pyFAI
|
|
|
|
`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
|
|
[Detector geometry](DETECTOR_GEOMETRY.md) and [Rugnux](RUGNUX.md).
|
|
|
|
* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
|
|
and places the beam centre wrongly along the slow axis.
|
|
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
|
|
marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
|
|
count. Pass the marker and the mask at integration time:
|
|
|
|
```python
|
|
ai = pyFAI.load("calibration.poni")
|
|
res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
|
|
```
|
|
|
|
with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
|
|
type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.
|
|
|
|
## CrystFEL
|
|
|
|
Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
|
|
exceeds the value — so set it from `saturation_value`.
|