leonarski_fandjungfrau
4dc2534dbf
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 18m57s
Build Packages / Unit tests (push) Skipped
Build Packages / build:windows:nocuda (push) Successful in 16m55s
Build Packages / build:windows:cuda (push) Successful in 18m48s
Build Packages / build:viewer-tgz:cpu (push) Successful in 13m10s
Build Packages / build:viewer-tgz:cuda (push) Successful in 14m45s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 22m23s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 20m12s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 23m7s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 20m43s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 23m9s
Build Packages / XDS test (durin plugin) (push) Successful in 12m26s
Build Packages / build:rpm (rocky9) (push) Successful in 24m58s
Build Packages / Generate python client (push) Successful in 50s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 23m20s
Build Packages / Create release (push) Skipped
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m37s
Build Packages / build:rpm (rocky8) (push) Successful in 27m58s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m38s
Build Packages / Build documentation (push) Successful in 59s
Build Packages / DIALS test (push) Successful in 23m16s
Build Packages / XDS test (neggia plugin) (push) Successful in 6m38s
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
113 lines
5.5 KiB
Markdown
113 lines
5.5 KiB
Markdown
# Integration with MX data processing software
|
||
|
||
Jungfraujoch writes NXmx HDF5 in three layouts (see [HDF5 / NeXus data format](HDF5.md)), and not
|
||
every downstream program reads all three. **`NXmxVDS` is the default and the one to use unless a
|
||
program specifically needs another.**
|
||
|
||
| | `NXmxLegacy` | `NXmxVDS` (default) | `NXmxIntegrated` |
|
||
|---|---|---|---|
|
||
| Jungfraujoch XDS plugin | yes | yes | yes |
|
||
| Durin (Global Phasing) | yes | yes | yes |
|
||
| Durin (Diamond, original) | yes | known bugs | known bugs |
|
||
| Neggia | yes | **no** — no virtual-dataset support | not tested |
|
||
| DIALS / xia2 | **only one data file** | yes | yes |
|
||
| CrystFEL | yes | yes | yes |
|
||
|
||
`NXmxLegacy` joins the data files to the master with external links, which is what DECTRIS's
|
||
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
|
||
single data file — see the DIALS section below.
|
||
|
||
## XDS
|
||
|
||
XDS reads HDF5 through a plugin, named in `XDS.INP`:
|
||
|
||
```
|
||
LIB="/opt/xds/libjfjoch_xds_plugin.so.1.0.0"
|
||
```
|
||
|
||
**Use the Jungfraujoch plugin.** It is Linux-only, is downloadable from the Gitea release directory
|
||
(built on RHEL 8), and also ships inside the [`jfjoch_viewer`](JFJOCH_VIEWER.md) RPM/APT packages.
|
||
The three numbers are the plugin version and change over time.
|
||
|
||
The alternatives, in order of preference:
|
||
|
||
* **Durin, Global Phasing build** — [github.com/CV-GPhL/durin](https://github.com/CV-GPhL/durin).
|
||
Prefer it over the original from Diamond Light Source, which has known bugs with non-DECTRIS files
|
||
(virtual datasets and the single-file layout). It is the only third-party plugin that reads
|
||
**signed** Jungfraujoch images correctly.
|
||
* **Neggia** — [github.com/dectris/neggia](https://github.com/dectris/neggia/). No virtual-dataset
|
||
support, so it cannot read the default layout. **It also mis-reads signed 16-bit images**: it
|
||
dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of `-2`
|
||
reaches XDS as `65534` and the `-32768` error marker as `32768`. Signed 32-bit degrades safely.
|
||
Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the
|
||
ordinary PSI case — do not use Neggia for it.
|
||
|
||
### `OVERLOAD` must be set by hand
|
||
|
||
No XDS plugin — ours, Durin or Neggia — reads `saturation_value` from the file. XDS therefore takes
|
||
its overload from `OVERLOAD=` in `XDS.INP`, and you must set it to the master file's
|
||
`/entry/instrument/detector/saturation_value`. XDS treats it inclusively: a pixel is overloaded when
|
||
it *exceeds* `OVERLOAD`.
|
||
|
||
### Signed images
|
||
|
||
`MINIMUM_VALID_PIXEL_VALUE=` may not be negative in current XDS, so a genuinely negative photon
|
||
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
|
||
There is no header field that changes this: if XDS is the target, consider collecting unsigned.
|
||
|
||
### Which pixels are masked
|
||
|
||
The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels:
|
||
|
||
| mask bit | meaning | jfjoch plugin | Durin / Neggia | DIALS |
|
||
|---|---|---|---|---|
|
||
| 0 | module gap | yes | yes | yes |
|
||
| 1, 4, 8, 9 | error, noisy, user mask, beam stop | yes | **no** | yes |
|
||
| 30 | module edge | yes | **no** | yes |
|
||
| 31 | chip gap | **no** | **no** | yes |
|
||
|
||
DIALS masks a pixel whenever *any* `pixel_mask` bit is set; Durin and Neggia look only at the low
|
||
byte and at bits 1–4. On a JUNGFRAU with the default edge masking this is a difference of order 2%
|
||
of the detector. Bits 30 and 31 mark pixels that are larger than normal rather than bad, which is
|
||
why the Jungfraujoch plugin passes chip-gap pixels through — but be aware that a dataset processed
|
||
by XDS and by DIALS will not have used exactly the same pixels.
|
||
|
||
## DIALS
|
||
|
||
Tested regularly against DIALS (currently 3.27.0), including the `xia2.ssx` pipeline for serial
|
||
crystallography.
|
||
|
||
* Use `NXmxVDS` or `NXmxIntegrated`. With `NXmxLegacy`, DIALS reads only the **first** data file and
|
||
reports a correspondingly short image count, without an error; if a goniometer is present it then
|
||
fails on the frames past the first file. A legacy run that fits in one data file is read
|
||
correctly — set `images_per_file` to cover the whole run.
|
||
* Unsigned 32-bit images require `bit_depth_readout`, which Jungfraujoch writes. For signed images
|
||
the field is deliberately omitted: DIALS remaps the top two codes of `2^bit_depth_readout`, and on
|
||
signed data those land inside the trusted range.
|
||
* `trusted_range` is inclusive at both ends, and is taken from `underload_value` and
|
||
`saturation_value`.
|
||
|
||
## pyFAI
|
||
|
||
`rugnux --mode calibration` writes a `.poni` file describing the detector geometry — see
|
||
[Detector geometry](DETECTOR_GEOMETRY.md) and [rugnux](RUGNUX.md).
|
||
|
||
* It declares `orientation`, so it needs **pyFAI 2024.01 or newer**. An older pyFAI ignores the key
|
||
and places the beam centre wrongly along the slow axis.
|
||
* A `.poni` file carries geometry only. pyFAI does **not** learn the saturation value, the error
|
||
marker or the pixel mask from it, and will happily integrate a masked pixel at `UINTx_MAX` as a
|
||
count. Pass the marker and the mask at integration time:
|
||
|
||
```python
|
||
ai = pyFAI.load("calibration.poni")
|
||
res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)
|
||
```
|
||
|
||
with `dummy` set to the master file's `/entry/instrument/detector/error_value` for the stored pixel
|
||
type, and `pixel_mask` read from `/entry/instrument/detector/pixel_mask`.
|
||
|
||
## CrystFEL
|
||
|
||
Jungfraujoch files are compatible with CrystFEL. `max_adu` is inclusive — a pixel is bad when it
|
||
exceeds the value — so set it from `saturation_value`.
|