leonarski_fandjungfrau
4dc2534dbf
**Files written by Jungfraujoch now import correctly in DIALS, XDS and pyFAI.** A tilted detector, a grid scan, a still recorded at a goniometer position, and saturated or unreadable pixels were each described in a way that a third-party program acted on wrongly. If you process Jungfraujoch data outside Jungfraujoch, prefer this release to any earlier one. * HDF5: the detector tilt (`rot1`/`rot2`/`rot3`) is exported correctly in the NXmx transformation chain; untilted geometries are unaffected. * HDF5: a still recorded at a goniometer position is no longer read back as a single image, and a grid scan records a stationary spindle so a program that requires a rotation axis can open it. * HDF5: the sample transformation chain is written in mounting order, with a Smargon head position told apart from the spindle, one entry per image, `module_offset` as a float unit vector, and `offset_units` on every offset. * HDF5: saturated, underloaded and unreadable pixels are described so a downstream program masks them - `saturation_value`, `underload_value`, `error_value` and `bit_depth_readout` are written correctly, and a data file missing next to a VDS master reads as the error marker rather than as zero counts. * HDF5: the rotation axis is read back under whatever name it carries, and `mirror_y` records whether the assembled image is mirrored in Y relative to the detector's raw readout. * A grid scan and a goniometer axis can both be set; they are no longer alternatives. * `images_per_file` is chosen from the acquisition when it is not given: a rotation sweep of at most 20000 images goes into a single data file, a grid scan splits on whole fast-axis rows, and stills and serial keep 1000. * The writer refuses a stream whose start message declares a different pixel format than its images carry, and a DECTRIS detector sending signed images is no longer declared unsigned. * The image stream can carry the sample transformation chain (`transformations`, in the END message); a producer that does not send it gets the same chain built by the writer. * rugnux: fixing the space group with `-S` no longer prevents the lattice from being found - a lattice indexed in a different setting is reindexed into that group's own setting, and a run whose crystal does not have that group's lattice stops and names the cell it indexed as, rather than reporting statistics that cannot describe it. * rugnux: the per-image resolution estimate now predicts the resolution the merged data reach rather than the highest-resolution spot found, and is reported as `SPOT_RESOLUTION_ESTIMATE`. * rugnux: two runs of the same command on the same images produce the same merged intensities; the azimuthal profile written alongside them is not yet reproducible in the same way. * rugnux: the offline lattice refinement is bounded by iterations rather than by a wall clock, so a loaded machine can no longer refine to a different lattice; a live acquisition keeps its real-time bound. * rugnux: the detector-frame modulation correction is fitted on a grid spanning the detector, so whether it is applied no longer depends on how far integration reached. * rugnux: the geometry pre-pass no longer writes `<prefix>_01.mtz`, `_01.cif`, `_01.hkl` and `_01_image.dat`; the refined second pass writes those files under `<prefix>`, and that is the result to use. * rugnux: `_process.h5` describes the pixel format of the images it links to, and is written on a thread of its own. * rugnux: the detector geometry is also logged in XDS's convention (`ORGX`/`ORGY`, detector axis vectors, rotation axis), so it can be compared with an XDS refinement. * rugnux: an image integrated in pyFAI through the `.poni` file written by `--mode calibration` comes out with the correct azimuth, and the file declares pyFAI's `orientation`, which needs pyFAI 2024.01 or newer. Radial integration is unchanged. * rugnux: a rotation run is substantially faster throughout - beam-stop detection, first-pass indexing, geometry refinement, integration, scaling and merging - and observations outside the scaling resolution range are dropped as they are ingested. The refined geometry, the space group chosen and the merged statistics are unchanged. * Faster spot finding and indexing, on the broker as well as in rugnux; the spots found and the lattices indexed are unchanged. * A run reserves substantially less GPU memory: nothing is allocated for buffers that are never read, and a worker builds only the engines it uses. * rugnux: with `-N` left at its default the per-image loop of `--mode mx` uses at most 16 workers per GPU, rather than one per hardware thread; an explicit `-N` is obeyed as given. * CUDA 12 builds now contain device code for Volta, so the RHEL 8 packages and the portable Linux `.tgz` run on a V100; the CUDA 13 artefacts (RHEL 9, Ubuntu, Windows) remain Turing and newer. * The build resolves a single Eigen for the whole project, and refuses to configure if Ceres picks up a different one; a build that mixed two Eigen versions was undefined behaviour and crashed at -O2. * Documentation: a security page, and the supported GPU generations and minimum NVIDIA driver version of every released artefact. **Breaking change to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.162, `frontend/src/client`): * `dataset_settings.images_per_file` is no longer `default: 1000` and no longer accepts `0`; it is optional, and its minimum is 1. A client sending `0` (previously "one file for the whole run") is now rejected - omit the field instead, which for a rotation sweep gives the same single file. * `file_writer_format` now defaults to `NXmxVDS`, matching the server's own default and the layout recommended for DIALS, XDS and CrystFEL. A generated client that fills in schema defaults and does not set the format explicitly will write VDS masters where it previously wrote legacy ones; set `NXmxLegacy` explicitly to keep them. --------- Co-authored-by: jungfrau <jungfrau@mx-aare-test.psi.ch> Reviewed-on: #72 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
5.5 KiB
Integration with MX data processing software
Jungfraujoch writes NXmx HDF5 in three layouts (see HDF5 / NeXus data format), and not
every downstream program reads all three. NXmxVDS is the default and the one to use unless a
program specifically needs another.
NXmxLegacy |
NXmxVDS (default) |
NXmxIntegrated |
|
|---|---|---|---|
| Jungfraujoch XDS plugin | yes | yes | yes |
| Durin (Global Phasing) | yes | yes | yes |
| Durin (Diamond, original) | yes | known bugs | known bugs |
| Neggia | yes | no — no virtual-dataset support | not tested |
| DIALS / xia2 | only one data file | yes | yes |
| CrystFEL | yes | yes | yes |
NXmxLegacy joins the data files to the master with external links, which is what DECTRIS's
filewriter-1 format did. Use it only for a program that needs it, and then keep the whole run in a
single data file — see the DIALS section below.
XDS
XDS reads HDF5 through a plugin, named in XDS.INP:
LIB="/opt/xds/libjfjoch_xds_plugin.so.1.0.0"
Use the Jungfraujoch plugin. It is Linux-only, is downloadable from the Gitea release directory
(built on RHEL 8), and also ships inside the jfjoch_viewer RPM/APT packages.
The three numbers are the plugin version and change over time.
The alternatives, in order of preference:
- Durin, Global Phasing build — github.com/CV-GPhL/durin. Prefer it over the original from Diamond Light Source, which has known bugs with non-DECTRIS files (virtual datasets and the single-file layout). It is the only third-party plugin that reads signed Jungfraujoch images correctly.
- Neggia — github.com/dectris/neggia. No virtual-dataset
support, so it cannot read the default layout. It also mis-reads signed 16-bit images: it
dispatches on the pixel size in bytes and always casts to an unsigned type, so a count of
-2reaches XDS as65534and the-32768error marker as32768. Signed 32-bit degrades safely. Since JUNGFRAU in photon-counting conversion writes signed images by default, this affects the ordinary PSI case — do not use Neggia for it.
OVERLOAD must be set by hand
No XDS plugin — ours, Durin or Neggia — reads saturation_value from the file. XDS therefore takes
its overload from OVERLOAD= in XDS.INP, and you must set it to the master file's
/entry/instrument/detector/saturation_value. XDS treats it inclusively: a pixel is overloaded when
it exceeds OVERLOAD.
Signed images
MINIMUM_VALID_PIXEL_VALUE= may not be negative in current XDS, so a genuinely negative photon
count — the reason signed output exists — cannot be declared valid. xia2 clamps the value to 0.
There is no header field that changes this: if XDS is the target, consider collecting unsigned.
Which pixels are masked
The plugins do not all act on the same mask bits, so XDS and DIALS do not mask the same pixels:
| mask bit | meaning | jfjoch plugin | Durin / Neggia | DIALS |
|---|---|---|---|---|
| 0 | module gap | yes | yes | yes |
| 1, 4, 8, 9 | error, noisy, user mask, beam stop | yes | no | yes |
| 30 | module edge | yes | no | yes |
| 31 | chip gap | no | no | yes |
DIALS masks a pixel whenever any pixel_mask bit is set; Durin and Neggia look only at the low
byte and at bits 1–4. On a JUNGFRAU with the default edge masking this is a difference of order 2%
of the detector. Bits 30 and 31 mark pixels that are larger than normal rather than bad, which is
why the Jungfraujoch plugin passes chip-gap pixels through — but be aware that a dataset processed
by XDS and by DIALS will not have used exactly the same pixels.
DIALS
Tested regularly against DIALS (currently 3.27.0), including the xia2.ssx pipeline for serial
crystallography.
- Use
NXmxVDSorNXmxIntegrated. WithNXmxLegacy, DIALS reads only the first data file and reports a correspondingly short image count, without an error; if a goniometer is present it then fails on the frames past the first file. A legacy run that fits in one data file is read correctly — setimages_per_fileto cover the whole run. - Unsigned 32-bit images require
bit_depth_readout, which Jungfraujoch writes. For signed images the field is deliberately omitted: DIALS remaps the top two codes of2^bit_depth_readout, and on signed data those land inside the trusted range. trusted_rangeis inclusive at both ends, and is taken fromunderload_valueandsaturation_value.
pyFAI
rugnux --mode calibration writes a .poni file describing the detector geometry — see
Detector geometry and rugnux.
-
It declares
orientation, so it needs pyFAI 2024.01 or newer. An older pyFAI ignores the key and places the beam centre wrongly along the slow axis. -
A
.ponifile carries geometry only. pyFAI does not learn the saturation value, the error marker or the pixel mask from it, and will happily integrate a masked pixel atUINTx_MAXas a count. Pass the marker and the mask at integration time:ai = pyFAI.load("calibration.poni") res = ai.integrate1d(image, 1000, dummy=65535, delta_dummy=0.5, mask=pixel_mask != 0)with
dummyset to the master file's/entry/instrument/detector/error_valuefor the stored pixel type, andpixel_maskread from/entry/instrument/detector/pixel_mask.
CrystFEL
Jungfraujoch files are compatible with CrystFEL. max_adu is inclusive — a pixel is bad when it
exceeds the value — so set it from saturation_value.