leonarski_f
538f3504d3
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * **rugnux: significantly better quality of results, and faster.** A large rework of integration, scaling, merging, geometry refinement and space-group determination, together with measurements the program previously made no attempt at - the direct beam before indexing, the beam stop, the goniometer rotation scale, and the stretches of a sweep the crystal did not deliver. A rotation dataset typically gains observations at better <I/sigma> and R_meas, and every `mx` and `scale` run writes a `<prefix>_report.txt` results report modelled on XDS's `CORRECT.LP`. Many defaults moved with it: spot detection is self-calibrating, beam-stop detection and rotation geometry post-refinement are on, resolution limits default to as far as the detector reaches, and ice-ring handling engages only where the crystal is measured to have ice. * **jfjoch_viewer:** the beam-stop shadow, the detector calibration and the beam-centre measurement are reachable from "Analyze dataset"; the settings panel reports how the sample moved and how polarized the beam was; image rendering and interaction are faster. * **Performance:** bitshuffle+LZ4 images are decoded on the GPU rather than on the host, with the bitshuffle inverse fused into preprocessing so the decompressed frame is never held in device memory. * **Broker, writer, packaging and build:** image-slot lifetime and locking fixes, per-image datasets sized by the images actually written, the Debian/Ubuntu broker package renamed to `jfjoch`, and `image_analysis` compiling under MSVC again. **Breaking change to the rugnux command line:** * `--azint-only` and `--scale` are **removed**, replaced by `--mode azint` and `--mode scale`; the full pipeline is `--mode mx` and remains the default. A script passing the old flags now fails with the list of valid modes rather than silently running the wrong one. * `-t`/`--stride` is **refused on rotation data**: skipping frames cuts every reflection's rocking curve, so the combined fulls and their partiality would be measured over frames the sweep never recorded. Select a contiguous range with `-s`/`-e` instead. `--mode azint` and `--force-still` still take a stride. **Breaking changes to OpenAPI** - regenerate the client (`jfjoch-client` 1.0.0-rc.161, `frontend/src/client`) or read the affected fields as optional: * `image_scale_b` is removed from the `plot_type` enum, so a client requesting that plot now gets an error rather than a curve. * `azim_int_settings.high_q_recipA`, `spot_finding_settings.high_resolution_limit` and `spot_finding_settings.low_resolution_limit` are no longer `required`. All three mean "no limit at that end" when unset and are omitted from the response instead of carrying a placeholder value, which raises in a client generated from an rc.160-or-earlier spec. A value of 0 is still accepted and means the same thing. **Breaking changes to the stored formats** - a consumer reading these fields must treat them as optional: * The per-image image-scale B factor is no longer computed, so `/entry/MX/imageScaleBFactor` is absent from newly written HDF5 files and the corresponding key is absent from the CBOR DataMessage and END blocks. Files written by rc.160 and earlier still contain it and still open; nothing in the pipeline reads it any more. * `_reflns.jfjoch_diffrn_ISa` now carries the whole-range `1/sqrt(a*b)` that XDS's ISa denotes, and the error-model `a` and `b` are reported in XDS's convention; the strong-reflection asymptote moves to `_reflns.jfjoch_diffrn_ISa_asymptotic`. **A file written by an earlier version carries the asymptote under the plain `ISa` name.** Reviewed-on: #71 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
5.6 KiB
FPGA data analysis
Jungfraujoch FPGA design has incorporated X-ray diffraction image analysis capabilities.
Pixel mask
Pixels can be masked. For each module a 32-bit map of pixels is loaded to FPGA, with non-zero value meaning masked pixels. According to this map, pixels will be assigned a special value (minimum number for signed types and maximum number for non-signed types) and will be excluded from a subsequent analysis.
ADU histogram
Before conversion to photons/energy, an ADU histogram can be calculated for a module. This allows to preserve some signature of unconverted values. This is done on a module-basis and works with bins with 32 ADU width.
For EIGER this can be used as just a histogram procedure.
JUNGFRAU conversion
For JUNGFRAU module images are converted from ADUs to energy value and divided by a given number to keV units. Result of the operation is rounded to integers.
Pixel thresholding
Pixel range can be specified. Pixels below a minimum threshold will be assigned zero. Pixels above a maximum threshold will be assigned saturated pixel value (the largest number for a given bit-width and sign type). This is specifically designed to operate on unsummed frames, so frame-specific parameters (overload/noise) can be handled.
Frame summation
Frames can be summed together (on a per-module basis) in Jungfraujoch, with a limit of 256 frames added together.
Azimuthal integration
To implement azimuthal integration, FPGA is able to sum pixels based on a provided integration map and per-pixel corrections. This way Jungfraujoch implements azimuthal integration with solid angle and polarization corrections. Corrections were implemented according to formulas developed by Jensen et al. (J. Synchr. Rad., 29, 1420-1428, 2022).
Given FPGA limitations, split-pixels cannot be implemented and number of bins is limited as 2048 per detector module. This way 2D azimuthal integration, as needed for example by SAS-TT, cannot be currently implemented with the FPGA card and needs to be done on a CPU. One needs to be careful with per-pixel corrections - their acceptable range is constrained by 16-bit pixed point integer implementation and is tuned for standard SAXS/WAXS range.
As with ROIs, azimuthal integration is also available on CPU through the shared analysis library, so it applies to both the FPGA-accelerated (JUNGFRAU/PSI) and the DECTRIS-driven (EIGER) workflows.
Spot finding
Jungfraujoch FPGA implements a built-in spot finder. Spot finder allows to apply the following criteria for finding strong pixels:
- Resolution criterion - pixels only within a provided resolution range can be considered as strong pixels (calculating resolution map needs to happen on CPU before data collection run).
- Bad pixels - pixels marked as bad, as well as chip edges and module edges are excluded from spot finding,
- Overloads - pixels marked as overloads on JUNGFRAU are always included in the strong pixel output, but are excluded for signal-to-noise ratio calculation,
- Pixel value - pixels above certain threshold value can be marked as strong,
- Signal-to-noise (SNR) ratio - pixels with SNR above a threshold can be marked as strong,
- Connected pixels - strong pixels can be discarded if they are "alone", so their 8 directly neighboring pixels are not counted as strong pixels.
While besides bad pixels criterion, all the above are optional (can be turned off), only pixels that fulfill all enabled criteria are selected as strong pixels.
SNR ratio calculation
Signal-to-noise ratio is calculated for a rectangular area.
In horizontal direction the area is fixed - line of 1024 pixels is divided into 32 areas each of 32 pixels.
This is dictated by the data flow within the FPGA.
In vertical direction the area is flexible - it is 15 lines above and below of the given pixel.
Given very large box size, approximation are made, for example that N ≈ N-1 in calculating standard deviation.
Region-of-interest (ROI) integration
There are 16 ROIs, and the ROI map holds a 16-bit mask per pixel, so a pixel can belong to any subset of them (including none). For each ROI, sum, sum of squares, max count, and number of valid pixels will be calculated. Jungfraujoch also calculates X and Y values weighted by pixel values, though this feature is not properly tested at the moment and not integrated in downstream analysis.
ROIs are not specific to the FPGA path. The same ROI definitions — box, circle, and azimuthal
(Q-range with an optional φ-sector) — are also evaluated on CPU by the shared image_analysis/roi/
engine, so ROI statistics are produced both for the FPGA-accelerated JUNGFRAU/PSI workflow and for
detectors driven through DECTRIS SIMPLON (e.g. EIGER), which have no FPGA acquisition path.
Pixel statisitics
The following statistics are collected for each module:
- Number of masked pixels
- Number of saturated pixels (excl.masked)
- Number of error pixels (excl. masked)
- Sum of valid pixels in the module
- Minimum value of valid pixels in the module
- Maximum value of valid pixels in the module Valid pixels are not masked, not saturated, not error pixels.
Square root compression
Jungfraujoch FPGA includes lossy compression preserving counting statistic properties of X-ray image, while reducing bit width of an image.
Scheme was described in Wakonig et al., J. Appl. Cryst., 53, 574-586, 2020.
Pixel value X is replaced with round(sqrt(N*N*X)), i.e. round(N*sqrt(X)), where N is integer constant in range 1 to 16.
N is what the host writes to the sqrtmult register; the FPGA squares it before multiplying the pixel value.