Build Packages / build:portable:linux-x86_64:nocuda (push) Successful in 16m43s
Build Packages / build:jfjoch:ubuntu2404:nocuda (push) Successful in 28m46s
Build Packages / build:jfjoch:rocky8:cuda-sls9 (push) Successful in 32m5s
Build Packages / build:jfjoch:rocky9:cuda (push) Successful in 39m44s
Build Packages / build:portable:macos-arm64:nocuda (push) Successful in 5m7s
Build Packages / Create release (push) Successful in 8s
Build Packages / Build documentation (push) Successful in 49s
Build Packages / build:portable:linux-x86_64:cuda (push) Successful in 22m30s
Build Packages / build:jfjoch:rocky8:nocuda (push) Successful in 31m54s
Build Packages / build:rugnux:linux-aarch64:cuda (push) Successful in 11m29s
Build Packages / build:portable:windows-x86_64:nocuda (push) Successful in 23m51s
Build Packages / build:jfjoch:ubuntu2204:nocuda (push) Successful in 26m7s
Build Packages / build:jfjoch:ubuntu2204:cuda (push) Successful in 29m38s
Build Packages / build:portable:windows-x86_64:cuda (push) Successful in 28m6s
Build Packages / HDF5 consumer tests (DIALS, XDS) (push) Successful in 29m44s
Build Packages / build:jfjoch:rocky9:nocuda (push) Successful in 33m22s
Build Packages / Generate python client (push) Successful in 47s
Build Packages / build:jfjoch:rocky9:cuda-sls9 (push) Successful in 34m6s
Build Packages / Unit tests (push) Successful in 1h2m47s
Build Packages / build:jfjoch:rocky8:cuda (push) Successful in 35m59s
Build Packages / build:jfjoch:ubuntu2404:cuda (push) Successful in 34m8s
Build Packages / Validation Battery (push) Successful in 38m31s
* Rugnux: Refinement several times faster, one protocol on the CPU and the GPU, parts of it on the Apple GPU; its random draws are the same on every platform. * Rugnux: Refinement follows common practice: the free set decides only the restraint weights and the displacement model (isotropic, TLS or anisotropic, by Hamilton's test), everything else is decided on the working set; lower R-free for placed and predicted models. * Rugnux: Restraint dictionaries via `--restraints` (or among the inputs) and more built-in ligands; a model whose space group is a subgroup of the data's is refined in its own group; reflections plus a model select `--mode refine` on their own. * Rugnux: `-C` fixes the crystal system on the axes as typed; `-S` alone fixes the space group only, the cell is still found and reduced; a space group or cell stored in the input file is ignored, with a line in the log. * Rugnux: More reliable multi-sweep merging; `--mode scale` merges several stills runs as one; `_plot.txt` and the report name each input; a second crystal form is reported for stills given a cell. * jfjoch_broker (experimental): Live Rugnux on FPGA detectors (JUNGFRAU) takes the same path as a DECTRIS stream, rotation and serial alike, model step included, with analysis threads per GPU; frames waiting for the analysis can be viewed; no host stage drops a frame - a source faster than the analysis is held at the card. * jfjoch_broker: Summation of more than 256 frames per image is split between the card and the host (`summation_mode` is then `hybrid`), and live Rugnux runs on such collections; pedestal runs take a receiver of their own. * jfjoch_broker (experimental): `stills_downsample` analyses 1 of every N serial images (chosen by a hash; the first 2000 all; the others written without analysis; reset to 1 by `/initialize`); `max_spot_count` left out means 250 for per-image indexing and up to 1000 for live stills; the live status gives the merge before the model work, the fitted resolution and the anomalous peak height of each unexplained blob. * jfjoch_broker: Without a `pcie` entry the broker runs on a mock of the card instead of the HLS simulation; its spot, ROI and azimuthal results are zero then. * Web frontend: Redesigned Rugnux live page; Vite 8, TypeScript 6, Mol* 5.13 and dependency security updates. * Tools: `jfjoch_fpga_stills_test` measures live stills on the cards with a stored dataset; `jfjoch_fpga_test` loses the HLS-simulation option. * Other: FFTW no longer fetched or linked (the CPU FFT indexer keeps the name `FFTW`); PCIe driver now GPL-2.0-or-later and third-party notices complete; several crash fixes. Reviewed-on: #98 Co-authored-by: Filip Leonarski <filip.leonarski@psi.ch>
150 lines
9.0 KiB
Markdown
150 lines
9.0 KiB
Markdown
# Tools
|
|
|
|
Besides the main services ([`jfjoch_broker`](JFJOCH_BROKER.md),
|
|
[`jfjoch_writer`](JFJOCH_WRITER.md), [`jfjoch_viewer`](JFJOCH_VIEWER.md)), the repository ships a
|
|
number of command-line tools. Each prints its own usage when run without arguments (`rugnux` has no `-h`).
|
|
|
|
## Data analysis
|
|
|
|
### Rugnux
|
|
Offline CLI tool that runs the full crystallographic analysis pipeline (spot finding, indexing,
|
|
integration, scaling/merging) on a stored HDF5 dataset, producing a `_process.h5` file and, when
|
|
merging, reflection files. Merging is on by default (`--no-merge` disables it). `--mode` picks what a
|
|
run does: `mx` (the above, the default), `azint` (only azimuthal integration, no spot
|
|
finding/indexing), `scale` (re-scale/merge the already-integrated reflections in a `_process.h5`
|
|
without re-integrating) or `calibration` (detector geometry from a calibrant's powder rings, written
|
|
as `<prefix>.poni` and `<prefix>.json`). See [Rugnux](RUGNUX.md).
|
|
|
|
`rugnux` installs on its own, as the `rugnux` package or as a standalone archive — see
|
|
[Installing Rugnux](RUGNUX_INSTALL.md).
|
|
|
|
### jfjoch_extract_hkl
|
|
Extracts reflections (HKL list) from a Jungfraujoch master file; can sum the same HKL across
|
|
neighbouring images and compare against an XDS `INTEGRATE.HKL` reference. A developer utility: it is
|
|
built from source but is not installed into any package.
|
|
|
|
### jfjoch_recompress
|
|
Re-compresses `/entry/data/data` of a Jungfraujoch `_data_NNNNNN.h5` file from bitshuffle/LZ4 to
|
|
bitshuffle/Zstandard; every other object in the file is reproduced unchanged.
|
|
|
|
### rugnux_refine_test
|
|
Runs the restrained refinement of a model against a rugnux merged MTZ without processing anything, for
|
|
measuring the refinement on finished runs against another program on the same files. Prints the outcome
|
|
and one JSON line for scripts.
|
|
|
|
## FPGA / PCIe card management
|
|
|
|
### jfjoch_pcie_status
|
|
Prints detailed status information about the card. Safe to run during data collection:
|
|
```
|
|
./jfjoch_pcie_status /dev/jfjoch0
|
|
```
|
|
|
|
### jfjoch_pcie_net_cfg
|
|
Reads and modifies the network configuration of the card's interfaces:
|
|
```
|
|
jfjoch_pcie_net_cfg <device name>
|
|
Read configuration for all network interfaces of a device
|
|
jfjoch_pcie_net_cfg <device name> <if number>|fgen
|
|
Read configuration for a particular network interface / internal frame generator
|
|
jfjoch_pcie_net_cfg <device name> <if number>|fgen ipv4 <IPv4 address>
|
|
Set IPv4 address for a particular network interface / internal frame generator
|
|
jfjoch_pcie_net_cfg <device name> <if number>|fgen direct 0|1
|
|
Set direct mode for a particular network interface / internal frame generator
|
|
jfjoch_pcie_net_cfg <device name> <if number>|fgen clear
|
|
Clear Ethernet counters for a particular network interface / internal frame generator
|
|
```
|
|
|
|
### jfjoch_pcie_clear_net_counters
|
|
Resets the card's Ethernet, UDP and ICMP packet counters (which otherwise run from power-on):
|
|
```
|
|
./jfjoch_pcie_clear_net_counters /dev/jfjoch0
|
|
```
|
|
|
|
## Testing, benchmarking and simulation
|
|
|
|
### jfjoch_udp_simulator
|
|
UDP packet simulator used to test the Jungfraujoch FPGA receiver.
|
|
|
|
### jfjoch_fpga_test
|
|
Exercises and benchmarks the FPGA data path and receiver on the FPGA card(s).
|
|
|
|
### jfjoch_fpga_stills_test
|
|
Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector:
|
|
|
|
```
|
|
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-S <group>] [-X <indexer>] [-t <threads>] [-k <F>] [-P <images>] [-m <frames>] [-v] [-K[<MiB>]] <master.h5>
|
|
```
|
|
|
|
- The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement
|
|
and orientation): the layout is the file's - its NXmx module groups, else the modules the gaps of its pixel
|
|
mask mark (where the file does not give a module's readout direction, the module is read out from the
|
|
top-left corner of its box; the multipixels are symmetric, so the frames are the same). Each image is folded back into module
|
|
frames, the inverse of the host's geometry transformation: the inter-chip pixels the assembled image
|
|
spreads one raw pixel over (2, or 4 at a chip corner) are summed back into it (the split conserves
|
|
the count); where one of them is an error, gap or masked pixel the raw pixel is an error pixel, where
|
|
one is saturated or the sum exceeds 16 bits it is saturated.
|
|
- The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU
|
|
conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon
|
|
counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live
|
|
on (`-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
|
|
F images on average, the rugnux live setting `stills_downsample`; `-P <N>`: the pre-pass on the first N images,
|
|
the rugnux live setting `stills_prepass_images`; `-m <N>`: N frames summed into each image, by
|
|
the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit
|
|
images). Beam centre, distance, energy and the cell
|
|
come from the file (`-C` overrides the cell; without one the frames are indexed de novo). Nothing is
|
|
written.
|
|
- Each card's generator holds module frames of its own modules only: as many images as its memory holds
|
|
module frames (less one, which the host keeps free) divided by the modules of that card - e.g. 128 module
|
|
frames: a 9M (18 modules) on one card 7 images, on two cards of 9 modules 14. The run takes the smallest
|
|
over the cards (and at most 128, the firmware's image counter; the broker's limit of 64 images does not
|
|
apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images
|
|
(default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and
|
|
indexing rate are meaningful, the merge and its statistics are not.
|
|
- One collection per invocation, as fast as it goes. Nothing on the host discards an image on its way to the
|
|
analysis: `FPGAImagePuller` waits for room in its queue to the receiver and the stills receiver for room in the
|
|
image buffer, so the card
|
|
keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through
|
|
the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP
|
|
packets would be lost at that point instead). The one exception is the writer: an analysed image that finds
|
|
no image-buffer slot for the writer within the wait is not written and is counted (`writer_queue_full`). The
|
|
card's own count of the cycles it waited for host buffers is printed per card: how long the chain behind it
|
|
held the source.
|
|
- The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different
|
|
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector,
|
|
which a common trigger keeps together; the run then goes at the pace of the slowest card.
|
|
- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time,
|
|
start to the end of the analysis) and the analysis rate after the pre-pass.
|
|
- At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing
|
|
rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and
|
|
distance it decided), the latency from the last image to the end of the analysis and rugnux's steps.
|
|
- The image buffer (`-B`, MiB, default 16384) has to hold the pre-pass's frames
|
|
(up to `StillsGPUAnalysis::PREPASS_MAX_IMAGES`, 2000) in half of it.
|
|
- `-K` runs on the mock of the card (`MockAcquisitionDevice`, which returns the generator's frames as
|
|
loaded) instead of `/dev/jfjoch*`, so the program runs without a card; the optional value is the mock's
|
|
generator memory in MiB (default 64 MiB = 128 module frames).
|
|
- To compare with the offline analysis of the same images, run it with `-i` equal to the number of
|
|
distinct images it reports (no repeats) and `rugnux --force-still -e <that number>` or
|
|
`jfjoch_stills_gpu_replay -n <that number>` on the same file.
|
|
|
|
### jfjoch_lite_perf_test
|
|
Performance test of the lite (CPU/GPU) analysis path — spot finding, indexing and optional file
|
|
writing.
|
|
|
|
### jfjoch_hdf5_test
|
|
Tests single-threaded HDF5 writer performance.
|
|
|
|
### jfjoch_simplon_test
|
|
Minimal test client for a DECTRIS SIMPLON detector API.
|
|
|
|
### jfjoch_stream2_replay
|
|
Replays a stored dataset as the image stream of a DECTRIS detector (a ZeroMQ PUSH socket the
|
|
broker's DECTRIS receiver connects to): start message, every image at the given rate, end message.
|
|
With a stand-in for the detector's control interface the broker then runs a real collection with no
|
|
detector.
|
|
|
|
### jfjoch_stills_gpu_replay
|
|
Replays the frames of a stored serial collection through the broker's GPU analysis of converted
|
|
JUNGFRAU frames: the pre-pass on the first images, then every frame analysed with the beam centre and
|
|
mask it decided. Reports the pre-pass, the indexed fraction and the sustained rate.
|