# Tools Besides the main services ([`jfjoch_broker`](JFJOCH_BROKER.md), [`jfjoch_writer`](JFJOCH_WRITER.md), [`jfjoch_viewer`](JFJOCH_VIEWER.md)), the repository ships a number of command-line tools. Each prints its own usage when run without arguments (`rugnux` has no `-h`). ## Data analysis ### Rugnux Offline CLI tool that runs the full crystallographic analysis pipeline (spot finding, indexing, integration, scaling/merging) on a stored HDF5 dataset, producing a `_process.h5` file and, when merging, reflection files. Merging is on by default (`--no-merge` disables it). `--mode` picks what a run does: `mx` (the above, the default), `azint` (only azimuthal integration, no spot finding/indexing), `scale` (re-scale/merge the already-integrated reflections in a `_process.h5` without re-integrating) or `calibration` (detector geometry from a calibrant's powder rings, written as `.poni` and `.json`). See [Rugnux](RUGNUX.md). `rugnux` installs on its own, as the `rugnux` package or as a standalone archive — see [Installing Rugnux](RUGNUX_INSTALL.md). ### jfjoch_extract_hkl Extracts reflections (HKL list) from a Jungfraujoch master file; can sum the same HKL across neighbouring images and compare against an XDS `INTEGRATE.HKL` reference. A developer utility: it is built from source but is not installed into any package. ### jfjoch_recompress Re-compresses `/entry/data/data` of a Jungfraujoch `_data_NNNNNN.h5` file from bitshuffle/LZ4 to bitshuffle/Zstandard; every other object in the file is reproduced unchanged. ### rugnux_refine_test Runs the restrained refinement of a model against a rugnux merged MTZ without processing anything, for measuring the refinement on finished runs against another program on the same files. Prints the outcome and one JSON line for scripts. ## FPGA / PCIe card management ### jfjoch_pcie_status Prints detailed status information about the card. Safe to run during data collection: ``` ./jfjoch_pcie_status /dev/jfjoch0 ``` ### jfjoch_pcie_net_cfg Reads and modifies the network configuration of the card's interfaces: ``` jfjoch_pcie_net_cfg Read configuration for all network interfaces of a device jfjoch_pcie_net_cfg |fgen Read configuration for a particular network interface / internal frame generator jfjoch_pcie_net_cfg |fgen ipv4 Set IPv4 address for a particular network interface / internal frame generator jfjoch_pcie_net_cfg |fgen direct 0|1 Set direct mode for a particular network interface / internal frame generator jfjoch_pcie_net_cfg |fgen clear Clear Ethernet counters for a particular network interface / internal frame generator ``` ### jfjoch_pcie_clear_net_counters Resets the card's Ethernet, UDP and ICMP packet counters (which otherwise run from power-on): ``` ./jfjoch_pcie_clear_net_counters /dev/jfjoch0 ``` ## Testing, benchmarking and simulation ### jfjoch_udp_simulator UDP packet simulator used to test the Jungfraujoch FPGA receiver. ### jfjoch_fpga_test Exercises and benchmarks the FPGA data path and receiver on the FPGA card(s). ### jfjoch_fpga_stills_test Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector: ``` jfjoch_fpga_stills_test [-i ] [-s ] [-N ] [-B ] [-C a,b,c,al,be,ga] [-S ] [-X ] [-t ] [-k ] [-P ] [-m ] [-v] [-K[]] ``` - The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement and orientation): the layout is the file's - its NXmx module groups, else the modules the gaps of its pixel mask mark (where the file does not give a module's readout direction, the module is read out from the top-left corner of its box; the multipixels are symmetric, so the frames are the same). Each image is folded back into module frames, the inverse of the host's geometry transformation: the inter-chip pixels the assembled image spreads one raw pixel over (2, or 4 at a chip corner) are summed back into it (the split conserves the count); where one of them is an error, gap or masked pixel the raw pixel is an error pixel, where one is saturated or the sum exceeds 16 bits it is saturated. - The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live on (`-v`: each card's progress every 0.5 s; `-k `: analyse 1 of every F images on average, the rugnux live setting `stills_downsample`; `-P `: the pre-pass on the first N images, the rugnux live setting `stills_prepass_images`; `-m `: N frames summed into each image, by the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit images). Beam centre, distance, energy and the cell come from the file (`-C` overrides the cell; without one the frames are indexed de novo). Nothing is written. - Each card's generator holds module frames of its own modules only: as many images as its memory holds module frames (less one, which the host keeps free) divided by the modules of that card - e.g. 128 module frames: a 9M (18 modules) on one card 7 images, on two cards of 9 modules 14. The run takes the smallest over the cards (and at most 128, the firmware's image counter; the broker's limit of 64 images does not apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images (default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and indexing rate are meaningful, the merge and its statistics are not. - One collection per invocation, as fast as it goes. Nothing on the host discards an image on its way to the analysis: `FPGAImagePuller` waits for room in its queue to the receiver and the stills receiver for room in the image buffer, so the card keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP packets would be lost at that point instead). The one exception is the writer: an analysed image that finds no image-buffer slot for the writer within the wait is not written and is counted (`writer_queue_full`). The card's own count of the cycles it waited for host buffers is printed per card: how long the chain behind it held the source. - The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector, which a common trigger keeps together; the run then goes at the pace of the slowest card. - It reports the rate images were delivered (first to last image), the sustained rate (images / wall time, start to the end of the analysis) and the analysis rate after the pre-pass. - At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and distance it decided), the latency from the last image to the end of the analysis and rugnux's steps. - The image buffer (`-B`, MiB, default 16384) has to hold the pre-pass's frames (up to `StillsGPUAnalysis::PREPASS_MAX_IMAGES`, 2000) in half of it. - `-K` runs on the mock of the card (`MockAcquisitionDevice`, which returns the generator's frames as loaded) instead of `/dev/jfjoch*`, so the program runs without a card; the optional value is the mock's generator memory in MiB (default 64 MiB = 128 module frames). - To compare with the offline analysis of the same images, run it with `-i` equal to the number of distinct images it reports (no repeats) and `rugnux --force-still -e ` or `jfjoch_stills_gpu_replay -n ` on the same file. ### jfjoch_lite_perf_test Performance test of the lite (CPU/GPU) analysis path — spot finding, indexing and optional file writing. ### jfjoch_hdf5_test Tests single-threaded HDF5 writer performance. ### jfjoch_simplon_test Minimal test client for a DECTRIS SIMPLON detector API. ### jfjoch_stream2_replay Replays a stored dataset as the image stream of a DECTRIS detector (a ZeroMQ PUSH socket the broker's DECTRIS receiver connects to): start message, every image at the given rate, end message. With a stand-in for the detector's control interface the broker then runs a real collection with no detector. ### jfjoch_stills_gpu_replay Replays the frames of a stored serial collection through the broker's GPU analysis of converted JUNGFRAU frames: the pre-pass on the first images, then every frame analysed with the beam centre and mask it decided. Reports the pre-pass, the indexed fraction and the sustained rate.