The serial pre-pass runs once on the first stills_prepass_images frames (new rugnux live setting, default 1000, 100-20000, reset to 1000 by POST /initialize) instead of the doubling stages (500 -> ... -> 2000 frames, stop at 200 geometry frames). One decision: with a cell and at least RUGNUX_LIVE_GEOMETRY_MIN_FRAMES (50) strong frames among them, the beam centre, distance and cell are refined on them; otherwise the header geometry is used exactly as given - the rugnux pre-pass then neither fits nor tries a centre (ProcessConfig::stills_geometry_min_frames), and StillsGPUAnalysis applies beam/distance/cell only where the fit ran. The X-ray-only steps (saturation, shadow, defective pixels, polarization, background ring, powder rings) run as before on the start of the window; the background beam-centre estimate is reported, not applied, when the geometry is kept. Log and live-status notes say "Geometry refined from K of N frames" or "Fewer than 50 strong frames in the first N: header geometry used". Where the hold (at most half of the image buffer) fills before N frames, the pre-pass runs on those held and says so. The first N images are always analysed under stills_downsample. Rotation is unaffected. Two fixes in the stills geometry: - The refit (bundle indexed again at the fitted geometry) was skipped on every modular detector whenever the modules were not being placed, i.e. always live: placing_modules now also needs refine_modules. Offline, where the first fit places the modules and so skips its refit, the fit is run again from the placed geometry, as after a centre the trial moved. - The local beam-centre grid no longer adopts a chance maximum: it moves the centre only from one that puts spots on the lattice. OpenAPI: rugnux_settings/rugnux_status.stills_prepass_images; C++ model, frontend client and python client docs regenerated; frontend settings field. jfjoch_stills_gpu_replay and jfjoch_fpga_stills_test take -P <images>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
9.0 KiB
Tools
Besides the main services (jfjoch_broker,
jfjoch_writer, jfjoch_viewer), the repository ships a
number of command-line tools. Each prints its own usage when run without arguments (rugnux has no -h).
Data analysis
Rugnux
Offline CLI tool that runs the full crystallographic analysis pipeline (spot finding, indexing,
integration, scaling/merging) on a stored HDF5 dataset, producing a _process.h5 file and, when
merging, reflection files. Merging is on by default (--no-merge disables it). --mode picks what a
run does: mx (the above, the default), azint (only azimuthal integration, no spot
finding/indexing), scale (re-scale/merge the already-integrated reflections in a _process.h5
without re-integrating) or calibration (detector geometry from a calibrant's powder rings, written
as <prefix>.poni and <prefix>.json). See Rugnux.
rugnux installs on its own, as the rugnux package or as a standalone archive — see
Installing Rugnux.
jfjoch_extract_hkl
Extracts reflections (HKL list) from a Jungfraujoch master file; can sum the same HKL across
neighbouring images and compare against an XDS INTEGRATE.HKL reference. A developer utility: it is
built from source but is not installed into any package.
jfjoch_recompress
Re-compresses /entry/data/data of a Jungfraujoch _data_NNNNNN.h5 file from bitshuffle/LZ4 to
bitshuffle/Zstandard; every other object in the file is reproduced unchanged.
rugnux_refine_test
Runs the restrained refinement of a model against a rugnux merged MTZ without processing anything, for measuring the refinement on finished runs against another program on the same files. Prints the outcome and one JSON line for scripts.
FPGA / PCIe card management
jfjoch_pcie_status
Prints detailed status information about the card. Safe to run during data collection:
./jfjoch_pcie_status /dev/jfjoch0
jfjoch_pcie_net_cfg
Reads and modifies the network configuration of the card's interfaces:
jfjoch_pcie_net_cfg <device name>
Read configuration for all network interfaces of a device
jfjoch_pcie_net_cfg <device name> <if number>|fgen
Read configuration for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen ipv4 <IPv4 address>
Set IPv4 address for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen direct 0|1
Set direct mode for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen clear
Clear Ethernet counters for a particular network interface / internal frame generator
jfjoch_pcie_clear_net_counters
Resets the card's Ethernet, UDP and ICMP packet counters (which otherwise run from power-on):
./jfjoch_pcie_clear_net_counters /dev/jfjoch0
Testing, benchmarking and simulation
jfjoch_udp_simulator
UDP packet simulator used to test the Jungfraujoch FPGA receiver.
jfjoch_fpga_test
Exercises and benchmarks the FPGA data path and receiver on the FPGA card(s).
jfjoch_fpga_stills_test
Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector:
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-S <group>] [-X <indexer>] [-t <threads>] [-k <F>] [-P <images>] [-m <frames>] [-v] [-K[<MiB>]] <master.h5>
- The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement and orientation): the layout is the file's - its NXmx module groups, else the modules the gaps of its pixel mask mark (where the file does not give a module's readout direction, the module is read out from the top-left corner of its box; the multipixels are symmetric, so the frames are the same). Each image is folded back into module frames, the inverse of the host's geometry transformation: the inter-chip pixels the assembled image spreads one raw pixel over (2, or 4 at a chip corner) are summed back into it (the split conserves the count); where one of them is an error, gap or masked pixel the raw pixel is an error pixel, where one is saturated or the sum exceeds 16 bits it is saturated.
- The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU
conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon
counts and rugnux live stills analyses them through
FPGAImagePuller, as the broker does with rugnux live on (-v: each card's progress every 0.5 s;-k <F>: analyse 1 of every F images on average, the rugnux live settingstills_downsample;-P <N>: the pre-pass on the first N images, the rugnux live settingstills_prepass_images;-m <N>: N frames summed into each image, by the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit images). Beam centre, distance, energy and the cell come from the file (-Coverrides the cell; without one the frames are indexed de novo). Nothing is written. - Each card's generator holds module frames of its own modules only: as many images as its memory holds
module frames (less one, which the host keeps free) divided by the modules of that card - e.g. 128 module
frames: a 9M (18 modules) on one card 7 images, on two cards of 9 modules 14. The run takes the smallest
over the cards (and at most 128, the firmware's image counter; the broker's limit of 64 images does not
apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of
-iimages (default: as many as the file has). Images repeat, so throughput, latency, spots per image and indexing rate are meaningful, the merge and its statistics are not. - One collection per invocation, as fast as it goes. Nothing on the host discards an image on its way to the
analysis:
FPGAImagePullerwaits for room in its queue to the receiver and the stills receiver for room in the image buffer, so the card keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP packets would be lost at that point instead). The one exception is the writer: an analysed image that finds no image-buffer slot for the writer within the wait is not written and is counted (writer_queue_full). The card's own count of the cycles it waited for host buffers is printed per card: how long the chain behind it held the source. - The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector, which a common trigger keeps together; the run then goes at the pace of the slowest card.
- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time, start to the end of the analysis) and the analysis rate after the pre-pass.
- At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and distance it decided), the latency from the last image to the end of the analysis and rugnux's steps.
- The image buffer (
-B, MiB, default 16384) has to hold the pre-pass's frames (up toStillsGPUAnalysis::PREPASS_MAX_IMAGES, 2000) in half of it. -Kruns on the mock of the card (MockAcquisitionDevice, which returns the generator's frames as loaded) instead of/dev/jfjoch*, so the program runs without a card; the optional value is the mock's generator memory in MiB (default 64 MiB = 128 module frames).- To compare with the offline analysis of the same images, run it with
-iequal to the number of distinct images it reports (no repeats) andrugnux --force-still -e <that number>orjfjoch_stills_gpu_replay -n <that number>on the same file.
jfjoch_lite_perf_test
Performance test of the lite (CPU/GPU) analysis path — spot finding, indexing and optional file writing.
jfjoch_hdf5_test
Tests single-threaded HDF5 writer performance.
jfjoch_simplon_test
Minimal test client for a DECTRIS SIMPLON detector API.
jfjoch_stream2_replay
Replays a stored dataset as the image stream of a DECTRIS detector (a ZeroMQ PUSH socket the broker's DECTRIS receiver connects to): start message, every image at the given rate, end message. With a stand-in for the detector's control interface the broker then runs a real collection with no detector.
jfjoch_stills_gpu_replay
Replays the frames of a stored serial collection through the broker's GPU analysis of converted JUNGFRAU frames: the pre-pass on the first images, then every frame analysed with the beam centre and mask it decided. Reports the pre-pass, the indexed fraction and the sustained rate.