Files
leonarski_fandClaude Opus 5.5 f251178531 Rugnux live stills: one pre-pass on the first N images, one geometry decision
The serial pre-pass runs once on the first stills_prepass_images frames (new rugnux live
setting, default 1000, 100-20000, reset to 1000 by POST /initialize) instead of the doubling
stages (500 -> ... -> 2000 frames, stop at 200 geometry frames). One decision: with a cell and at
least RUGNUX_LIVE_GEOMETRY_MIN_FRAMES (50) strong frames among them, the beam centre, distance
and cell are refined on them; otherwise the header geometry is used exactly as given - the
rugnux pre-pass then neither fits nor tries a centre (ProcessConfig::stills_geometry_min_frames),
and StillsGPUAnalysis applies beam/distance/cell only where the fit ran. The X-ray-only steps
(saturation, shadow, defective pixels, polarization, background ring, powder rings) run as before
on the start of the window; the background beam-centre estimate is reported, not applied, when
the geometry is kept. Log and live-status notes say "Geometry refined from K of N frames" or
"Fewer than 50 strong frames in the first N: header geometry used". Where the hold (at most half
of the image buffer) fills before N frames, the pre-pass runs on those held and says so. The
first N images are always analysed under stills_downsample. Rotation is unaffected.

Two fixes in the stills geometry:
- The refit (bundle indexed again at the fitted geometry) was skipped on every modular detector
  whenever the modules were not being placed, i.e. always live: placing_modules now also needs
  refine_modules. Offline, where the first fit places the modules and so skips its refit, the
  fit is run again from the placed geometry, as after a centre the trial moved.
- The local beam-centre grid no longer adopts a chance maximum: it moves the centre only from
  one that puts spots on the lattice.

OpenAPI: rugnux_settings/rugnux_status.stills_prepass_images; C++ model, frontend client and
python client docs regenerated; frontend settings field. jfjoch_stills_gpu_replay and
jfjoch_fpga_stills_test take -P <images>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
2026-11-19 10:15:17 +01:00

150 lines
9.0 KiB
Markdown

# Tools
Besides the main services ([`jfjoch_broker`](JFJOCH_BROKER.md),
[`jfjoch_writer`](JFJOCH_WRITER.md), [`jfjoch_viewer`](JFJOCH_VIEWER.md)), the repository ships a
number of command-line tools. Each prints its own usage when run without arguments (`rugnux` has no `-h`).
## Data analysis
### Rugnux
Offline CLI tool that runs the full crystallographic analysis pipeline (spot finding, indexing,
integration, scaling/merging) on a stored HDF5 dataset, producing a `_process.h5` file and, when
merging, reflection files. Merging is on by default (`--no-merge` disables it). `--mode` picks what a
run does: `mx` (the above, the default), `azint` (only azimuthal integration, no spot
finding/indexing), `scale` (re-scale/merge the already-integrated reflections in a `_process.h5`
without re-integrating) or `calibration` (detector geometry from a calibrant's powder rings, written
as `<prefix>.poni` and `<prefix>.json`). See [Rugnux](RUGNUX.md).
`rugnux` installs on its own, as the `rugnux` package or as a standalone archive — see
[Installing Rugnux](RUGNUX_INSTALL.md).
### jfjoch_extract_hkl
Extracts reflections (HKL list) from a Jungfraujoch master file; can sum the same HKL across
neighbouring images and compare against an XDS `INTEGRATE.HKL` reference. A developer utility: it is
built from source but is not installed into any package.
### jfjoch_recompress
Re-compresses `/entry/data/data` of a Jungfraujoch `_data_NNNNNN.h5` file from bitshuffle/LZ4 to
bitshuffle/Zstandard; every other object in the file is reproduced unchanged.
### rugnux_refine_test
Runs the restrained refinement of a model against a rugnux merged MTZ without processing anything, for
measuring the refinement on finished runs against another program on the same files. Prints the outcome
and one JSON line for scripts.
## FPGA / PCIe card management
### jfjoch_pcie_status
Prints detailed status information about the card. Safe to run during data collection:
```
./jfjoch_pcie_status /dev/jfjoch0
```
### jfjoch_pcie_net_cfg
Reads and modifies the network configuration of the card's interfaces:
```
jfjoch_pcie_net_cfg <device name>
Read configuration for all network interfaces of a device
jfjoch_pcie_net_cfg <device name> <if number>|fgen
Read configuration for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen ipv4 <IPv4 address>
Set IPv4 address for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen direct 0|1
Set direct mode for a particular network interface / internal frame generator
jfjoch_pcie_net_cfg <device name> <if number>|fgen clear
Clear Ethernet counters for a particular network interface / internal frame generator
```
### jfjoch_pcie_clear_net_counters
Resets the card's Ethernet, UDP and ICMP packet counters (which otherwise run from power-on):
```
./jfjoch_pcie_clear_net_counters /dev/jfjoch0
```
## Testing, benchmarking and simulation
### jfjoch_udp_simulator
UDP packet simulator used to test the Jungfraujoch FPGA receiver.
### jfjoch_fpga_test
Exercises and benchmarks the FPGA data path and receiver on the FPGA card(s).
### jfjoch_fpga_stills_test
Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector:
```
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-S <group>] [-X <indexer>] [-t <threads>] [-k <F>] [-P <images>] [-m <frames>] [-v] [-K[<MiB>]] <master.h5>
```
- The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement
and orientation): the layout is the file's - its NXmx module groups, else the modules the gaps of its pixel
mask mark (where the file does not give a module's readout direction, the module is read out from the
top-left corner of its box; the multipixels are symmetric, so the frames are the same). Each image is folded back into module
frames, the inverse of the host's geometry transformation: the inter-chip pixels the assembled image
spreads one raw pixel over (2, or 4 at a chip corner) are summed back into it (the split conserves
the count); where one of them is an error, gap or masked pixel the raw pixel is an error pixel, where
one is saturated or the sum exceeds 16 bits it is saturated.
- The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU
conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon
counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live
on (`-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
F images on average, the rugnux live setting `stills_downsample`; `-P <N>`: the pre-pass on the first N images,
the rugnux live setting `stills_prepass_images`; `-m <N>`: N frames summed into each image, by
the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit
images). Beam centre, distance, energy and the cell
come from the file (`-C` overrides the cell; without one the frames are indexed de novo). Nothing is
written.
- Each card's generator holds module frames of its own modules only: as many images as its memory holds
module frames (less one, which the host keeps free) divided by the modules of that card - e.g. 128 module
frames: a 9M (18 modules) on one card 7 images, on two cards of 9 modules 14. The run takes the smallest
over the cards (and at most 128, the firmware's image counter; the broker's limit of 64 images does not
apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images
(default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and
indexing rate are meaningful, the merge and its statistics are not.
- One collection per invocation, as fast as it goes. Nothing on the host discards an image on its way to the
analysis: `FPGAImagePuller` waits for room in its queue to the receiver and the stills receiver for room in the
image buffer, so the card
keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through
the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP
packets would be lost at that point instead). The one exception is the writer: an analysed image that finds
no image-buffer slot for the writer within the wait is not written and is counted (`writer_queue_full`). The
card's own count of the cycles it waited for host buffers is printed per card: how long the chain behind it
held the source.
- The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector,
which a common trigger keeps together; the run then goes at the pace of the slowest card.
- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time,
start to the end of the analysis) and the analysis rate after the pre-pass.
- At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing
rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and
distance it decided), the latency from the last image to the end of the analysis and rugnux's steps.
- The image buffer (`-B`, MiB, default 16384) has to hold the pre-pass's frames
(up to `StillsGPUAnalysis::PREPASS_MAX_IMAGES`, 2000) in half of it.
- `-K` runs on the mock of the card (`MockAcquisitionDevice`, which returns the generator's frames as
loaded) instead of `/dev/jfjoch*`, so the program runs without a card; the optional value is the mock's
generator memory in MiB (default 64 MiB = 128 module frames).
- To compare with the offline analysis of the same images, run it with `-i` equal to the number of
distinct images it reports (no repeats) and `rugnux --force-still -e <that number>` or
`jfjoch_stills_gpu_replay -n <that number>` on the same file.
### jfjoch_lite_perf_test
Performance test of the lite (CPU/GPU) analysis path — spot finding, indexing and optional file
writing.
### jfjoch_hdf5_test
Tests single-threaded HDF5 writer performance.
### jfjoch_simplon_test
Minimal test client for a DECTRIS SIMPLON detector API.
### jfjoch_stream2_replay
Replays a stored dataset as the image stream of a DECTRIS detector (a ZeroMQ PUSH socket the
broker's DECTRIS receiver connects to): start message, every image at the given rate, end message.
With a stand-in for the detector's control interface the broker then runs a real collection with no
detector.
### jfjoch_stills_gpu_replay
Replays the frames of a stored serial collection through the broker's GPU analysis of converted
JUNGFRAU frames: the pre-pass on the first images, then every frame analysed with the beam centre and
mask it decided. Reports the pre-pass, the indexed fraction and the sustained rate.