Jungfraujoch/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this is

Jungfraujoch is the data-acquisition and analysis system for the PSI JUNGFRAU and EIGER X-ray detectors. It receives detector data, runs it through an FPGA-accelerated pipeline (spot finding, azimuthal/ROI integration, compression), streams images out over ZeroMQ for writing to HDF5, and runs crystallographic analysis (indexing, integration, scaling/merging). Most authoritative documentation lives in docs/ and on Read The Docs (https://jungfraujoch.readthedocs.io). When changing CLI behaviour, the program's own usage message is the source of truth, not the docs.

Build

Out-of-source CMake build, C++20, heavy use of FetchContent (spdlog, zstd, HDF5, slsDetectorPackage, Catch2, cpp-httplib, libzmq, Ceres, fast-feedback-indexer are downloaded and statically linked — the first configure needs network access and is slow).

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc) jfjoch_broker      # the main service; build other targets by name

Key CMake options:

• JFJOCH_USE_CUDA (default ON) — GPU path. Needs CUDA ≥ 12.8. Provides the ffbidx and fft GPU indexers; without it only the CPU fftw indexer is available (requires FFTW at configure time, auto-detected). CUDA absence is not a build error.
• JFJOCH_WRITER_ONLY (default OFF) — builds only the HDF5 writer; skips broker, FPGA, receiver, analysis, tests, frontend.
• JFJOCH_VIEWER_BUILD (default OFF) — builds the Qt6 jfjoch_viewer desktop app.
• SLS9 (default OFF) — build against slsDetectorPackage 9.2.0 instead of 8.0.2.

The frontend is a separate custom target: make frontend (runs npm ci && npm run build in frontend/). It is not built by default unless installing.

Test

Tests use Catch2 and are collected into a single binary tests/jfjoch_test.

make -j$(nproc) jfjoch_test
cd tests
./jfjoch_test                       # all tests
./jfjoch_test "<test name>"         # one test case (exact name in TEST_CASE)
./jfjoch_test "[tag]"               # by tag
./jfjoch_test -r junit -o report.xml

make jfjoch_hdf5_test builds the HDF5 write-speed benchmark, also used by CI to produce files that are validated against XDS (Durin/Neggia) and CrystFEL.

Lint config is .clang-tidy (broad * check set with many exceptions; namespaces lower_case, classes CamelCase, global constants UPPER_CASE).

Code style

The overriding principle is simple, readable code — favour the smallest, most direct implementation that a reader can verify at a glance. Extra abstraction, speculative guards, and clever-but-dense constructs are treated as actively harmful, not as polish. When torn between a tidy abstraction and a flat, obvious version, pick the obvious one.

This matters most in image_analysis/pixel_refinement/ (e.g. PixelRefine.cpp), which is an experimental prototype where readability is how the physics gets verified — keep it especially plain. Do not add defensive/unrequested code (extra validation, rejection heuristics, "just in case" branches) without asking first; if a guard isn't clearly needed, leave it out.

Match the surrounding code's idiom, naming, and comment density rather than importing a different style.

No sample identities in the repository

Never put sample names or sample-specific measured values into source code, comments, documentation, commit messages, or test fixtures. Datasets belong to users and may be confidential or embargoed; anything committed can leak outside the group working on them. This is a hard rule, not a preference.

• Forbidden: sample/dataset names or internal codes (a protein name, a beamline dataset ID, a run label), and measured unit-cell parameters tied to a real sample (this is the most sensitive — never hardcode "protein X has cell a,b,c").
• Fine: general crystallographic descriptors — space group / Laue class ("a P2₁ crystal", "a holohedral 422 case"), lattice centering, twinning, "a crystal whose true axis is reported as its 3× harmonic", pseudo-symmetry, etc. Describe the crystallographic situation, not the specimen.
• Exception — plain lysozyme. Lysozyme (HEWL) is the field's universal standard test specimen, not a user's confidential dataset, so naming it and using its well-known reference cell (tetragonal ~79/79/38, P4₃2₁2) in tests, docs and comments is allowed. This carve-out is only for generic lysozyme as a benchmark; a named user dataset that happens to be lysozyme is still off limits, and every other real sample remains forbidden.
• Tests must otherwise use neutral names (e.g. tetragonal_uc, not a specimen-named variable) and, where a cell is needed, a synthetic cell chosen for the test — not a real dataset's parameters.

When a bug was found on a specific dataset, commit the behaviour ("de-novo indexing adopted a spurious axis-multiple supercell"), never the dataset.

Local end-to-end run (no detector / no FPGA)

The FPGA HLS logic can be simulated on the CPU (HLSSimulatedDevice), so the full software stack runs without hardware (slowly — fixed-point math on CPU). See docs/JFJOCH_BROKER.md for the canonical walkthrough.

cd build/broker
./jfjoch_broker ../../etc/broker_local.json 5232      # config JSON + HTTP port
# then, separately:
cd tests/test_data && python jfjoch_broker_test.py     # feeds a test image, starts collection
# observe at http://localhost:5232 ; HDF5 is written under build/broker

etc/broker_local.json, broker_eiger.json, broker_crmx.json are example broker configs (schema = jfjoch_settings in broker/jfjoch_api.yaml).

Architecture

Data flow (online): detector → FPGA acquisition (fpga/, acquisition_device/) → receiver/ builds full images from per-module FPGA output → image_pusher/ streams CBOR-encoded images over ZeroMQ → jfjoch_writer (writer/) consumes the stream and writes NXmx HDF5. The broker also emits a low-rate preview stream and a metadata stream (preview/).

Writer file split: one acquisition produces one _master.h5 plus many _data_NNNNNN.h5 files. Dataset-wide metadata (geometry, detector config, ROI/azimuthal definitions — anything fixed for the whole run) is written to the master file in writer/HDF5NXmx.cpp (the NXmx class). Per-image arrays (one entry per frame) are written to the data files by the HDF5DataFilePlugin subclasses in writer/. Put shared metadata in NXmx, not in a data-file plugin.

The HDF5 master/data layout is one of three FileWriterFormats (common/JFJochMessages.h), all NXmx: NXmxLegacy (master + _data_NNNNNN.h5 joined by external links), NXmxVDS (master + data joined by HDF5 virtual datasets — the default), and NXmxIntegrated (a single self-contained file, no separate data files). Per-image plugins must work for all three; with NXmxIntegrated "master" and "data" are the same file.

Two acquisition workflows: the FPGA-accelerated path (JUNGFRAU at PSI; FPGA does masking, summation, spot finding, ROI/azimuthal integration, compression) and the DECTRIS SIMPLON path (EIGER), which has no FPGA — masking/ROI/azimuthal analysis then runs on CPU through the shared image_analysis/ library. Treat ROI and azimuthal features as available in both workflows, not FPGA-only.

jfjoch_broker (broker/) is the central online service: HTTP/REST + OpenAPI control plane, FPGA configuration, image building, ZeroMQ output. JFJochStateMachine drives acquisition state; JFJochServices wires the pieces; OpenAPIConvert/JFJochBrokerParser translate between the generated API model and internal types.

Three analysis frontends share one analysis library (image_analysis/, built as JFJochImageAnalysis):

• jfjoch_broker — online, real-time (FPGA + GPU).
• jfjoch_viewer — interactive Qt desktop (viewer/), results not persisted.
• rugnux (tools/rugnux_cli.cpp, built on the Rugnux library in rugnux/) — offline batch over a stored HDF5; writes _process.h5 and .mtz/.cif/.hkl. Merging is on by default (--no-merge to disable); --azint-only runs only azimuthal integration and --scale re-scales/merges the already-integrated reflections in a _process.h5. (rugnux = the data-processing half of the system; see docs/NAMING.md.)

image_analysis/ pipeline (subdirs): spot_finding, indexing (ffbidx/fft GPU, fftw CPU), lattice_search, geom_refinement, pixel_refinement, bragg_prediction, bragg_integration, rotation_indexer, azint, roi, scale_merge. Least-squares refinement uses Ceres (fetched, built with miniglog, no MKL, CXX_THREADS). ffbidx needs a known cell (-C) and suits sparse serial stills; fft/fftw index de novo and suit strong rotation data.

FPGA (fpga/): hls/ is the Vitis HLS source (image-analysis kernels), hls_simulation/ runs that same HLS on CPU for hardware-free testing, host_library/ is the host-side driver, pcie_driver/ is the kernel module. The HLS algorithms are documented in docs/FPGA_DATA_ANALYSIS.md.

Detector control (detector_control/): wrappers for SLS (JUNGFRAU) and DECTRIS SIMPLON (EIGER). jungfrau/: JUNGFRAU ADU→energy gain/pedestal calibration.

Other libs: common/ (geometry, diffraction experiment, image buffer, CUDA wrappers — the shared core, linked nearly everywhere), compression/ (zstd + bitshuffle + sqrt lossy), frame_serialize/ (CBOR stream codec), gemmi_gph/ (vendored GEMMI for MTZ/XDS_ASCII I/O), xds-plugin/ (XDS HDF5 read plugin).

Portability (jfjoch_viewer)

Cross-platform support is a goal only for jfjoch_viewer and its dependency tree — keep that code, and any shared library it transitively links (common/, image_analysis/, reader/, gemmi_gph/, etc.), MSVC-compatible so the viewer can build on Windows. The rest of the project (broker, receiver, FPGA host, detector control, …) is Linux-only and does not need to be portable; don't constrain it for portability's sake.

• Windows/MSVC is the primary portability target. The end goal is a Windows viewer built with MSVC and CUDA (JFJOCH_USE_CUDA=ON): GPU processing is a wanted feature, not optional, so the intended Windows config is the full GPU path (ffbidx, GPU fft), not a CPU-only fallback. MSVC is required regardless, because CUDA on Windows requires it. Avoid GCC/Clang-only extensions, POSIX-only APIs, and other non-MSVC constructs in viewer-reachable code, and keep CUDA-reachable viewer code (ffbidx, GPU indexers) MSVC-buildable too.
• macOS is a nice-to-have for the viewer. It rules out CUDA, so anything the viewer depends on must also have a working CPU-only / non-CUDA path (the JFJOCH_USE_CUDA=OFF, fftw-indexer configuration). This non-CUDA path must keep working, but it is the macOS fallback — not the intended Windows configuration.

A self-contained Windows build still needs a few dependencies that the Linux build picks up from the system and that are not auto-provided via FetchContent (besides Qt, supplied externally): ZLIB (pulled by hdf5/libtiff/gemmi/libzmq — a bundled zlib-ng in ZLIB_COMPAT mode is the intended fix, presented as the ZLIB::ZLIB target so the scattered find_package(ZLIB) calls resolve), libjpeg-turbo (used by preview/; upstream discourages add_subdirectory, so bring it in via ExternalProject), and Eigen (header-only; needed by Ceres, by the analysis libs directly, and by ffbidx under CUDA — fetch it and point Ceres' internal find_package(Eigen3) at it). Wire each guarded so the Linux build, which finds these on the system, stays unchanged.

OpenAPI is the single source of truth

broker/jfjoch_api.yaml defines the entire REST API and the shared data schemas. From it, update_version.sh regenerates three clients — do not hand-edit generated code:

• C++ server model → broker/gen/ (cpp-pistache-server generator; compiled as JFJochAPI).
• Python client → python-client/ (and gen_python_client.sh, published as PyPI jfjoch-client).
• TypeScript frontend client → frontend/src/client/ (hey-api openapi-ts, npm run openapi).

When you change jfjoch_api.yaml, regenerate the relevant client(s); for a version bump write the new version into VERSION and run update_version.sh (which reads it and rewrites frontend/src/version.ts, docs/conf.py, the python client and the Redoc html).

Frontend

React 19 + TypeScript + MUI + Vite (frontend/). Data layer is generated from the OpenAPI spec (@hey-api/openapi-ts → fetch client + TanStack Query hooks + zod schemas). Scripts: npm start (dev server), npm run build (tsc + vite), npm run openapi (regen client), npm run redocly4broker (regen broker/redoc-static.html).

Adding a per-image scalar quantity

A per-image scalar (e.g. ice_ring_score, bkg_estimate, mosaicity) flows analysis → message → CBOR → HDF5 → scan-result/plot → API → viewer/frontend. To add one, mirror an existing float scalar (bkg_estimate is a clean template) at every layer:

1. Compute where the azint profile is finalized: image_analysis/MXAnalysisWithoutFPGA.cpp (CPU), receiver/JFJochReceiverFPGA.cpp (FPGA), and the offline azint worker in rugnux/Rugnux.cpp.
2. Message (common/JFJochMessages.h): std::optional<float> in DataMessage, std::vector<float> in EndMessage.
3. CBOR: encode in frame_serialize/CBORStream2Serializer.cpp (DataMessage block and END block), decode in CBORStream2Deserializer.cpp (both). Optional fields are back-compatible — no version bump.
4. HDF5 write: writer/HDF5DataFilePluginMX.{h,cpp} — an AutoIncrVector<float> with reserve / per-image write / SaveVector("/entry/MX/<name>") (per-image arrays live in the data-file plugin); plus the NXmx master write in writer/HDF5NXmx.cpp (SaveVectorIfMissing(..., end.<name>)). HDF5 read-back (so a stored file re-opens, e.g. in the viewer) is in reader/HDF5MetadataSource.cpp, NOT JFJochHDF5Reader.cpp: mirror the three bkgEstimate sites — master ReadOptVector, data-file ReadVector into the dataset, and the per-image message population.
5. Scan result: common/ScanResult.h (ScanResultElem) + common/ScanResultGenerator.cpp (copy in Add, resize+fill in FillEndMessage).
6. Receiver plot: common/Plot.h (PlotType) + common/JFJochReceiverPlots.{h,cpp} (StatusVector
   • Clear / AddElement / GetPlots / GetPlotRaw cases).
7. API: add to the plot_type enum and the scan_result images schema in broker/jfjoch_api.yaml, regenerate the C++ model (java -jar openapi-generator-cli.jar generate -i broker/jfjoch_api.yaml -o broker/gen -g cpp-pistache-server) and the frontend client (cd frontend && npm run openapi), then wire broker/OpenAPIConvert.cpp (ConvertPlotType string→enum and the Convert(ScanResult) setter).
8. Reader/viewer: reader/JFJochReaderDataset.h + reader/JFJochHttpReader.cpp (GetPlot_i) and viewer/JFJochViewerDatasetInfo.cpp (combo item + ExtractMetric).
9. Frontend: frontend/src/components/DataProcessingPlots.tsx (MenuItem) + DataProcessingPlot.tsx (y-axis label).
10. Docs: docs/CBOR.md, docs/HDF5.md, docs/CPU_DATA_ANALYSIS.md.

Gotcha: an existing build/ dir needs a cmake . reconfigure to pick up a newly-added broker/gen source file (the source list is a configure-time glob).