Jungfraujoch/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

Jungfraujoch is the data-acquisition and analysis system for the PSI JUNGFRAU and EIGER
X-ray detectors. It receives detector data, runs it through an FPGA-accelerated pipeline
(spot finding, azimuthal/ROI integration, compression), streams images out over ZeroMQ for
writing to HDF5, and runs crystallographic analysis (indexing, integration, scaling/merging).
Most authoritative documentation lives in `docs/` and on Read The Docs
(https://jungfraujoch.readthedocs.io). When changing CLI behaviour, the program's own
usage message is the source of truth, not the docs.

## Build

Out-of-source CMake build, C++20, heavy use of `FetchContent` (spdlog, zstd, HDF5,
slsDetectorPackage, Catch2, cpp-httplib, libzmq, Ceres, fast-feedback-indexer are downloaded
and statically linked — the **first configure needs network access** and is slow).

```
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc) jfjoch_broker      # the main service; build other targets by name
```

Key CMake options:
- `JFJOCH_USE_CUDA` (default ON) — GPU path. Needs CUDA ≥ 12.8. Provides the `ffbidx` and `fft`
  GPU indexers; without it only the CPU `fftw` indexer is available (requires FFTW at configure
  time, auto-detected). CUDA absence is not a build error.
- `JFJOCH_WRITER_ONLY` (default OFF) — builds only the HDF5 writer; skips broker, FPGA,
  receiver, analysis, tests, frontend.
- `JFJOCH_VIEWER_BUILD` (default OFF) — builds the Qt6 `jfjoch_viewer` desktop app.
- `SLS9` (default OFF) — build against slsDetectorPackage 9.2.0 instead of 8.0.2.

The frontend is a separate custom target: `make frontend` (runs `npm ci && npm run build` in
`frontend/`). It is not built by default unless installing.

## Test

Tests use **Catch2** and are collected into a single binary `tests/jfjoch_test`.

```
make -j$(nproc) jfjoch_test
cd tests
./jfjoch_test                       # all tests
./jfjoch_test "<test name>"         # one test case (exact name in TEST_CASE)
./jfjoch_test "[tag]"               # by tag
./jfjoch_test -r junit -o report.xml
```

`make jfjoch_hdf5_test` builds the HDF5 write-speed benchmark, also used by CI to produce files
that are validated against XDS (Durin/Neggia) and CrystFEL.

Lint config is `.clang-tidy` (broad `*` check set with many exceptions; namespaces lower_case,
classes CamelCase, global constants UPPER_CASE).

## Code style

The overriding principle is **simple, readable code** — favour the smallest, most direct
implementation that a reader can verify at a glance. Extra abstraction, speculative guards, and
clever-but-dense constructs are treated as actively harmful, not as polish. When torn between a
tidy abstraction and a flat, obvious version, pick the obvious one.

This matters most in `image_analysis/pixel_refinement/` (e.g. `PixelRefine.cpp`), which is an
experimental prototype where readability is how the physics gets verified — keep it especially
plain. **Do not add defensive/unrequested code** (extra validation, rejection heuristics, "just in
case" branches) without asking first; if a guard isn't clearly needed, leave it out.

Match the surrounding code's idiom, naming, and comment density rather than importing a different
style.

## No sample identities in the repository

**Never put sample names or sample-specific measured values into source code, comments,
documentation, commit messages, or test fixtures.** Datasets belong to users and may be
confidential or embargoed; anything committed can leak outside the group working on them. This is
a hard rule, not a preference.

- **Forbidden**: sample/dataset names or internal codes (a protein name, a beamline dataset ID, a
  run label), and **measured unit-cell parameters tied to a real sample** (this is the most
  sensitive — never hardcode "protein X has cell a,b,c").
- **Fine**: general crystallographic descriptors — space group / Laue class ("a P2₁ crystal", "a
  holohedral 422 case"), lattice centering, twinning, "a crystal whose true axis is reported as its
  3× harmonic", pseudo-symmetry, etc. Describe the *crystallographic situation*, not the specimen.
- **Exception — plain lysozyme.** Lysozyme (HEWL) is the field's universal standard test specimen,
  not a user's confidential dataset, so naming it and using its well-known reference cell
  (tetragonal ~79/79/38, P4₃2₁2) in tests, docs and comments is allowed. This carve-out is only for
  generic lysozyme as a benchmark; a *named user dataset* that happens to be lysozyme is still off
  limits, and every other real sample remains forbidden.
- **Tests** must otherwise use neutral names (e.g. `tetragonal_uc`, not a specimen-named variable)
  and, where a cell is needed, a synthetic cell chosen for the test — not a real dataset's parameters.

When a bug was found on a specific dataset, commit the *behaviour* ("de-novo indexing adopted a
spurious axis-multiple supercell"), never the dataset.

## Local end-to-end run (no detector / no FPGA)

The FPGA HLS logic can be simulated on the CPU (`HLSSimulatedDevice`), so the full software
stack runs without hardware (slowly — fixed-point math on CPU). See
`docs/JFJOCH_BROKER.md` for the canonical walkthrough.

```
cd build/broker
./jfjoch_broker ../../etc/broker_local.json 5232      # config JSON + HTTP port
# then, separately:
cd tests/test_data && python jfjoch_broker_test.py     # feeds a test image, starts collection
# observe at http://localhost:5232 ; HDF5 is written under build/broker
```

`etc/broker_local.json`, `broker_eiger.json`, `broker_crmx.json` are example broker configs
(schema = `jfjoch_settings` in `broker/jfjoch_api.yaml`).

## Architecture

**Data flow (online):** detector → FPGA acquisition (`fpga/`, `acquisition_device/`) →
`receiver/` builds full images from per-module FPGA output → `image_pusher/` streams CBOR-encoded
images over ZeroMQ → `jfjoch_writer` (`writer/`) consumes the stream and writes NXmx HDF5. The
broker also emits a low-rate preview stream and a metadata stream (`preview/`).

**Writer file split:** one acquisition produces one `_master.h5` plus many `_data_NNNNNN.h5`
files. Dataset-wide metadata (geometry, detector config, ROI/azimuthal definitions — anything
fixed for the whole run) is written to the **master** file in `writer/HDF5NXmx.cpp` (the `NXmx`
class). Per-image arrays (one entry per frame) are written to the **data** files by the
`HDF5DataFilePlugin` subclasses in `writer/`. Put shared metadata in `NXmx`, not in a data-file
plugin.

The HDF5 master/data layout is one of three `FileWriterFormat`s (`common/JFJochMessages.h`),
all NXmx: **`NXmxLegacy`** (master + `_data_NNNNNN.h5` joined by external links),
**`NXmxVDS`** (master + data joined by HDF5 virtual datasets — the default), and
**`NXmxIntegrated`** (a single self-contained file, no separate data files). Per-image plugins
must work for all three; with `NXmxIntegrated` "master" and "data" are the same file.

**Two acquisition workflows:** the FPGA-accelerated path (JUNGFRAU at PSI; FPGA does masking,
summation, spot finding, ROI/azimuthal integration, compression) and the DECTRIS SIMPLON path
(EIGER), which has no FPGA — masking/ROI/azimuthal analysis then runs on CPU through the shared
`image_analysis/` library. Treat ROI and azimuthal features as available in **both** workflows,
not FPGA-only.

**`jfjoch_broker`** (`broker/`) is the central online service: HTTP/REST + OpenAPI control plane,
FPGA configuration, image building, ZeroMQ output. `JFJochStateMachine` drives acquisition state;
`JFJochServices` wires the pieces; `OpenAPIConvert`/`JFJochBrokerParser` translate between the
generated API model and internal types.

**Three analysis frontends share one analysis library** (`image_analysis/`, built as
`JFJochImageAnalysis`):
- `jfjoch_broker` — online, real-time (FPGA + GPU).
- `jfjoch_viewer` — interactive Qt desktop (`viewer/`), results not persisted.
- `rugnux` (`tools/rugnux_cli.cpp`, built on the `Rugnux` library in `rugnux/`) — offline batch over
  a stored HDF5; writes `_process.h5` and `.mtz`/`.cif`/`.hkl`. Merging is on by default (`--no-merge`
  to disable); `--azint-only` runs only azimuthal integration and `--scale` re-scales/merges the
  already-integrated reflections in a `_process.h5`. (rugnux = the data-processing half of the system;
  see `docs/NAMING.md`.)

**`image_analysis/` pipeline** (subdirs): `spot_finding`, `indexing` (`ffbidx`/`fft` GPU,
`fftw` CPU), `lattice_search`, `geom_refinement`, `pixel_refinement`, `bragg_prediction`,
`bragg_integration`, `rotation_indexer`, `azint`, `roi`, `scale_merge`. Least-squares refinement
uses **Ceres** (fetched, built with miniglog, no MKL, CXX_THREADS). `ffbidx` needs a known cell
(`-C`) and suits sparse serial stills; `fft`/`fftw` index de novo and suit strong rotation data.

**FPGA** (`fpga/`): `hls/` is the Vitis HLS source (image-analysis kernels), `hls_simulation/`
runs that same HLS on CPU for hardware-free testing, `host_library/` is the host-side driver,
`pcie_driver/` is the kernel module. The HLS algorithms are documented in
`docs/FPGA_DATA_ANALYSIS.md`.

**Detector control** (`detector_control/`): wrappers for SLS (JUNGFRAU) and DECTRIS SIMPLON
(EIGER). **`jungfrau/`**: JUNGFRAU ADU→energy gain/pedestal calibration.

**Other libs:** `common/` (geometry, diffraction experiment, image buffer, CUDA wrappers — the
shared core, linked nearly everywhere), `compression/` (zstd + bitshuffle + sqrt lossy),
`frame_serialize/` (CBOR stream codec), `gemmi_gph/` (vendored GEMMI for MTZ/XDS_ASCII I/O),
`xds-plugin/` (XDS HDF5 read plugin).

## Portability (jfjoch_viewer)

Cross-platform support is a goal **only for `jfjoch_viewer` and its dependency tree** — keep that
code, and any shared library it transitively links (`common/`, `image_analysis/`, `reader/`,
`gemmi_gph/`, etc.), **MSVC-compatible** so the viewer can build on Windows. The rest of the
project (broker, receiver, FPGA host, detector control, …) is Linux-only and does not need to be
portable; don't constrain it for portability's sake.

- **Windows/MSVC** is the primary portability target. The end goal is a Windows viewer built
  with **MSVC *and* CUDA** (`JFJOCH_USE_CUDA=ON`): GPU processing is a wanted feature, not
  optional, so the intended Windows config is the full GPU path (`ffbidx`, GPU `fft`), not a
  CPU-only fallback. MSVC is required regardless, because CUDA on Windows requires it. Avoid
  GCC/Clang-only extensions, POSIX-only APIs, and other non-MSVC constructs in viewer-reachable
  code, and keep CUDA-reachable viewer code (`ffbidx`, GPU indexers) MSVC-buildable too.
- **macOS** is a nice-to-have for the viewer. It rules out CUDA, so anything the viewer depends on
  must also have a working CPU-only / non-CUDA path (the `JFJOCH_USE_CUDA=OFF`, `fftw`-indexer
  configuration). This non-CUDA path must keep working, but it is the macOS fallback — *not* the
  intended Windows configuration.

A self-contained Windows build still needs a few dependencies that the Linux build picks up from
the system and that are **not** auto-provided via `FetchContent` (besides Qt, supplied
externally): **ZLIB** (pulled by hdf5/libtiff/gemmi/libzmq — a bundled zlib-ng in `ZLIB_COMPAT`
mode is the intended fix, presented as the `ZLIB::ZLIB` target so the scattered
`find_package(ZLIB)` calls resolve), **libjpeg-turbo** (used by `preview/`; upstream discourages
`add_subdirectory`, so bring it in via `ExternalProject`), and **Eigen** (header-only; needed by
Ceres, by the analysis libs directly, and by `ffbidx` under CUDA — fetch it and point Ceres'
internal `find_package(Eigen3)` at it). Wire each guarded so the Linux build, which finds these on
the system, stays unchanged.

## OpenAPI is the single source of truth

`broker/jfjoch_api.yaml` defines the entire REST API **and the shared data schemas**. From it,
`update_version.sh` regenerates three clients — **do not hand-edit generated code**:
- C++ server model → `broker/gen/` (cpp-pistache-server generator; compiled as `JFJochAPI`).
- Python client → `python-client/` (and `gen_python_client.sh`, published as PyPI `jfjoch-client`).
- TypeScript frontend client → `frontend/src/client/` (hey-api `openapi-ts`, `npm run openapi`).

When you change `jfjoch_api.yaml`, regenerate the relevant client(s); for a version bump run
`update_version.sh` (also rewrites `VERSION`, `frontend/src/version.ts`, and the Redoc html).

## Frontend

React 19 + TypeScript + MUI + Vite (`frontend/`). Data layer is generated from the OpenAPI spec
(`@hey-api/openapi-ts` → fetch client + TanStack Query hooks + zod schemas). Scripts:
`npm start` (dev server), `npm run build` (tsc + vite), `npm run openapi` (regen client),
`npm run redocly4broker` (regen `broker/redoc-static.html`).

## Adding a per-image scalar quantity

A per-image scalar (e.g. `ice_ring_score`, `bkg_estimate`, `mosaicity`) flows analysis → message → CBOR
→ HDF5 → scan-result/plot → API → viewer/frontend. To add one, mirror an existing float scalar
(`bkg_estimate` is a clean template) at every layer:

1. **Compute** where the azint profile is finalized: `image_analysis/MXAnalysisWithoutFPGA.cpp` (CPU),
   `receiver/JFJochReceiverFPGA.cpp` (FPGA), and the offline azint worker in `rugnux/Rugnux.cpp`.
2. **Message** (`common/JFJochMessages.h`): `std::optional<float>` in `DataMessage`, `std::vector<float>`
   in `EndMessage`.
3. **CBOR**: encode in `frame_serialize/CBORStream2Serializer.cpp` (DataMessage block *and* END block),
   decode in `CBORStream2Deserializer.cpp` (both). Optional fields are back-compatible — no version bump.
4. **HDF5** write: `writer/HDF5DataFilePluginMX.{h,cpp}` — an `AutoIncrVector<float>` with reserve /
   per-image write / `SaveVector("/entry/MX/<name>")` (per-image arrays live in the data-file plugin);
   plus the NXmx master write in `writer/HDF5NXmx.cpp` (`SaveVectorIfMissing(..., end.<name>)`). HDF5
   **read-back** (so a stored file re-opens, e.g. in the viewer) is in `reader/HDF5MetadataSource.cpp`,
   NOT `JFJochHDF5Reader.cpp`: mirror the three `bkgEstimate` sites — master `ReadOptVector`, data-file
   `ReadVector` into the dataset, and the per-image message population.
5. **Scan result**: `common/ScanResult.h` (`ScanResultElem`) + `common/ScanResultGenerator.cpp`
   (copy in `Add`, resize+fill in `FillEndMessage`).
6. **Receiver plot**: `common/Plot.h` (`PlotType`) + `common/JFJochReceiverPlots.{h,cpp}` (`StatusVector`
   + Clear / AddElement / GetPlots / GetPlotRaw cases).
7. **API**: add to the `plot_type` enum and the `scan_result` images schema in `broker/jfjoch_api.yaml`,
   regenerate the C++ model (`java -jar openapi-generator-cli.jar generate -i broker/jfjoch_api.yaml -o
   broker/gen -g cpp-pistache-server`) and the frontend client (`cd frontend && npm run openapi`), then
   wire `broker/OpenAPIConvert.cpp` (`ConvertPlotType` string→enum and the `Convert(ScanResult)` setter).
8. **Reader/viewer**: `reader/JFJochReaderDataset.h` + `reader/JFJochHttpReader.cpp` (`GetPlot_i`) and
   `viewer/JFJochViewerDatasetInfo.cpp` (combo item + `ExtractMetric`).
9. **Frontend**: `frontend/src/components/DataProcessingPlots.tsx` (`MenuItem`) + `DataProcessingPlot.tsx`
   (y-axis label).
10. **Docs**: `docs/CBOR.md`, `docs/HDF5.md`, `docs/CPU_DATA_ANALYSIS.md`.

Gotcha: an existing `build/` dir needs a `cmake .` reconfigure to pick up a newly-added `broker/gen`
source file (the source list is a configure-time glob).