Jungfraujoch/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this is

Jungfraujoch is the data-acquisition and analysis system for the PSI JUNGFRAU and EIGER
X-ray detectors. It receives detector data, runs it through an FPGA-accelerated pipeline
(spot finding, azimuthal/ROI integration, compression), streams images out over ZeroMQ for
writing to HDF5, and runs crystallographic analysis (indexing, integration, scaling/merging).
Most authoritative documentation lives in `docs/` and on Read The Docs
(https://jungfraujoch.readthedocs.io). When changing CLI behaviour, the program's own
usage message is the source of truth, not the docs.

## Build

Out-of-source CMake build, C++20, heavy use of `FetchContent` (spdlog, zstd, HDF5,
slsDetectorPackage, Catch2, cpp-httplib, libzmq, libtiff, FFTW, Ceres, fast-feedback-indexer, zlib-ng
and Eigen are downloaded and statically linked — the **first configure needs network access** and is
slow). **No library has to come from the host**: `zlib-devel`/`eigen3-devel` are no longer needed.
zlib (as zlib-ng in zlib-compat mode) is *built during the configure* into a prefix in the build tree
and found there via `ZLIB_ROOT`; Eigen's headers are fetched and a generated `Eigen3Config.cmake` in
the build tree is what every `find_package(Eigen3)` resolves — ours, Ceres' and ffbidx'. A copy
already on the machine is used instead if the configure is given `-DZLIB_ROOT=` / `-DEigen3_DIR=`.
Do **not** turn either of these into `FetchContent ... OVERRIDE_FIND_PACKAGE` — see the Eigen note in
`CMakeLists.txt` for the crash that rules it out.
libjpeg-turbo is built via `ExternalProject` in `preview/`; libcurl is fetched only for viewer builds.

```
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc) jfjoch_broker      # the main service; build other targets by name
```

Key CMake options:
- `JFJOCH_USE_CUDA` (default ON) — GPU path. Needs CUDA ≥ 12.8 (older is warned about and
  ignored). Provides the `ffbidx` and `fft` GPU indexers; without it only the CPU `fftw` indexer is
  available. FFTW is fetched and `JFJOCH_USE_FFTW` defined unconditionally, so `fftw` is always
  there. CUDA absence is not a build error — nvcc is looked for on `PATH`, `CUDA_PATH` and
  `/usr/local/cuda`.
- `JFJOCH_WRITER_ONLY` (default OFF) — builds only the HDF5 writer; skips broker, FPGA,
  receiver, analysis, tests, frontend.
- `JFJOCH_VIEWER_BUILD` (default OFF) — builds the Qt6 `jfjoch_viewer` desktop app in addition to
  the server stack.
- `JFJOCH_VIEWER_ONLY` (default OFF on Linux, **forced ON on Windows/macOS**) — builds only
  `jfjoch_viewer`, `rugnux` and the libraries they link; skips receiver, FPGA, detector control,
  tests and the frontend.
- `SLS9` (default OFF) — build against slsDetectorPackage 9.2.0 instead of 8.0.2.
- `JFJOCH_INSTALL_DRIVER_SOURCE` (default OFF) — install the PCIe driver source for DKMS/RPM.

### `-march` and LTO are deliberately NOT set in CMakeLists

The build system sets no architecture or LTO flags, so a site can pick its own — `x86-64-v4` on an
AVX-512 cluster, `-march=native`, or the plain baseline. **CI passes them explicitly**
(`.gitea/workflows/build_and_test.yml`): every Linux configure gets `LINUX_CMAKE_FLAGS`
(`-march=x86-64-v3 -flto=auto`), and the MSVC viewer job gets `/arch:AVX` (MSVC has no x86-64-v2
spelling; `/arch:AVX` is the nearest and implies SSE4.1/4.2). LTO is CI-only on purpose: measured on
rugnux it removes 7–10% of retired instructions and 9% of the binary but only ~1.5% of the wall
clock (the pipeline is GPU- and I/O-bound), while costing ~3× on an incremental rebuild — a bad
trade for a developer, a fine one for a build that happens once and is shipped.

**This matters when profiling.** A plain `cmake -DCMAKE_BUILD_TYPE=Release ..` produces a
*baseline* binary that is not what CI or production runs, and the difference is not uniform:
GPU-bound work is unaffected, but the CPU/Eigen-bound phases — first-pass indexing and
scaling/merging — are **~26% slower** without the flags (measured). Eigen in particular has no
vectorised `round` below SSE4.1 and falls back to a libm call per element, which can make
rounding look like ~10% of all cycles when it is nearly free in a real build. Pass the CI flags
when measuring anything CPU-side, or the profile will point at the wrong code:

```
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS="-march=x86-64-v3" \
      -DCMAKE_C_FLAGS="-march=x86-64-v3" ..
```

The frontend is a separate custom target: `make frontend` (in `frontend/`: `npm ci`,
`npm run build`, plus the third-party-licenses, Redoc and Sphinx-docs bundling steps). It is never
built automatically — `make install` only copies whatever already sits in `frontend/dist/`.

## Test

Tests use **Catch2** and are collected into a single binary `tests/jfjoch_test`.

```
make -j$(nproc) jfjoch_test
cd tests
./jfjoch_test "<test name>"         # one test case (exact name in TEST_CASE)
./jfjoch_test "Prefix*"             # by name prefix
./jfjoch_test "[tag]"               # by tag
./jfjoch_test -r junit -o report.xml
```

**Do not run the whole suite** (`./jfjoch_test` with no filter). It takes far too long to be worth
running locally, and **CI runs it on every push anyway**. Run only the cases or tags covering the
code you changed and stop there; a bare `./jfjoch_test` is justified only for a genuinely sweeping
change. The same applies to the validation battery (below) — run it when a change plausibly moves
merged results, not as a reflex.

### Battery

`tools/battery/battery.py` is the one canonical way to validate rugnux on the rotation datasets
(open arm: public PDB depositions vs the deposition; inhouse: standard crystals + no-crystal
controls vs XDS; private: local-only). `tools/battery/README.md` is the guide: data layout, site
config, run directory, results schema, report, protocol. Do not write or revive ad-hoc runner
scripts.

```
tools/battery/battery.py run --rugnux build/rugnux/rugnux --only 5reo,lyso_ref   # named sets
tools/battery/battery.py run --rugnux build/rugnux/rugnux --tier smoke           # smoke tier
tools/battery/battery.py compare RUN_A RUN_B [--rerun-changed]
```

Each set runs **once**, with one command per arm: `rugnux --model <deposited coordinates>` on the
open arm (without `--model` where there is none), and `rugnux [-A] --report-resolution <ref d_min>,<XDS
d_max>` on the XDS arms (`-A` where XDS was anomalous). Neither option changes the processing, so
the row carries both rugnux's own result (own cut vs the reference: what a user gets) and the
like-for-like numbers (the REFRES_* table vs XDS; where rugnux's cut is coarser than XDS's, its
completeness there is coverage, not quality). Open-arm space groups are scored on the data's own
determination, not on a model-adopted hand; rugnux's `--model` R-free is a placement-only trend
number. The REFMAC check is opt-in (`--model-check`).

- **Never start a full battery run (no `--only`/`--tier`) without the owner's go-ahead.**
- Judge a change against a **persisted baseline run** (the site config's `baseline`) with
  `compare`, never against numbers remembered from an earlier session.
- The private arm's names, paths and values never enter the repository or a public report.

`make jfjoch_hdf5_test` builds the HDF5 write-speed benchmark (it lives in `tools/`, so the binary
is `build/tools/jfjoch_hdf5_test`); CI also uses it to produce files that are validated against XDS
(Durin/Neggia), DIALS and CrystFEL. `jfjoch_hdf5_enospc_test` + the `enospc_shim` module test
out-of-space handling.

There is no lint config in the repo (no `.clang-tidy`/`.clang-format`, no CI lint step) — follow the
conventions the code already uses: namespaces lower_case, classes CamelCase, global constants
UPPER_CASE.

## Git

**Never `git push`.** Commit when asked; pushing is a separate decision that belongs to the
maintainer, and is done only when explicitly asked for.

**Never stage review reports, work plans or investigation notes.** They are working material for a
single task and go stale as soon as the code moves; `docs/review/` is gitignored for that reason,
and anything left under `docs/` would also be published by Sphinx. The same goes for generated
build artefacts — `common/GitInfo.cpp` is configured from `GitInfo.cpp.in` into the *binary* dir,
so a copy in the source tree is only ever the residue of an in-source configure.

**Stage by explicit path.** `git add -A` / `git add .` is how both of the above got committed by
accident; name the files the change actually touches, and read `git status` before committing.

## Changelog

`docs/CHANGELOG.md` is written **for users**. It does not replace commit messages or developer
notes, so keep out of it anything that belongs there instead:

- **One line per entry**, ideally one sentence. Two only when a user genuinely needs both.
- **Say what changed** — not why, not what the investigation found, not how the change was arrived
  at, not what it measured on. Rationale, measurements and history go in the commit message.
- **No sample identities**, exactly as everywhere else (see below).

## Design direction

Three standing preferences. They are **directional**, not absolute: they say which way to lean when a
design choice is otherwise balanced, and they are meant to be proposed against, not obeyed blindly.

**Spend cycles to buy quality.** rugnux is fast enough that raw speed is no longer the scarce
resource, so computation that a traditional pipeline would call wasteful is worth proposing when it
buys a stronger result - doing something twice, exploring a branch that will be discarded, measuring
what usually only confirms what we already assumed. Performance is still a goal; what has changed is
that it is now also a **budget**. Ask what a slower design buys, not merely whether it is slower.

**Prefer hypotheses to thresholds.** Crystallography traditionally decides early, at a threshold, and
passes a single answer down a linear pipeline. That has hurt this code repeatedly: a gate calibrated
on one population refuses another, and a decision taken before the evidence arrives cannot be
revisited. Where it is affordable, carry **several variants forward and decide late**, when there is
more to decide on - a branched flow with a late decision beats an early irreversible one. This is
what the compute budget above is for.

**Autodetect - the target is `rugnux <file>` with nothing else.** The goal is automated processing:
the best answer we can give must come out of a bare invocation, with no flags and no user guidance.
A capability that only works when someone already knows to switch it on has not solved the problem,
so treat a required parameter as a defect to design out rather than a feature to document.

## Code style

The overriding principle is **simple, readable code** — favour the smallest, most direct
implementation that a reader can verify at a glance. Extra abstraction, speculative guards, and
clever-but-dense constructs are treated as actively harmful, not as polish. When torn between a
tidy abstraction and a flat, obvious version, pick the obvious one.

This matters most in the experimental analysis code under `image_analysis/`, where readability is
how the physics gets verified — keep those parts especially plain. **Do not add
defensive/unrequested code** (extra validation, rejection heuristics, "just in case" branches)
without asking first; if a guard isn't clearly needed, leave it out.

Match the surrounding code's idiom, naming, and comment density rather than importing a different
style.

## No sample identities in the repository

**Never put sample names or sample-specific measured values into source code, comments,
documentation, commit messages, or test fixtures.** Datasets belong to users and may be
confidential or embargoed; anything committed can leak outside the group working on them. This is
a hard rule, not a preference.

- **Forbidden**: sample/dataset names or internal codes (a protein name, a beamline dataset ID, a
  run label), and **measured unit-cell parameters tied to a real sample** (this is the most
  sensitive — never hardcode "protein X has cell a,b,c").
- **Fine**: general crystallographic descriptors — space group / Laue class ("a P2₁ crystal", "a
  holohedral 422 case"), lattice centering, twinning, "a crystal whose true axis is reported as its
  3× harmonic", pseudo-symmetry, etc. Describe the *crystallographic situation*, not the specimen.
- **Exception — the standard test proteins.** Lysozyme (HEWL), thaumatin, insulin, cytochrome C
  and myoglobin are the field's standard reference specimens, not users' confidential datasets, so
  naming them and using their well-known reference cells (e.g. lysozyme tetragonal ~79/79/38,
  P4₃2₁2) in tests, docs, comments and the in-house battery manifest is allowed, and so are the
  XDS-derived reference values of our own in-house measurements of them. The same goes for the
  in-house no-crystal controls. This carve-out is only for these generic specimens as benchmarks; a
  *named user dataset* that happens to be one of them is still off limits, and every other real
  sample remains forbidden.
- **Exception — the open battery arm.** Its datasets are public PDB depositions (plus a few openly
  published small-molecule sets), so their PDB IDs and deposited space groups, cells and
  resolutions may be named in the battery manifest, reports and commits. The private battery arm
  is the opposite: its names, paths, cells and statistics never enter the repository.
- **Tests** must otherwise use neutral names (e.g. `tetragonal_uc`, not a specimen-named variable)
  and, where a cell is needed, a synthetic cell chosen for the test — not a real dataset's parameters.

When a bug was found on a specific dataset, commit the *behaviour* ("de-novo indexing adopted a
spurious axis-multiple supercell"), never the dataset.

## Licences and academic credit

Two different obligations, discharged in two different places; neither substitutes for the other.

- **Vendoring or linking someone's CODE** creates a **licence** obligation — `licenses/` and
  `THIRD_PARTY_NOTICES.md`.
- **Reimplementing an algorithm from a PAPER** creates no licence obligation at all, but it creates
  an obligation of **academic credit** — `docs/ACKNOWLEDGEMENT.md` and a comment at the algorithm.

Taking someone's source *and* following their paper incurs both. Do both.

**Licences.** Vendored code keeps the upstream licence file beside it (`compression/lz4/LICENSE`,
`gemmi_gph/LICENSE.txt`, …) and the adapted file's own header names the upstream URL and licence
(`image_analysis/spot_finding/StrongPixelSet.cpp` is the model). Then: add the path to
`licenses/COLLECT.sh` and run `bash licenses/COLLECT.sh`, which writes the verbatim text to
`licenses/<component>.txt` (committed, so a source checkout ships the notices without a build); and
add a row to the root `THIRD_PARTY_NOTICES.md` — component, path, copyright, SPDX id, link to the
licence text. Both are required; a licence text nobody lists is not attribution.
`docs/THIRD_PARTY_NOTICES.md` is **generated** from the root file by `update_version.sh` — never
hand-edit it. `CMakeLists.txt` installs `LICENSE`, `THIRD_PARTY_NOTICES.md` and all of `licenses/`
into `share/doc/jfjoch` for every package component, so nothing further is needed to reach the built
product.

A **build-time dependency** (`FetchContent`/`ExternalProject`, statically linked) is handled the
same way with one difference: there is no in-tree copy, so `COLLECT.sh` reads its licence out of the
populated `_deps/` tree — which means that collection step needs a configured build — and the row
goes in the "Fetched at build time" table instead. npm dependencies need nothing by hand: `npm run
licenses` regenerates `frontend/dist/THIRD_PARTY_LICENSES.txt` as part of `make frontend`.

**Acknowledgements.** The acknowledgement of an external *work* — a paper, a program, an approach
taken from someone else — goes in **`docs/ACKNOWLEDGEMENT.md`**: a short paragraph saying what was
taken and from whom, with the citation. If the method is also described in
`docs/CPU_DATA_ANALYSIS.md`, add the paper to that page's `## References` list as well.

**Citation form.** Verify every DOI. If you cannot verify one, write the reference without it and
say so — never invent a DOI.

- Paper — authors, "title" (year), journal, volume, pages, DOI as a link where one exists:
  ```
  P. R. Evans, "An introduction to data reduction: space-group determination, scaling and intensity
  statistics" (2011), Acta Cryst. D67, 282-292 [doi:10.1107/S090744491003982X](https://doi.org/10.1107/S090744491003982X).
  ```
- Software package — name and link, plus its canonical citation paper if it has one:
  ```
  [DIALS](https://dials.github.io/): G. Winter, D. G. Waterman, J. M. Parkhurst et al., "DIALS:
  implementation and evaluation of a new integration package" (2018), Acta Cryst. D74, 85-97
  [doi:10.1107/S2059798317017235](https://doi.org/10.1107/S2059798317017235).
  ```

**In-source credit** belongs **at the algorithm** — the function, the loop, or the setting it
governs — not in the file header, which carries only SPDX. One line: program and/or author, journal,
volume, pages, year. No DOI, no URL. The prevailing style:

```cpp
// Following Kabsch (2010) Acta Cryst. D66, 133-144
```

Check first and do not duplicate a credit an adjacent comment already carries.

## Local end-to-end run (no detector / no FPGA)

The FPGA HLS logic can be simulated on the CPU (`HLSSimulatedDevice`), so the full software
stack runs without hardware (slowly — fixed-point math on CPU). See
`docs/JFJOCH_BROKER.md` for the canonical walkthrough.

```
cd build/broker
./jfjoch_broker ../../etc/broker_local.json 5232      # config JSON + HTTP port
# then, separately:
cd tests/test_data && python jfjoch_broker_test.py     # feeds a test image, starts collection
# observe at http://localhost:5232 ; HDF5 is written under build/broker
```

`etc/broker_local.json`, `broker_eiger.json`, `broker_crmx.json` are example broker configs
(schema = `jfjoch_settings` in `broker/jfjoch_api.yaml`).

## Architecture

**Data flow (online):** detector → FPGA acquisition (`fpga/`, `acquisition_device/`) →
`receiver/` builds full images from per-module FPGA output → `image_pusher/` streams CBOR-encoded
images over ZeroMQ (or TCP) → the consuming side (`image_puller/`) feeds `jfjoch_writer`
(`writer/`), which writes NXmx HDF5. The broker also emits a low-rate preview stream and a
metadata stream (`preview/`).

**Writer file split:** one acquisition produces one `_master.h5` plus many `_data_NNNNNN.h5`
files. Dataset-wide metadata (geometry, detector config, ROI/azimuthal definitions — anything
fixed for the whole run) is written to the **master** file in `writer/HDF5NXmx.cpp` (the `NXmx`
class). Per-image arrays (one entry per frame) are written to the **data** files by the
`HDF5DataFilePlugin` subclasses in `writer/`. Put shared metadata in `NXmx`, not in a data-file
plugin.

The HDF5 master/data layout is one of three `FileWriterFormat`s (`common/JFJochMessages.h`),
all NXmx: **`NXmxLegacy`** (master + `_data_NNNNNN.h5` joined by external links),
**`NXmxVDS`** (master + data joined by HDF5 virtual datasets — the default), and
**`NXmxIntegrated`** (a single self-contained file, no separate data files). Per-image plugins
must work for all three; with `NXmxIntegrated` "master" and "data" are the same file. (The enum
also has non-NXmx `DataOnly` and `NoFile`; values 4/5 are retired CBF/TIFF, kept only in the
OpenAPI enum for back compatibility.)

**Two acquisition workflows:** the FPGA-accelerated path (JUNGFRAU at PSI; FPGA does masking,
summation, spot finding, ROI/azimuthal integration, compression) and the DECTRIS SIMPLON path
(EIGER), which has no FPGA — masking/ROI/azimuthal analysis then runs on CPU through the shared
`image_analysis/` library. Treat ROI and azimuthal features as available in **both** workflows,
not FPGA-only.

**`jfjoch_broker`** (`broker/`) is the central online service: HTTP/REST + OpenAPI control plane,
FPGA configuration, image building, ZeroMQ output. `JFJochStateMachine` drives acquisition state;
`JFJochServices` wires the pieces; `OpenAPIConvert`/`JFJochBrokerParser` translate between the
generated API model and internal types.

**Three analysis frontends share one analysis library** (`image_analysis/`, built as
`JFJochImageAnalysis`):
- `jfjoch_broker` — online, real-time (FPGA + GPU).
- `jfjoch_viewer` — interactive Qt desktop (`viewer/`), results not persisted.
- `rugnux` (`rugnux/rugnux_cli.cpp`, built on the `Rugnux` library in the same directory) — offline
  batch over a stored HDF5, invoked as `rugnux {<options>} <input.h5>` (it has **no `--help`**; run
  it with no arguments to print the usage, which is the authority on its flags). Rotation vs stills
  is auto-detected from the goniometer axis. Merging is on by default (`--no-merge` to disable);
  merging writes `.mtz`/`.cif`/`.hkl` and skips the bulky `_process.h5` unless `--write-process-h5`,
  while `--no-merge` writes only `_process.h5`. `--mode` picks what a run does: `mx` (the full
  pipeline, the default), `azint` (azimuthal integration only), `scale` (re-scale/merge the
  already-integrated reflections in a `_process.h5`) or `calibration` (detector geometry from a
  calibrant's powder rings, written as a `.poni`; `--calibrant`, `--calibration rings|spots`).
  (rugnux = the data-processing half of the system; see `docs/NAMING.md`.)

**`image_analysis/` pipeline** (subdirs): `spot_finding`, `indexing` (`ffbidx`/`fft` GPU,
`fftw` CPU), `lattice_search`, `geom_refinement`, `bragg_prediction`, `bragg_integration`,
`image_preprocessing`, `azint`, `roi`, `scale_merge`, plus `rotation_indexer/` and
`dark_mask_analysis/` and `beam_stop/` compiled straight into `JFJochImageAnalysis`
(`beam_stop/ShadowFinder` is built and runs by default in `rugnux` — it is no longer the unbuilt
prototype an older version of this file described). Least-squares refinement uses **Ceres** (fetched in
`image_analysis/CMakeLists.txt`, built with miniglog, no MKL, no Ceres-CUDA, CXX_THREADS).
The indexer is chosen with `-X` (`FFBIDX|FFT|FFTW|Auto|None`, default `Auto`, which resolves to a
GPU indexer when one is present and `fftw` otherwise): `ffbidx` wants a known cell (`-C`) and suits
sparse serial stills; `fft`/`fftw` index de novo and suit strong rotation data.

**FPGA** (`fpga/`): `hls/` is the Vitis HLS source (image-analysis kernels), `hls_simulation/`
runs that same HLS on CPU for hardware-free testing, `hdl/` is the Verilog RTL, `host_library/` is
the host-side driver, `pcie_driver/` is the kernel module. The HLS algorithms are documented in
`docs/FPGA_DATA_ANALYSIS.md`.

**Detector control** (`detector_control/`): wrappers for SLS (JUNGFRAU) and DECTRIS SIMPLON
(EIGER). **`jungfrau/`**: JUNGFRAU ADU→energy gain/pedestal calibration.

**Other libs:** `common/` (geometry, diffraction experiment, image buffer, CUDA wrappers — the
shared core, linked nearly everywhere), `compression/` (vendored bitshuffle + LZ4 + zstd; the
algorithms are `BSHUF_LZ4`, `BSHUF_ZSTD`, `BSHUF_ZSTD_RLE`, `BSHUF_ZSTD_RLE_HUFF`),
`frame_serialize/` (CBOR stream codec), `reader/` (HDF5 dataset read-back), `gemmi_gph/` (vendored
GEMMI for MTZ/XDS_ASCII I/O), `xds-plugin/` (XDS HDF5 read plugin).

## Portability (jfjoch_viewer)

Cross-platform support is a goal **only for `jfjoch_viewer` and its dependency tree** — keep that
code, and any shared library it transitively links (`common/`, `image_analysis/`, `reader/`,
`gemmi_gph/`, etc.), **MSVC-compatible** so the viewer can build on Windows. The rest of the
project (broker, receiver, FPGA host, detector control, …) is Linux-only and does not need to be
portable; don't constrain it for portability's sake.

- **Windows/MSVC** is the primary portability target. The end goal is a Windows viewer built
  with **MSVC *and* CUDA** (`JFJOCH_USE_CUDA=ON`): GPU processing is a wanted feature, not
  optional, so the intended Windows config is the full GPU path (`ffbidx`, GPU `fft`), not a
  CPU-only fallback. MSVC is required regardless, because CUDA on Windows requires it. Avoid
  GCC/Clang-only extensions, POSIX-only APIs, and other non-MSVC constructs in viewer-reachable
  code, and keep CUDA-reachable viewer code (`ffbidx`, GPU indexers) MSVC-buildable too.
- **macOS** is a nice-to-have for the viewer. It rules out CUDA, so anything the viewer depends on
  must also have a working CPU-only / non-CUDA path (the `JFJOCH_USE_CUDA=OFF`, `fftw`-indexer
  configuration). This non-CUDA path must keep working, but it is the macOS fallback — *not* the
  intended Windows configuration.

`JFJOCH_VIEWER_ONLY` is forced ON on Windows and macOS, so a plain configure there already builds
just the portable subset. libtiff, libjpeg-turbo (via `ExternalProject` in `preview/`) and libcurl
(viewer builds only) are now brought in by the build itself, so they need nothing from the host.
**ZLIB** and **Eigen** (header-only; needed by Ceres, by the analysis libs directly, and by `ffbidx`
under CUDA) are brought in by the build too, so on Windows only Qt still has to be installed first.
Both are still reached through an ordinary `find_package`, and that is the point: **vendoring Eigen
through `FetchContent` with `OVERRIDE_FIND_PACKAGE` remains specifically ruled out** — it segfaults
the CMake bundled with Visual Studio, ~1 in 3 fresh configures and 100% with Ceres CUDA on, in
CMake's own FetchContent variable-stack cleanup (see the long note in `CMakeLists.txt`). The same
ban applies to ZLIB, whose consumers (HDF5, libtiff, libcurl) call `find_package(ZLIB)` from inside
their own `FetchContent_MakeAvailable` — the exact faulting pattern. Instead each is made findable
the ordinary way before anything looks for it: zlib-ng is configured, built and installed into a
build-tree prefix during the configure and found through `ZLIB_ROOT`, and a generated
`Eigen3Config.cmake` in the build tree is pointed at by `Eigen3_DIR`.
Qt is supplied externally as before.

## OpenAPI is the single source of truth

`broker/jfjoch_api.yaml` defines the entire REST API **and the shared data schemas**. From it,
`update_version.sh` regenerates three clients — **do not hand-edit generated code**:
- C++ server model → `broker/gen/` (cpp-pistache-server generator; compiled as `JFJochAPI`).
- Python client → `python-client/` (and `gen_python_client.sh`, published as PyPI `jfjoch-client`).
- TypeScript frontend client → `frontend/src/client/` (hey-api `openapi-ts`, `npm run openapi`).

When you change `jfjoch_api.yaml`, regenerate the relevant client(s); for a version bump write the new version into `VERSION` and run
`update_version.sh` (which reads it and rewrites the `version:` in the YAML itself,
`frontend/src/version.ts`, `frontend/package.json`, `docs/conf.py`, the python client, the Redoc
html — and also the FPGA HDL and PCIe-driver version strings). It downloads
`openapi-generator-cli.jar` and runs `npm install`, so it needs network access.

## Frontend

React 19 + TypeScript + MUI 6 + Vite 7 (`frontend/`); charts are Plotly (`react-plotly.js`). Data
layer is generated from the OpenAPI spec (`@hey-api/openapi-ts` → fetch client + TanStack Query
hooks + zod schemas; config in `frontend/openapi-ts.config.ts`). Scripts:
`npm start` (dev server), `npm run build` (tsc + vite), `npm run openapi` (regen client),
`npm run redocly4broker` (regen `broker/redoc-static.html`).

## Adding a per-image scalar quantity

A per-image scalar (e.g. `ice_ring_score`, `bkg_estimate`, `mosaicity`) flows analysis → message → CBOR
→ HDF5 → scan-result/plot → API → viewer/frontend. To add one, mirror an existing float scalar
(`bkg_estimate` is a clean template) at every layer:

1. **Compute** where the azint profile is finalized: `image_analysis/MXAnalysisWithoutFPGA.cpp` (CPU),
   `receiver/JFJochReceiverFPGA.cpp` (FPGA), and the offline azint worker in `rugnux/Rugnux.cpp`.
2. **Message** (`common/JFJochMessages.h`): `std::optional<float> <name>` in `DataMessage`; in
   `EndMessage` **two** members — `std::vector<float> v_<name>` (the per-image array) and an
   `std::optional<float> <name>` run-mean scalar. Mind the `v_` prefix.
3. **CBOR**: encode in `frame_serialize/CBORStream2Serializer.cpp` — one key in the DataMessage block
   (`SerializeImageInternal`) *and* two in the END block (`SerializeSequenceEnd`: `<name>` and
   `v_<name>`); decode the same three in `CBORStream2Deserializer.cpp`. Optional fields are
   back-compatible — no version bump.
4. **HDF5** write: `writer/HDF5DataFilePluginMX.{h,cpp}` — an `AutoIncrVector<float>` with reserve /
   per-image write / `SaveVector("/entry/MX/<name>")` (per-image arrays live in the data-file plugin);
   plus the NXmx master writes in `writer/HDF5NXmx.cpp` (`SaveVectorIfMissing(..., end.v_<name>)` and
   a `SaveScalar(... "Mean", end.<name>)`). **HDF5 dataset names are camelCase even though the message
   fields are snake_case** — `bkg_estimate` is stored as `/entry/MX/bkgEstimate` (+ `bkgEstimateMean`).
   HDF5 **read-back** (so a stored file re-opens, e.g. in the viewer) is in
   `reader/HDF5MetadataSource.cpp`, NOT `JFJochHDF5Reader.cpp`: mirror the three `bkgEstimate` sites —
   master `ReadOptVector`, data-file `ReadVector` into the dataset, and the per-image message population.
5. **Scan result**: `common/ScanResult.h` (`ScanResultElem`) + `common/ScanResultGenerator.cpp`
   (copy in `Add`, resize+fill in `FillEndMessage`). The name is **not** preserved here — the
   `bkg_estimate` member is called just `bkg`, and so is the API property in step 7.
6. **Receiver plot**: `common/Plot.h` (`PlotType`) + `common/JFJochReceiverPlots.{h,cpp}` — a
   `StatusVector` member, cleared in `Setup`, fed in `Add`, plus the `GetPlots` / `GetPlotRaw` cases
   and (if the run-mean scalar of step 2 is wanted) a `GetBkgEstimate`-style accessor.
7. **API**: add to the `plot_type` enum and the `scan_result` images schema in `broker/jfjoch_api.yaml`,
   regenerate the C++ model (`java -jar openapi-generator-cli.jar generate -i broker/jfjoch_api.yaml -o
   broker/gen -g cpp-pistache-server`) and the frontend client (`cd frontend && npm run openapi`), then
   wire `broker/OpenAPIConvert.cpp` (`ConvertPlotType` string→enum and the `Convert(ScanResult)` setter).
8. **Reader/viewer**: `reader/JFJochReaderDataset.h` + `viewer/JFJochHttpReader.cpp` (`GetPlot_i`) and
   `viewer/JFJochViewerDatasetInfo.cpp` (combo item + `ExtractMetric`).
9. **Frontend**: `frontend/src/components/DataProcessingPlots.tsx` (`MenuItem`) + `DataProcessingPlot.tsx`
   in the same directory (y-axis label in `AxisTypeY`).
10. **Docs**: `docs/CBOR.md` and `docs/HDF5.md` name the fields literally; `docs/CPU_DATA_ANALYSIS.md`
    describes the quantity in prose.

Gotcha: an existing `build/` dir needs a `cmake .` reconfigure to pick up a newly-added
`broker/gen/model` source file — `broker/CMakeLists.txt` collects it with `AUX_SOURCE_DIRECTORY`,
a configure-time directory scan. (`gen/api` is include-path-only and needs no reconfigure.)