# Repository guidelines ## Scope and project goals These instructions apply to the entire repository unless a more specific `AGENTS.md` exists in a subdirectory. Aare is a data-analysis library for PSI hybrid detectors. The C++17 core is the canonical implementation, while Python is the main user-facing interface through pybind11. Changes should preserve the project's priorities: fast and efficient processing, simple interfaces, API stability, and a small dependency footprint. ## Repository layout - `include/aare/` contains the public C++ API and header-defined templates. - `src/` contains C++ implementations and colocated Catch2 tests named `*.test.cpp`, grouped into `src/fit/`, `src/hist/` and `src/utils/` where a component has several private files. - `tests/` contains the C++ test executable, configuration, and shared helpers. - `python/src/` contains pybind11 bindings and module registration. - `python/aare/` contains Python facades, convenience APIs, and public exports. - `python/tests/` contains the pytest suite. - `docs/src/` contains Sphinx/reStructuredText documentation. Doxygen and Breathe integrate the C++ API into the Sphinx site. - `benchmarks/` contains Google Benchmark programs. - `cmake/`, `conda-recipe/`, and `pyproject.toml` support builds and packaging. - `RELEASE.md` contains pending and published release notes. ## Build environment The project requires CMake 3.15 or newer, C++17 with compiler extensions disabled, and Python 3.11 or newer. `etc/dev-env.yml` defines the Conda development environment. CMake fetches several dependencies by default. Use `-DAARE_SYSTEM_LIBRARIES=ON` only when all required system or Conda packages are available. Use this configuration for normal development: ```bash cmake -S . -B build \ -DCMAKE_BUILD_TYPE=Debug \ -DAARE_TESTS=ON \ -DAARE_PYTHON_BINDINGS=ON cmake --build build -j4 ``` Useful optional settings include `AARE_DOCS`, `AARE_BENCHMARKS`, `AARE_ASAN`, `AARE_WARNINGS_AS_ERRORS`, `AARE_VERBOSE` (raises the compile-time log level), and `AARE_TUNE_LOCAL` (`-march=native`, not portable). Reconfigure an existing build directory instead of creating alternate in-tree build layouts unless isolation is needed; `build/CMakeCache.txt` records the options and interpreter it was configured with. `-Werror=return-type` is always enabled. New code should also compile cleanly under the project's `-Wall -Wextra -pedantic -Wshadow -Wold-style-cast -Wdouble-promotion` flags, which `AARE_WARNINGS_AS_ERRORS` turns into errors. The build copies `python/aare/*.py` and the compiled `_aare` extension into `build/aare/`, so `PYTHONPATH=build` makes the package importable. The extension only imports into the interpreter CMake was configured with, which is normally the Conda or venv Python active at configure time. Activate that environment, or resolve the interpreter from the cache: ```bash PY=$(grep -E '^_?Python_EXECUTABLE' build/CMakeCache.txt | head -1 | cut -d= -f2) PYTHONPATH="$PWD/build" "$PY" -m pytest python/tests ``` Do not fall back to a different system Python. With `AARE_BENCHMARKS=ON`, Google Benchmark programs build as `build/run_benchmarks` and `build/fit_benchmark`. ## Architecture ### Core containers - `NDArray` owns contiguous C-order memory; `NDView` is a non-owning view with the same shape and stride API. Both derive from `ArrayExpr` (expression templates for element-wise math). Shapes are `std::array`. `NDView` converts to `NDView`. - `Frame` is a byte buffer plus rows, cols, and `Dtype`, the lowest common denominator returned by file readers. `Dtype` is a runtime type tag built from a bitdepth, a NumPy descriptor, or `typeid`. - `defs.hpp` holds detector enums (`DetectorType`, `TimingMode`, `FrameDiscardPolicy`, `ReadoutMode`), the on-disk `DetectorHeader`, chip constants (`Moench04`, `Matterhorn02`, ...), the `LOCATION` macro used in error messages, and `AARE_ASSERT` (active only with `AARE_CUSTOM_ASSERT`). - `logger.hpp` provides `LOG(logLEVEL) << ...`; levels above `AARE_LOG_LEVEL` (set by CMake) are compiled out. ### File reading `File` is the generic RAII facade. It holds a `unique_ptr` and dispatches on file extension to one of the concrete readers: - `RawFile` reads slsDetector `.raw` data with a `_master_N.json` (or legacy `.raw`) master file. It owns a `RawMasterFile` (master JSON, older raw-text masters, scan parameters, ROIs, disabled UDP ports) and a two-dimensional table of `RawSubFile` indexed `[roi][module]`. Each `RawSubFile` handles one module's sequence of `_dN_fM_` data files and per-frame `DetectorHeader`s. Module placement into the assembled image goes through `DetectorGeometry`, `ModuleGeometry`, and `ROIGeometry` (module gaps, ROIs, UDP-port halves). Frame numbers from the detector are distinct from frame indices in the file; the API uses both and names them accordingly. - `NumpyFile` (`.npy`), `JungfrauDataFile` (legacy `.dat`), and `CtbRawFile` (chip test board; raw ADC samples decoded through `decode.hpp` and the `PixelMap` generators, exposed in Python as `aare.transform` callables). - `experimental::MultiThreadedFileReader` opens one `File` per worker and fills a caller-owned buffer in file order. Some chip-specific decoding (Moench03/05 pixel maps, ADC SAR transforms) lives in `python/aare/transform.py` on top of C++ generators. ### Pedestal and cluster finding - `Pedestal` and `FastPedestal` maintain per-pixel running mean and standard deviation, always accumulating in `double`. `FastPedestal` must see `min_pedestal_samples` frames before `ready()`. - `ClusterFinder` is single-threaded: push pedestal frames, then `find_clusters(view, frame_number)` appends to an internal `ClusterVector` that callers take with `steal_clusters()`. Thresholds are `n_sigma * std` and are recomputed by `update_threshold()`. - `ClusterFinderMT` wraps N `ClusterFinder`s, one per thread, each with a `CircularFifo` input queue backed by a preallocated `FramePool` and a `ProducerConsumerQueue` output. A collector thread funnels all outputs into a single sink queue. Consumers of that sink run their own thread: `ClusterCollector` (accumulate in memory, `steal_clusters()`) or `ClusterFileSink` (write the binary cluster file). The `stop()` and `sync()` ordering matters; read the existing tests before changing lifecycle code. - `Cluster` is a POD (x, y, data[]) and `ClusterVector>` is a frame-tagged `std::vector` of them. `ClusterFile` reads and writes the binary format (`int32 frame_number`, `uint32 n_clusters`, raw cluster bytes). The format carries no shape or dtype metadata, so the caller must supply matching template arguments. `CalculateEta.hpp` and `Interpolator` operate on `ClusterVector`s; `algorithm.hpp` holds small 1-D search helpers. - `hist/PixelHistogram` and `PedestalTrackingPixelHistogram` build per-pixel histograms across a frame series, multithreaded by row bands. ### Fitting `Fit.hpp`, `FitModel`, and `Models.hpp` (`Gaussian`, `Pol1`, `Pol2`, `RisingScurve`, `FallingScurve`, `GaussianErfcPlateau`, `GaussianChargeSharing`, ...) form the public fitting API. Each model provides `eval`, `eval_and_grad`, `is_valid`, `estimate_par`, and `param_info`; the separable models also declare their linear parameters (`linear_par`, `basis_and_grad`, `basis_columns`). `FitModel` is plain data (limits, fixed flags, start values, minimizer settings) and selects one of four minimizers through `aare::Minimizer`: Minuit2's Migrad (default) and Fumili, or the built-in Levenberg-Marquardt and variable projection solvers. The private implementation lives in `src/fit/`: `Fit.cpp` holds the `FitModel` method bodies, the per-thread `detail::PixelFitter` that dispatches on the minimizer and falls back from VarPro to Levenberg-Marquardt, `fit_pixel`/`fit_3d`, and the explicit instantiations; `FitHelpers.hpp` holds the start-value precedence and shape checks; `DampedGaussNewton.hpp` is the damped Gauss-Newton driver shared by `LevenbergMarquardt.hpp` and `VariableProjection.hpp`, with the small fixed-size linear algebra on `std::array` (`Vec`, `Mat`). `FitMinuit2.cpp` (with `Chi2.hpp` and `MinuitSteps.hpp`) is the only translation unit that includes Minuit2 headers; Minuit2 is a private, `BUILD_INTERFACE`-only dependency of `aare_core`. The `[fit]` tests run every configuration through all four minimizers and check that they agree; run them when changing any solver. ### Python layer - `python/src/module.cpp` registers everything. Templated classes are instantiated per value type and cluster size through the `DEFINE_CLUSTER_BINDINGS` and `DEFINE_BINDINGS_CLUSTERFINDER` macros, producing names such as `ClusterFinder_Cluster3x3i`, `Pedestal_d`, and `FastPedestal_i16`. Supported cluster sizes are 2x2, 3x3, 5x5, 7x7, and 9x9; value types are `i`, `f`, `d`, and `i16` (3x3 only). `module_config.hpp` fixes the module-wide pedestal type (`pd_type`). - `python/aare/factory.py` maps NumPy dtypes to those suffixes. The facades in `python/aare/*.py` are thin factory functions or subclasses that select the right binding and add Pythonic conveniences (iteration, context managers, `ScanParameters`). - `np_helper.hpp` provides `return_image_data` (moves an `NDArray` into a NumPy array with a capsule deleter) and `make_view_1d/2d/3d` (views into a NumPy buffer). - `aare.experimental` and `_aare.experimental` hold unstable APIs. ## Implementation conventions - Put public declarations in `include/aare/` and private implementation details in `src/`. - Add new compiled headers, sources, and C++ tests to the explicit lists in the root `CMakeLists.txt`. - Keep template implementations in headers unless the supported types are explicitly instantiated. - Follow the naming in adjacent code. Broadly, use the `aare` namespace, CamelCase types, and snake_case functions. - Treat ownership, lifetime, const-correctness, array shapes, and buffer contiguity as part of the API when working with `NDArray`, `NDView`, or NumPy bindings. - Avoid unnecessary allocations and copies in detector-data and per-pixel processing paths. Add or update a benchmark when performance is central to a change. - Place APIs that may change without notice under the existing experimental namespace/module. - Start new source files with `SPDX-License-Identifier: MPL-2.0`, using the appropriate comment syntax. - Code copied from other projects keeps its original license. Retain its copyright and license notice, set the SPDX identifier to that license, and list the license in `LICENSES/`, `pyproject.toml`, `conda-recipe/meta.yaml`, and `README.md`. `etc/add_license.py` skips files with a license notice. - When adding or bumping a fetched dependency that is compiled into the Python extension, update its version and notice in `THIRD-PARTY-NOTICES.txt` and the license expressions in `pyproject.toml` and `conda-recipe/meta.yaml`. - Prefer descriptive names to comments that only restate the code. - Do not add a dependency unless the benefit justifies the packaging and deployment cost. ## Python-facing changes A Python-facing feature can require coordinated changes in several layers: 1. Update the C++ public API and implementation. 2. Add or update its binding in `python/src/`. 3. Register new bindings in `python/src/module.cpp`. 4. Update the facade or public exports in `python/aare/`. 5. If adding a Python module, add it to `PYTHON_FILES` in `python/CMakeLists.txt` so it is copied and installed. 6. Add Python tests and update user documentation. Bindings must validate NumPy dimensions and data types before constructing views. Do not return a view whose backing C++ or Python storage can expire while the view remains reachable. ## Tests Run tests that do not require external detector data with: ```bash ctest --test-dir build --output-on-failure -j4 PYTHONPATH="$PWD/build" python -m pytest python/tests ``` For focused runs, use a Catch2 tag or test name, or an individual pytest file or test: ```bash build/run_tests "[tag]" build/run_tests "Exact test case name" PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py -k name ``` Large detector test files live outside this repository in `https://gitea.psi.ch/detectors/aare-test-data.git` (Git LFS). To include data-backed tests, set `AARE_TEST_DATA` and opt in explicitly: ```bash export AARE_TEST_DATA=/path/to/aare-test-data build/run_tests "[.with-data]" PYTHONPATH="$PWD/build" python -m pytest python/tests --with-data ``` - Start bug fixes with a failing regression test when practical. - Put C++ tests beside the relevant implementation as `src/Thing.test.cpp` and add them to `TestSources` in the root `CMakeLists.txt`. - Mark C++ tests requiring external files with `[.with-data]`. - Mark Python tests requiring external files with `@pytest.mark.withdata` and use the `test_data_path` fixture from `python/tests/conftest.py`. - Prefer synthesizing small input files over depending on external data. `tests/raw_file_helpers.hpp` builds throwaway raw files in C++, and many Python tests write small raw files into `tmp_path`. - `tests/test_config.hpp.in` provides `test_data_path()`; `tests/test_macros.hpp` and `tests/friend_test.hpp` (`TEST_CASE_PRIVATE` with `FRIEND_TEST`) let a test reach private members. - Run both suites for changes that cross the C++/Python boundary. - Do not silently skip required data-backed coverage. Report when the external test data is unavailable. ## Formatting and static analysis The project uses pre-commit. Install the git hook once per clone with `pre-commit install` so it runs on every commit. The hooks in `.pre-commit-config.yaml` are clang-format on C++, cmake-format on CMake files, `typos` on `docs/` (project words are whitelisted in `typos.toml`), and basic YAML, TOML, JSON, and large-file checks. CI runs the same hooks on pull requests. Before handing off a broad change, run the relevant checks: ```bash pre-commit install # once per clone pre-commit run --all-files # what CI runs cmake --build build --target check-format # diff-based clang-format check cmake --build build --target format-files # apply clang-format to all .cpp/.hpp cmake --build build --target clang-tidy # uses build/compile_commands.json ``` C++ formatting follows `.clang-format` (four-space indentation and an 80-column limit). CMake files are checked by `cmake-format`. Avoid formatting unrelated code as part of a focused change. ## Documentation and release notes - Update the relevant `.rst` pages for public behavior or API changes. C++ pages live at `docs/src/.rst`; Python pages live under `docs/src/python//`. - Add new pages to `docs/src/index.rst` or the relevant nested toctree. - Update `RELEASE.md` under `## Next` for user-visible features, bug fixes, and API changes. - Do not change `VERSION` unless performing an explicitly requested release. - Preserve compatibility with existing detector formats and older recorded files where practical. Call out intentional API or format incompatibilities. Build the documentation with: ```bash cmake -S . -B build \ -DAARE_DOCS=ON \ -DAARE_PYTHON_BINDINGS=ON cmake --build build --target docs ``` ## Working practices and handoff - Inspect the adjacent implementation, tests, and documentation before editing. - Keep changes focused and preserve unrelated modifications in the worktree. - Do not edit generated files or fetched dependency sources under `build/`. - For large features, prefer independently testable increments. - At handoff, summarize the behavior and important files changed, checks run, checks not run and why, and any compatibility or performance considerations.