aare/AGENTS.md

Repository guidelines

Scope and project goals

These instructions apply to the entire repository unless a more specific AGENTS.md exists in a subdirectory.

Aare is a data-analysis library for PSI hybrid detectors. The C++17 core is the canonical implementation, while Python is the main user-facing interface through pybind11. Changes should preserve the project's priorities: fast and efficient processing, simple interfaces, API stability, and a small dependency footprint.

Repository layout

• include/aare/ contains the public C++ API and header-defined templates.
• src/ contains C++ implementations and colocated Catch2 tests named *.test.cpp, grouped into src/fit/, src/hist/ and src/utils/ where a component has several private files.
• tests/ contains the C++ test executable, configuration, and shared helpers.
• python/src/ contains pybind11 bindings and module registration.
• python/aare/ contains Python facades, convenience APIs, and public exports.
• python/tests/ contains the pytest suite.
• docs/src/ contains Sphinx/reStructuredText documentation. Doxygen and Breathe integrate the C++ API into the Sphinx site.
• benchmarks/ contains Google Benchmark programs.
• cmake/, conda-recipe/, and pyproject.toml support builds and packaging.
• RELEASE.md contains pending and published release notes.

Build environment

The project requires CMake 3.15 or newer, C++17 with compiler extensions disabled, and Python 3.11 or newer. etc/dev-env.yml defines the Conda development environment.

CMake fetches several dependencies by default. Use -DAARE_SYSTEM_LIBRARIES=ON only when all required system or Conda packages are available.

Use this configuration for normal development:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Debug \
  -DAARE_TESTS=ON \
  -DAARE_PYTHON_BINDINGS=ON
cmake --build build -j4

Useful optional settings include AARE_DOCS, AARE_BENCHMARKS, AARE_ASAN, AARE_WARNINGS_AS_ERRORS, AARE_VERBOSE (raises the compile-time log level), and AARE_TUNE_LOCAL (-march=native, not portable). Reconfigure an existing build directory instead of creating alternate in-tree build layouts unless isolation is needed; build/CMakeCache.txt records the options and interpreter it was configured with.

-Werror=return-type is always enabled. New code should also compile cleanly under the project's -Wall -Wextra -pedantic -Wshadow -Wold-style-cast -Wdouble-promotion flags, which AARE_WARNINGS_AS_ERRORS turns into errors.

The build copies python/aare/*.py and the compiled _aare extension into build/aare/, so PYTHONPATH=build makes the package importable. The extension only imports into the interpreter CMake was configured with, which is normally the Conda or venv Python active at configure time. Activate that environment, or resolve the interpreter from the cache:

PY=$(grep -E '^_?Python_EXECUTABLE' build/CMakeCache.txt | head -1 | cut -d= -f2)
PYTHONPATH="$PWD/build" "$PY" -m pytest python/tests

Do not fall back to a different system Python.

With AARE_BENCHMARKS=ON, Google Benchmark programs build as build/run_benchmarks and build/fit_benchmark.

Architecture

Core containers

• NDArray<T, Ndim> owns contiguous C-order memory; NDView<T, Ndim> is a non-owning view with the same shape and stride API. Both derive from ArrayExpr (expression templates for element-wise math). Shapes are std::array<ssize_t, Ndim>. NDView<T> converts to NDView<const T>.
• Frame is a byte buffer plus rows, cols, and Dtype, the lowest common denominator returned by file readers. Dtype is a runtime type tag built from a bitdepth, a NumPy descriptor, or typeid.
• defs.hpp holds detector enums (DetectorType, TimingMode, FrameDiscardPolicy, ReadoutMode), the on-disk DetectorHeader, chip constants (Moench04, Matterhorn02, ...), the LOCATION macro used in error messages, and AARE_ASSERT (active only with AARE_CUSTOM_ASSERT).
• logger.hpp provides LOG(logLEVEL) << ...; levels above AARE_LOG_LEVEL (set by CMake) are compiled out.

File reading

File is the generic RAII facade. It holds a unique_ptr<FileInterface> and dispatches on file extension to one of the concrete readers:

• RawFile reads slsDetector .raw data with a _master_N.json (or legacy .raw) master file. It owns a RawMasterFile (master JSON, older raw-text masters, scan parameters, ROIs, disabled UDP ports) and a two-dimensional table of RawSubFile indexed [roi][module]. Each RawSubFile handles one module's sequence of _dN_fM_ data files and per-frame DetectorHeaders. Module placement into the assembled image goes through DetectorGeometry, ModuleGeometry, and ROIGeometry (module gaps, ROIs, UDP-port halves). Frame numbers from the detector are distinct from frame indices in the file; the API uses both and names them accordingly.
• NumpyFile (.npy), JungfrauDataFile (legacy .dat), and CtbRawFile (chip test board; raw ADC samples decoded through decode.hpp and the PixelMap generators, exposed in Python as aare.transform callables).
• experimental::MultiThreadedFileReader opens one File per worker and fills a caller-owned buffer in file order.

Some chip-specific decoding (Moench03/05 pixel maps, ADC SAR transforms) lives in python/aare/transform.py on top of C++ generators.

Pedestal and cluster finding

• Pedestal<T> and FastPedestal<T> maintain per-pixel running mean and standard deviation, always accumulating in double. FastPedestal must see min_pedestal_samples frames before ready().
• ClusterFinder<ClusterType, FRAME_TYPE, PEDESTAL_TYPE> is single-threaded: push pedestal frames, then find_clusters(view, frame_number) appends to an internal ClusterVector that callers take with steal_clusters(). Thresholds are n_sigma * std and are recomputed by update_threshold().
• ClusterFinderMT wraps N ClusterFinders, one per thread, each with a CircularFifo<FrameRef> input queue backed by a preallocated FramePool and a ProducerConsumerQueue output. A collector thread funnels all outputs into a single sink queue. Consumers of that sink run their own thread: ClusterCollector (accumulate in memory, steal_clusters()) or ClusterFileSink (write the binary cluster file). The stop() and sync() ordering matters; read the existing tests before changing lifecycle code.
• Cluster<T, SizeX, SizeY, CoordType> is a POD (x, y, data[]) and ClusterVector<Cluster<...>> is a frame-tagged std::vector of them. ClusterFile reads and writes the binary format (int32 frame_number, uint32 n_clusters, raw cluster bytes). The format carries no shape or dtype metadata, so the caller must supply matching template arguments. CalculateEta.hpp and Interpolator operate on ClusterVectors; algorithm.hpp holds small 1-D search helpers.
• hist/PixelHistogram and PedestalTrackingPixelHistogram build per-pixel histograms across a frame series, multithreaded by row bands.

Fitting

Fit.hpp, FitModel<Model>, and Models.hpp (Gaussian, Pol1, Pol2, RisingScurve, FallingScurve, GaussianErfcPlateau, GaussianChargeSharing, ...) form the public fitting API. Each model provides eval, eval_and_grad, is_valid, estimate_par, and param_info; the separable models also declare their linear parameters (linear_par, basis_and_grad, basis_columns). FitModel is plain data (limits, fixed flags, start values, minimizer settings) and selects one of four minimizers through aare::Minimizer: Minuit2's Migrad (default) and Fumili, or the built-in Levenberg-Marquardt and variable projection solvers. The private implementation lives in src/fit/: Fit.cpp holds the FitModel method bodies, the per-thread detail::PixelFitter that dispatches on the minimizer and falls back from VarPro to Levenberg-Marquardt, fit_pixel/fit_3d, and the explicit instantiations; FitHelpers.hpp holds the start-value precedence and shape checks; DampedGaussNewton.hpp is the damped Gauss-Newton driver shared by LevenbergMarquardt.hpp and VariableProjection.hpp, with the small fixed-size linear algebra on std::array (Vec<N>, Mat<N>). FitMinuit2.cpp (with Chi2.hpp and MinuitSteps.hpp) is the only translation unit that includes Minuit2 headers; Minuit2 is a private, BUILD_INTERFACE-only dependency of aare_core. The [fit] tests run every configuration through all four minimizers and check that they agree; run them when changing any solver.

Python layer

• python/src/module.cpp registers everything. Templated classes are instantiated per value type and cluster size through the DEFINE_CLUSTER_BINDINGS and DEFINE_BINDINGS_CLUSTERFINDER macros, producing names such as ClusterFinder_Cluster3x3i, Pedestal_d, and FastPedestal_i16. Supported cluster sizes are 2x2, 3x3, 5x5, 7x7, and 9x9; value types are i, f, d, and i16 (3x3 only). module_config.hpp fixes the module-wide pedestal type (pd_type).
• python/aare/factory.py maps NumPy dtypes to those suffixes. The facades in python/aare/*.py are thin factory functions or subclasses that select the right binding and add Pythonic conveniences (iteration, context managers, ScanParameters).
• np_helper.hpp provides return_image_data (moves an NDArray into a NumPy array with a capsule deleter) and make_view_1d/2d/3d (views into a NumPy buffer).
• aare.experimental and _aare.experimental hold unstable APIs.

Implementation conventions

• Put public declarations in include/aare/ and private implementation details in src/.
• Add new compiled headers, sources, and C++ tests to the explicit lists in the root CMakeLists.txt.
• Keep template implementations in headers unless the supported types are explicitly instantiated.
• Follow the naming in adjacent code. Broadly, use the aare namespace, CamelCase types, and snake_case functions.
• Treat ownership, lifetime, const-correctness, array shapes, and buffer contiguity as part of the API when working with NDArray, NDView, or NumPy bindings.
• Avoid unnecessary allocations and copies in detector-data and per-pixel processing paths. Add or update a benchmark when performance is central to a change.
• Place APIs that may change without notice under the existing experimental namespace/module.
• Start new source files with SPDX-License-Identifier: MPL-2.0, using the appropriate comment syntax.
• Code copied from other projects keeps its original license. Retain its copyright and license notice, set the SPDX identifier to that license, and list the license in LICENSES/, pyproject.toml, conda-recipe/meta.yaml, and README.md. etc/add_license.py skips files with a license notice.
• When adding or bumping a fetched dependency that is compiled into the Python extension, update its version and notice in THIRD-PARTY-NOTICES.txt and the license expressions in pyproject.toml and conda-recipe/meta.yaml.
• Prefer descriptive names to comments that only restate the code.
• Do not add a dependency unless the benefit justifies the packaging and deployment cost.

Python-facing changes

A Python-facing feature can require coordinated changes in several layers:

1. Update the C++ public API and implementation.
2. Add or update its binding in python/src/.
3. Register new bindings in python/src/module.cpp.
4. Update the facade or public exports in python/aare/.
5. If adding a Python module, add it to PYTHON_FILES in python/CMakeLists.txt so it is copied and installed.
6. Add Python tests and update user documentation.

Bindings must validate NumPy dimensions and data types before constructing views. Do not return a view whose backing C++ or Python storage can expire while the view remains reachable.

Tests

Run tests that do not require external detector data with:

ctest --test-dir build --output-on-failure -j4
PYTHONPATH="$PWD/build" python -m pytest python/tests

For focused runs, use a Catch2 tag or test name, or an individual pytest file or test:

build/run_tests "[tag]"
build/run_tests "Exact test case name"
PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py
PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py -k name

Large detector test files live outside this repository in https://gitea.psi.ch/detectors/aare-test-data.git (Git LFS). To include data-backed tests, set AARE_TEST_DATA and opt in explicitly:

export AARE_TEST_DATA=/path/to/aare-test-data
build/run_tests "[.with-data]"
PYTHONPATH="$PWD/build" python -m pytest python/tests --with-data

• Start bug fixes with a failing regression test when practical.
• Put C++ tests beside the relevant implementation as src/Thing.test.cpp and add them to TestSources in the root CMakeLists.txt.
• Mark C++ tests requiring external files with [.with-data].
• Mark Python tests requiring external files with @pytest.mark.withdata and use the test_data_path fixture from python/tests/conftest.py.
• Prefer synthesizing small input files over depending on external data. tests/raw_file_helpers.hpp builds throwaway raw files in C++, and many Python tests write small raw files into tmp_path.
• tests/test_config.hpp.in provides test_data_path(); tests/test_macros.hpp and tests/friend_test.hpp (TEST_CASE_PRIVATE with FRIEND_TEST) let a test reach private members.
• Run both suites for changes that cross the C++/Python boundary.
• Do not silently skip required data-backed coverage. Report when the external test data is unavailable.

Formatting and static analysis

The project uses pre-commit. Install the git hook once per clone with pre-commit install so it runs on every commit. The hooks in .pre-commit-config.yaml are clang-format on C++, cmake-format on CMake files, typos on docs/ (project words are whitelisted in typos.toml), and basic YAML, TOML, JSON, and large-file checks. CI runs the same hooks on pull requests.

Before handing off a broad change, run the relevant checks:

pre-commit install                            # once per clone
pre-commit run --all-files                    # what CI runs
cmake --build build --target check-format     # diff-based clang-format check
cmake --build build --target format-files     # apply clang-format to all .cpp/.hpp
cmake --build build --target clang-tidy       # uses build/compile_commands.json

C++ formatting follows .clang-format (four-space indentation and an 80-column limit). CMake files are checked by cmake-format. Avoid formatting unrelated code as part of a focused change.

Documentation and release notes

• Update the relevant .rst pages for public behavior or API changes. C++ pages live at docs/src/<Class>.rst; Python pages live under docs/src/python/<area>/.
• Add new pages to docs/src/index.rst or the relevant nested toctree.
• Update RELEASE.md under ## Next for user-visible features, bug fixes, and API changes.
• Do not change VERSION unless performing an explicitly requested release.
• Preserve compatibility with existing detector formats and older recorded files where practical. Call out intentional API or format incompatibilities.

Build the documentation with:

cmake -S . -B build \
  -DAARE_DOCS=ON \
  -DAARE_PYTHON_BINDINGS=ON
cmake --build build --target docs

Working practices and handoff

• Inspect the adjacent implementation, tests, and documentation before editing.
• Keep changes focused and preserve unrelated modifications in the worktree.
• Do not edit generated files or fetched dependency sources under build/.
• For large features, prefer independently testable increments.
• At handoff, summarize the behavior and important files changed, checks run, checks not run and why, and any compatibility or performance considerations.