Merge branch 'main' into dev/filereading_for_disabled_udp

AGENTS.md

@@ -0,0 +1,166 @@
+# Repository guidelines
+
+## Scope and project goals
+
+These instructions apply to the entire repository unless a more specific
+`AGENTS.md` exists in a subdirectory.
+
+Aare is a data-analysis library for PSI hybrid detectors. The C++17 core is
+the canonical implementation, while Python is the main user-facing interface
+through pybind11. Changes should preserve the project's priorities: fast and
+efficient processing, simple interfaces, API stability, and a small dependency
+footprint.
+
+## Repository layout
+
+- `include/aare/` contains the public C++ API and header-defined templates.
+- `src/` contains C++ implementations and colocated Catch2 tests named
+  `*.test.cpp`.
+- `tests/` contains the C++ test executable, configuration, and shared helpers.
+- `python/src/` contains pybind11 bindings and module registration.
+- `python/aare/` contains Python facades, convenience APIs, and public exports.
+- `python/tests/` contains the pytest suite.
+- `docs/src/` contains Sphinx/reStructuredText documentation. Doxygen and
+  Breathe integrate the C++ API into the Sphinx site.
+- `benchmarks/` contains Google Benchmark programs.
+- `cmake/`, `conda-recipe/`, and `pyproject.toml` support builds and packaging.
+- `RELEASE.md` contains pending and published release notes.
+
+## Build environment
+
+The project requires CMake 3.15 or newer, C++17 with compiler extensions
+disabled, and Python 3.11 or newer. `etc/dev-env.yml` defines the Conda
+development environment.
+
+CMake fetches several dependencies by default. Use
+`-DAARE_SYSTEM_LIBRARIES=ON` only when all required system or Conda packages
+are available.
+
+Use this configuration for normal development:
+
+```bash
+cmake -S . -B build \
+  -DCMAKE_BUILD_TYPE=Debug \
+  -DAARE_TESTS=ON \
+  -DAARE_PYTHON_BINDINGS=ON
+cmake --build build -j4
+```
+
+Useful optional settings include `AARE_DOCS`, `AARE_BENCHMARKS`, `AARE_ASAN`,
+and `AARE_WARNINGS_AS_ERRORS`. Reconfigure an existing build directory instead
+of creating alternate in-tree build layouts unless isolation is needed.
+
+## Implementation conventions
+
+- Put public declarations in `include/aare/` and private implementation details
+  in `src/`.
+- Add new compiled headers, sources, and C++ tests to the explicit lists in the
+  root `CMakeLists.txt`.
+- Keep template implementations in headers unless the supported types are
+  explicitly instantiated.
+- Follow the naming in adjacent code. Broadly, use the `aare` namespace,
+  CamelCase types, and snake_case functions.
+- Treat ownership, lifetime, const-correctness, array shapes, and buffer
+  contiguity as part of the API when working with `NDArray`, `NDView`, or NumPy
+  bindings.
+- Avoid unnecessary allocations and copies in detector-data and per-pixel
+  processing paths. Add or update a benchmark when performance is central to a
+  change.
+- Place APIs that may change without notice under the existing experimental
+  namespace/module.
+- Start new source files with `SPDX-License-Identifier: MPL-2.0`, using the
+  appropriate comment syntax.
+- Prefer descriptive names to comments that only restate the code.
+- Do not add a dependency unless the benefit justifies the packaging and
+  deployment cost.
+
+## Python-facing changes
+
+A Python-facing feature can require coordinated changes in several layers:
+
+1. Update the C++ public API and implementation.
+2. Add or update its binding in `python/src/`.
+3. Register new bindings in `python/src/module.cpp`.
+4. Update the facade or public exports in `python/aare/`.
+5. If adding a Python module, add it to `PYTHON_FILES` in
+   `python/CMakeLists.txt` so it is copied and installed.
+6. Add Python tests and update user documentation.
+
+Bindings must validate NumPy dimensions and data types before constructing
+views. Do not return a view whose backing C++ or Python storage can expire while
+the view remains reachable.
+
+## Tests
+
+Run tests that do not require external detector data with:
+
+```bash
+ctest --test-dir build --output-on-failure -j4
+PYTHONPATH="$PWD/build" python -m pytest python/tests
+```
+
+For focused runs, use a Catch2 tag or an individual pytest file/test:
+
+```bash
+build/run_tests "[tag]"
+PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py
+```
+
+Large detector test files live outside this repository. To include data-backed
+tests, set `AARE_TEST_DATA` and opt in explicitly:
+
+```bash
+export AARE_TEST_DATA=/path/to/aare-test-data
+build/run_tests "[.with-data]"
+PYTHONPATH="$PWD/build" python -m pytest python/tests --with-data
+```
+
+- Start bug fixes with a failing regression test when practical.
+- Put C++ tests beside the relevant implementation as `src/Thing.test.cpp`.
+- Mark C++ tests requiring external files with `[.with-data]`.
+- Mark Python tests requiring external files with `@pytest.mark.withdata`.
+- Run both suites for changes that cross the C++/Python boundary.
+- Do not silently skip required data-backed coverage. Report when the external
+  test data is unavailable.
+
+## Formatting and static analysis
+
+Before handing off a broad change, run the relevant checks:
+
+```bash
+pre-commit run --all-files
+cmake --build build --target check-format
+cmake --build build --target clang-tidy
+```
+
+C++ formatting follows `.clang-format` (four-space indentation and an 80-column
+limit). CMake files are checked by `cmake-format`. Avoid formatting unrelated
+code as part of a focused change.
+
+## Documentation and release notes
+
+- Update the relevant `.rst` pages for public behavior or API changes.
+- Add new pages to `docs/src/index.rst` or the relevant nested toctree.
+- Update `RELEASE.md` under `## Next` for user-visible features, bug fixes, and
+  API changes.
+- Do not change `VERSION` unless performing an explicitly requested release.
+- Preserve compatibility with existing detector formats and older recorded
+  files where practical. Call out intentional API or format incompatibilities.
+
+Build the documentation with:
+
+```bash
+cmake -S . -B build \
+  -DAARE_DOCS=ON \
+  -DAARE_PYTHON_BINDINGS=ON
+cmake --build build --target docs
+```
+
+## Working practices and handoff
+
+- Inspect the adjacent implementation, tests, and documentation before editing.
+- Keep changes focused and preserve unrelated modifications in the worktree.
+- Do not edit generated files or fetched dependency sources under `build/`.
+- For large features, prefer independently testable increments.
+- At handoff, summarize the behavior and important files changed, checks run,
+  checks not run and why, and any compatibility or performance considerations.

RELEASE.md

@@ -4,6 +4,8 @@
 
 ### API Changes:
 
+- Exposed ``ClusterVector.empty()`` in the Python API.
+- Added ClusterVector.estimate_n_clusters
 - Removed the lmfit dependency and the legacy ``fit_gaus``, ``fit_pol1``,
   ``fit_scurve``, and ``fit_scurve2`` APIs. Use ``Gaussian``, ``Pol1``,
   ``RisingScurve``, or ``FallingScurve`` and call ``model.fit(...)`` (or
@@ -11,6 +13,8 @@
 - Removed the legacy ``gaus``, ``pol1``, ``scurve``, and ``scurve2`` function
   evaluators. Model objects are callable and provide the replacement, for
   example ``Gaussian()(x, par)``.
+- ``NDView<T, Ndim>`` now converts to ``NDView<const T, Ndim>``;
+  ``expand4to8bit`` and ``expand24to32bit`` accept const input views.
 
 ### Bugfixes:
 - Fixed broken reading of old (pre reordering) Moench03
@@ -166,4 +170,3 @@ dhanya.thattil@psi.ch
 
 
 
-

include/aare/ClusterFile.hpp

@@ -144,6 +144,19 @@ class ClusterFile {
      */
     size_t chunk_size() const { return m_chunk_size; }
 
+    /**
+     * @brief Estimate the number of clusters in the file from its size
+     *
+     * Frame headers are included in the estimate, so the result may be slightly
+     * larger than the actual number of clusters. The file position is not
+     * changed.
+     *
+     * @throws std::runtime_error if the file is not opened for reading
+     */
+    size_t estimate_n_clusters() const {
+        return std::filesystem::file_size(m_filename) / sizeof(ClusterType);
+    }
+
     /**
      * @brief Set the region of interest to use when reading
      * clusters. If set only clusters within the ROI will be

include/aare/NDView.hpp

@@ -12,6 +12,7 @@
 #include <iostream>
 #include <numeric>
 #include <stdexcept>
+#include <type_traits>
 #include <vector>
 namespace aare {
 
@@ -92,6 +93,12 @@ class NDView : public ArrayExpr<NDView<T, Ndim>, Ndim> {
         : buffer_(buffer), strides_(c_strides<Ndim>(shape)), shape_(shape),
           size_(num_elements(shape)) {}
 
+    template <typename U, std::enable_if_t<
+                              std::is_convertible_v<U (*)[], T (*)[]>, int> = 0>
+    NDView(const NDView<U, Ndim> &other) noexcept
+        : buffer_(other.buffer_), strides_(other.strides_),
+          shape_(other.shape_), size_(other.size_) {}
+
     template <typename... Ix>
     std::enable_if_t<sizeof...(Ix) == Ndim, T &> operator()(Ix... index) {
         return buffer_[element_offset(strides_, index...)];
@@ -216,6 +223,8 @@ class NDView : public ArrayExpr<NDView<T, Ndim>, Ndim> {
     }
 
   private:
+    template <typename, ssize_t> friend class NDView;
+
     T *buffer_{nullptr};
     std::array<ssize_t, Ndim> strides_{};
     std::array<ssize_t, Ndim> shape_{};
@@ -263,4 +272,4 @@ template <typename T> NDView<T, 1> make_view(std::vector<T> &vec) {
     return NDView<T, 1>(vec.data(), {static_cast<ssize_t>(vec.size())});
 }
 
-} // namespace aare
+} // namespace aare

include/aare/decode.hpp

@@ -35,7 +35,7 @@ uint32_t mask32to24bits(uint32_t input, BitOffset offset = {});
  * @param offset Offset within the first byte to where the data starts (0-7
  * bits)
  */
-void expand24to32bit(NDView<uint8_t, 1> input, NDView<uint32_t, 1> output,
+void expand24to32bit(NDView<const uint8_t, 1> input, NDView<uint32_t, 1> output,
                      BitOffset offset = {});
 
 /**
@@ -43,7 +43,7 @@ void expand24to32bit(NDView<uint8_t, 1> input, NDView<uint32_t, 1> output,
  * @param input input buffer with 4 bit values packed into 8 bit
  * @param output output buffer with 8 bit values
  */
-void expand4to8bit(NDView<uint8_t, 1> input, NDView<uint8_t, 1> output);
+void expand4to8bit(NDView<const uint8_t, 1> input, NDView<uint8_t, 1> output);
 
 /**
  * @brief Apply custom weights to a 16-bit input value. Will sum up

python/src/bind_ClusterFile.hpp

@@ -47,6 +47,8 @@ void define_ClusterFile(py::module &m, const std::string &typestr) {
              })
         .def("set_roi", &ClusterFile<ClusterType>::set_roi, py::arg("roi"))
         .def("tell", &ClusterFile<ClusterType>::tell)
+        .def("estimate_n_clusters",
+             &ClusterFile<ClusterType>::estimate_n_clusters)
         .def(
             "set_noise_map",
             [](ClusterFile<ClusterType> &self, py::array_t<int32_t> noise_map) {
@@ -82,4 +84,4 @@ void define_ClusterFile(py::module &m, const std::string &typestr) {
         });
 }
 
-#pragma GCC diagnostic pop
+#pragma GCC diagnostic pop

python/src/bind_ClusterVector.hpp

@@ -72,6 +72,7 @@ void define_ClusterVector(py::module &m, const std::string &typestr) {
             R"(calculates sum of 2x2 subcluster with highest energy and index relative to cluster center 0: top_left, 1: top_right, 2: bottom_left, 3: bottom_right
             )")
         .def_property_readonly("size", &ClusterVector<ClusterType>::size)
+        .def("empty", &ClusterVector<ClusterType>::empty)
         .def("item_size", &ClusterVector<ClusterType>::item_size)
         .def_property_readonly("fmt",
                                [typestr](ClusterVector<ClusterType> &self) {
@@ -169,4 +170,4 @@ void define_3x3_reduction(py::module &m) {
         py::arg("clustervector"));
 }
 
-#pragma GCC diagnostic pop
+#pragma GCC diagnostic pop

python/tests/test_ClusterFile.py

@@ -14,6 +14,8 @@ from conftest import test_data_path
 def test_cluster_file(test_data_path): 
     """Test ClusterFile""" 
     f =  ClusterFile(test_data_path / "clust/single_frame_97_clustrers.clust") 
+    assert f.estimate_n_clusters() == 97
+    assert f.tell() == 0
     cv = f.read_clusters(10) #conversion does not work
 
 
@@ -62,4 +64,4 @@ def test_read_clusters_and_fill_histogram(test_data_path):
         hist_py = pickle.load(f)
         
     #Compare the two histograms
-    assert hist_aare == hist_py
+    assert hist_aare == hist_py

python/tests/test_ClusterVector.py

@@ -16,6 +16,7 @@ def test_create_cluster_vector():
     assert cv.cluster_size_x == 3
     assert cv.cluster_size_y == 3
     assert cv.size == 0
+    assert cv.empty()
 
 
 def test_push_back_on_cluster_vector():
@@ -27,6 +28,7 @@ def test_push_back_on_cluster_vector():
     cluster = _aare.Cluster2x2i(19, 22, np.ones(4, dtype=np.int32))
     cv.push_back(cluster)
     assert cv.size == 1
+    assert not cv.empty()
 
     arr = np.array(cv, copy=False)
     assert arr[0]['x'] == 19
@@ -127,4 +129,4 @@ def test_masking():
     
     assert cv_masked_array[0]["x"] == 1
     assert cv_masked_array[0]["y"] == 2
-    assert (cv_masked_array[0]["data"] == np.ones((3,3),dtype=np.int32)).all()
+    assert (cv_masked_array[0]["data"] == np.ones((3,3),dtype=np.int32)).all()

src/ClusterFile.test.cpp

@@ -17,6 +17,8 @@ TEST_CASE("Read one frame from a cluster file", "[.with-data]") {
     REQUIRE(std::filesystem::exists(fpath));
 
     ClusterFile<Cluster<int32_t, 3, 3>> f(fpath);
+    CHECK(f.estimate_n_clusters() == 97);
+    CHECK(f.tell() == 0);
     auto clusters = f.read_frame();
     CHECK(clusters.size() == 97);
     CHECK(clusters.frame_number() == 135);
@@ -250,6 +252,8 @@ TEST_CASE("Read cluster from multiple frame file", "[.with-data]") {
 
     SECTION("Read clusters from both frames") {
         ClusterFile<ClusterType> f(fpath);
+        CHECK(f.estimate_n_clusters() == 8);
+        CHECK(f.tell() == 0);
         auto clusters = f.read_clusters(2);
         REQUIRE(clusters.size() == 2);
         REQUIRE(clusters.frame_number() == 0);

src/NDView.test.cpp

@@ -5,12 +5,16 @@
 #include <cstddef>
 #include <iostream>
 #include <numeric>
+#include <type_traits>
 #include <vector>
 
 using aare::NDView;
 using aare::num_elements;
 using aare::Shape;
 
+static_assert(std::is_convertible_v<NDView<int, 2>, NDView<const int, 2>>);
+static_assert(!std::is_convertible_v<NDView<const int, 2>, NDView<int, 2>>);
+
 TEST_CASE("Calculate size from a shape") {
     Shape<3> shape{2, 3, 4};
     REQUIRE(num_elements(shape) == 24);
@@ -88,6 +92,17 @@ TEST_CASE("Element reference 1D with a const NDView") {
     }
 }
 
+TEST_CASE("Convert a mutable NDView to a const NDView") {
+    std::vector<int> vec{1, 2, 3, 4};
+    NDView<int, 2> mutable_view(vec.data(), Shape<2>{2, 2});
+
+    NDView<const int, 2> const_view = mutable_view;
+
+    REQUIRE(const_view.data() == mutable_view.data());
+    REQUIRE(const_view.shape() == mutable_view.shape());
+    REQUIRE(const_view.strides() == mutable_view.strides());
+}
+
 TEST_CASE("Element reference 2D") {
     std::vector<int> vec(12);
     std::iota(vec.begin(), vec.end(), 0);
@@ -279,4 +294,4 @@ TEST_CASE("NDView over byte") {
     auto v = aare::make_view(buf);
     REQUIRE(v.shape()[0] == 5);
     REQUIRE(v[0] == std::byte{0});
-}
+}

src/decode.cpp

@@ -144,7 +144,7 @@ uint32_t mask32to24bits(uint32_t input, BitOffset offset) {
     return (input >> offset.value()) & mask24bits;
 }
 
-void expand4to8bit(NDView<uint8_t, 1> input, NDView<uint8_t, 1> output) {
+void expand4to8bit(NDView<const uint8_t, 1> input, NDView<uint8_t, 1> output) {
 
     if (2 * input.size() != output.size())
         throw std::runtime_error(
@@ -162,7 +162,7 @@ void expand4to8bit(NDView<uint8_t, 1> input, NDView<uint8_t, 1> output) {
     }
 }
 
-void expand24to32bit(NDView<uint8_t, 1> input, NDView<uint32_t, 1> output,
+void expand24to32bit(NDView<const uint8_t, 1> input, NDView<uint32_t, 1> output,
                      BitOffset bit_offset) {
 
     ssize_t bytes_per_channel = 3; // 24bit

src/decode.test.cpp

@@ -154,6 +154,20 @@ TEST_CASE("Expand container with 24 bit data to 32") {
     }
 }
 
+TEST_CASE("Expand 24 bit values to 32 bit values from a const buffer") {
+    const uint8_t buffer[] = {
+        0x0F, 0x00, 0x00, 0xFF, 0x00, 0x00, 0xFF, 0xFF, 0xFF,
+    };
+
+    aare::NDView<const uint8_t, 1> input(buffer, {9});
+    aare::NDArray<uint32_t, 1> out({3});
+    aare::expand24to32bit(input, out.view());
+
+    CHECK(out(0) == 0xF);
+    CHECK(out(1) == 0xFF);
+    CHECK(out(2) == 0xFFFFFF);
+}
+
 TEST_CASE("Expand 4 bit values packed into 8 bit to 8 bit values") {
     {
         uint8_t buffer[] = {
@@ -172,4 +186,25 @@ TEST_CASE("Expand 4 bit values packed into 8 bit to 8 bit values") {
             CHECK(out(i) == expected_output[i]);
         }
     }
-}
+}
+
+TEST_CASE("Expand 4 bit values packed into 8 bit to 8 bit values from a const "
+          "buffer") {
+    {
+        const uint8_t buffer[] = {
+            0x00, 0xF0, 0xFF, 0x00, 0xF0, 0xFF,
+        };
+
+        aare::NDView<const uint8_t, 1> input(&buffer[0], {6});
+        aare::NDArray<uint8_t, 1> out({12});
+        aare::expand4to8bit(input, out.view());
+
+        uint8_t expected_output[] = {
+            0x0, 0x0, 0x0, 0xF, 0xF, 0xF,
+            0x0, 0x0, 0x0, 0xF, 0xF, 0xF}; // assuming little endian
+
+        for (size_t i = 0; i < 12; ++i) {
+            CHECK(out(i) == expected_output[i]);
+        }
+    }
+}