diff --git a/AGENTS.md b/AGENTS.md index be6fda68..7311e097 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,8 +47,126 @@ cmake --build build -j4 ``` Useful optional settings include `AARE_DOCS`, `AARE_BENCHMARKS`, `AARE_ASAN`, -and `AARE_WARNINGS_AS_ERRORS`. Reconfigure an existing build directory instead -of creating alternate in-tree build layouts unless isolation is needed. +`AARE_WARNINGS_AS_ERRORS`, `AARE_VERBOSE` (raises the compile-time log +level), and `AARE_TUNE_LOCAL` (`-march=native`, not portable). Reconfigure an +existing build directory instead of creating alternate in-tree build layouts +unless isolation is needed; `build/CMakeCache.txt` records the options and +interpreter it was configured with. + +`-Werror=return-type` is always enabled. New code should also compile cleanly +under the project's `-Wall -Wextra -pedantic -Wshadow -Wold-style-cast +-Wdouble-promotion` flags, which `AARE_WARNINGS_AS_ERRORS` turns into errors. + +The build copies `python/aare/*.py` and the compiled `_aare` extension into +`build/aare/`, so `PYTHONPATH=build` makes the package importable. The +extension only imports into the interpreter CMake was configured with, which +is normally the Conda or venv Python active at configure time. Activate that +environment, or resolve the interpreter from the cache: + +```bash +PY=$(grep -E '^_?Python_EXECUTABLE' build/CMakeCache.txt | head -1 | cut -d= -f2) +PYTHONPATH="$PWD/build" "$PY" -m pytest python/tests +``` + +Do not fall back to a different system Python. + +With `AARE_BENCHMARKS=ON`, Google Benchmark programs build as +`build/run_benchmarks` and `build/fit_benchmark`. + +## Architecture + +### Core containers + +- `NDArray` owns contiguous C-order memory; `NDView` is a + non-owning view with the same shape and stride API. Both derive from + `ArrayExpr` (expression templates for element-wise math). Shapes are + `std::array`. `NDView` converts to `NDView`. +- `Frame` is a byte buffer plus rows, cols, and `Dtype`, the lowest common + denominator returned by file readers. `Dtype` is a runtime type tag built + from a bitdepth, a NumPy descriptor, or `typeid`. +- `defs.hpp` holds detector enums (`DetectorType`, `TimingMode`, + `FrameDiscardPolicy`, `ReadoutMode`), the on-disk `DetectorHeader`, chip + constants (`Moench04`, `Matterhorn02`, ...), the `LOCATION` macro used in + error messages, and `AARE_ASSERT` (active only with `AARE_CUSTOM_ASSERT`). +- `logger.hpp` provides `LOG(logLEVEL) << ...`; levels above `AARE_LOG_LEVEL` + (set by CMake) are compiled out. + +### File reading + +`File` is the generic RAII facade. It holds a `unique_ptr` and +dispatches on file extension to one of the concrete readers: + +- `RawFile` reads slsDetector `.raw` data with a `_master_N.json` (or legacy + `.raw`) master file. It owns a `RawMasterFile` (master JSON, older raw-text + masters, scan parameters, ROIs, disabled UDP ports) and a two-dimensional + table of `RawSubFile` indexed `[roi][module]`. Each `RawSubFile` handles one + module's sequence of `_dN_fM_` data files and per-frame `DetectorHeader`s. + Module placement into the assembled image goes through `DetectorGeometry`, + `ModuleGeometry`, and `ROIGeometry` (module gaps, ROIs, UDP-port halves). + Frame numbers from the detector are distinct from frame indices in the + file; the API uses both and names them accordingly. +- `NumpyFile` (`.npy`), `JungfrauDataFile` (legacy `.dat`), and `CtbRawFile` + (chip test board; raw ADC samples decoded through `decode.hpp` and the + `PixelMap` generators, exposed in Python as `aare.transform` callables). +- `experimental::MultiThreadedFileReader` opens one `File` per worker and + fills a caller-owned buffer in file order. + +Some chip-specific decoding (Moench03/05 pixel maps, ADC SAR transforms) +lives in `python/aare/transform.py` on top of C++ generators. + +### Pedestal and cluster finding + +- `Pedestal` and `FastPedestal` maintain per-pixel running mean and + standard deviation, always accumulating in `double`. `FastPedestal` must + see `min_pedestal_samples` frames before `ready()`. +- `ClusterFinder` is single-threaded: + push pedestal frames, then `find_clusters(view, frame_number)` appends to an + internal `ClusterVector` that callers take with `steal_clusters()`. + Thresholds are `n_sigma * std` and are recomputed by `update_threshold()`. +- `ClusterFinderMT` wraps N `ClusterFinder`s, one per thread, each with a + `CircularFifo` input queue backed by a preallocated `FramePool` + and a `ProducerConsumerQueue` output. A collector thread funnels all + outputs into a single sink queue. Consumers of that sink run their own + thread: `ClusterCollector` (accumulate in memory, `steal_clusters()`) or + `ClusterFileSink` (write the binary cluster file). The `stop()` and + `sync()` ordering matters; read the existing tests before changing + lifecycle code. +- `Cluster` is a POD (x, y, data[]) and + `ClusterVector>` is a frame-tagged `std::vector` of them. + `ClusterFile` reads and writes the binary format (`int32 frame_number`, + `uint32 n_clusters`, raw cluster bytes). The format carries no shape or + dtype metadata, so the caller must supply matching template arguments. + `CalculateEta.hpp` and `Interpolator` operate on `ClusterVector`s; + `algorithm.hpp` holds small 1-D search helpers. +- `hist/PixelHistogram` and `PedestalTrackingPixelHistogram` build per-pixel + histograms across a frame series, multithreaded by row bands. + +### Fitting + +`Fit.hpp`, `FitModel`, and `Models.hpp` (`Gaussian`, `Pol1`, `Pol2`, +`RisingScurve`, `FallingScurve`, `GaussianErfcPlateau`, +`GaussianChargeSharing`, ...) wrap Minuit2. Template bodies and explicit +instantiations live in `src/Fit.cpp`; `FitModelImpl` is a pimpl so Minuit2 +headers never leak into the public API. Minuit2 is a private, +`BUILD_INTERFACE`-only dependency of `aare_core`. + +### Python layer + +- `python/src/module.cpp` registers everything. Templated classes are + instantiated per value type and cluster size through the + `DEFINE_CLUSTER_BINDINGS` and `DEFINE_BINDINGS_CLUSTERFINDER` macros, + producing names such as `ClusterFinder_Cluster3x3i`, `Pedestal_d`, and + `FastPedestal_i16`. Supported cluster sizes are 2x2, 3x3, 5x5, 7x7, and + 9x9; value types are `i`, `f`, `d`, and `i16` (3x3 only). + `module_config.hpp` fixes the module-wide pedestal type (`pd_type`). +- `python/aare/factory.py` maps NumPy dtypes to those suffixes. The facades + in `python/aare/*.py` are thin factory functions or subclasses that select + the right binding and add Pythonic conveniences (iteration, context + managers, `ScanParameters`). +- `np_helper.hpp` provides `return_image_data` (moves an `NDArray` into a + NumPy array with a capsule deleter) and `make_view_1d/2d/3d` (views into a + NumPy buffer). +- `aare.experimental` and `_aare.experimental` hold unstable APIs. ## Implementation conventions @@ -99,15 +217,19 @@ ctest --test-dir build --output-on-failure -j4 PYTHONPATH="$PWD/build" python -m pytest python/tests ``` -For focused runs, use a Catch2 tag or an individual pytest file/test: +For focused runs, use a Catch2 tag or test name, or an individual pytest +file or test: ```bash build/run_tests "[tag]" +build/run_tests "Exact test case name" PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py +PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py -k name ``` -Large detector test files live outside this repository. To include data-backed -tests, set `AARE_TEST_DATA` and opt in explicitly: +Large detector test files live outside this repository in +`https://gitea.psi.ch/detectors/aare-test-data.git` (Git LFS). To include +data-backed tests, set `AARE_TEST_DATA` and opt in explicitly: ```bash export AARE_TEST_DATA=/path/to/aare-test-data @@ -116,21 +238,38 @@ PYTHONPATH="$PWD/build" python -m pytest python/tests --with-data ``` - Start bug fixes with a failing regression test when practical. -- Put C++ tests beside the relevant implementation as `src/Thing.test.cpp`. +- Put C++ tests beside the relevant implementation as `src/Thing.test.cpp` + and add them to `TestSources` in the root `CMakeLists.txt`. - Mark C++ tests requiring external files with `[.with-data]`. -- Mark Python tests requiring external files with `@pytest.mark.withdata`. +- Mark Python tests requiring external files with `@pytest.mark.withdata` and + use the `test_data_path` fixture from `python/tests/conftest.py`. +- Prefer synthesizing small input files over depending on external data. + `tests/raw_file_helpers.hpp` builds throwaway raw files in C++, and many + Python tests write small raw files into `tmp_path`. +- `tests/test_config.hpp.in` provides `test_data_path()`; + `tests/test_macros.hpp` and `tests/friend_test.hpp` (`TEST_CASE_PRIVATE` + with `FRIEND_TEST`) let a test reach private members. - Run both suites for changes that cross the C++/Python boundary. - Do not silently skip required data-backed coverage. Report when the external test data is unavailable. ## Formatting and static analysis +The project uses pre-commit. Install the git hook once per clone with +`pre-commit install` so it runs on every commit. The hooks in +`.pre-commit-config.yaml` are clang-format on C++, cmake-format on CMake +files, `typos` on `docs/` (project words are whitelisted in `typos.toml`), and +basic YAML, TOML, JSON, and large-file checks. CI runs the same hooks on pull +requests. + Before handing off a broad change, run the relevant checks: ```bash -pre-commit run --all-files -cmake --build build --target check-format -cmake --build build --target clang-tidy +pre-commit install # once per clone +pre-commit run --all-files # what CI runs +cmake --build build --target check-format # diff-based clang-format check +cmake --build build --target format-files # apply clang-format to all .cpp/.hpp +cmake --build build --target clang-tidy # uses build/compile_commands.json ``` C++ formatting follows `.clang-format` (four-space indentation and an 80-column @@ -139,7 +278,9 @@ code as part of a focused change. ## Documentation and release notes -- Update the relevant `.rst` pages for public behavior or API changes. +- Update the relevant `.rst` pages for public behavior or API changes. C++ + pages live at `docs/src/.rst`; Python pages live under + `docs/src/python//`. - Add new pages to `docs/src/index.rst` or the relevant nested toctree. - Update `RELEASE.md` under `## Next` for user-visible features, bug fixes, and API changes. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..040b659a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,8 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +All project guidance lives in `AGENTS.md`: layout, build and test commands, +architecture, conventions, formatting, documentation, and release notes. +Read it and follow it. Keep new guidance there rather than here so that every +agent and contributor sees the same instructions. diff --git a/CMakeLists.txt b/CMakeLists.txt index 33fa709b..c3d2c75d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -394,6 +394,7 @@ set(PUBLICHEADERS include/aare/hist/PixelHistogramImpl.hpp include/aare/hist/PedestalTrackingPixelHistogram.hpp include/aare/GainMap.hpp + include/aare/ROI.hpp include/aare/ROIGeometry.hpp include/aare/DetectorGeometry.hpp include/aare/JungfrauDataFile.hpp @@ -408,6 +409,7 @@ set(PUBLICHEADERS include/aare/RawMasterFile.hpp include/aare/RawSubFile.hpp include/aare/VarClusterFinder.hpp + include/aare/to_string.hpp include/aare/utils/task.hpp include/aare/utils/ifstream_helpers.hpp include/aare/utils/math_helpers.hpp) @@ -428,6 +430,7 @@ set(SourceFiles ${CMAKE_CURRENT_SOURCE_DIR}/src/CtbRawFile.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/decode.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/defs.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/ROI.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/ROIGeometry.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/DetectorGeometry.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PedestalTrackingPixelHistogram.cpp @@ -476,7 +479,7 @@ if(AARE_VERBOSE) target_compile_definitions(aare_core PUBLIC AARE_VERBOSE) target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logDEBUG5) else() - target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logERROR) + target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logWARNING) endif() if(AARE_CUSTOM_ASSERT) @@ -499,11 +502,13 @@ endif() if(AARE_TESTS) set(TestSources + ${CMAKE_CURRENT_SOURCE_DIR}/tests/raw_file_helpers.hpp ${CMAKE_CURRENT_SOURCE_DIR}/src/algorithm.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/calibration.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/defs.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/decode.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Dtype.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/FilePtr.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Fit.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Frame.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/DetectorGeometry.test.cpp @@ -518,6 +523,7 @@ if(AARE_TESTS) ${CMAKE_CURRENT_SOURCE_DIR}/src/ClusterFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/ClusterFinderMT.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Pedestal.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/FastPedestal.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PixelHistogramImpl.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PixelHistogram.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PedestalTrackingPixelHistogram.test.cpp @@ -527,6 +533,7 @@ if(AARE_TESTS) ${CMAKE_CURRENT_SOURCE_DIR}/src/NumpyHelpers.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/RawFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/RawSubFile.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/ROI.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/utils/task.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/to_string.test.cpp) target_sources(tests PRIVATE ${TestSources}) diff --git a/RELEASE.md b/RELEASE.md index f4ed7745..7074ef53 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -4,6 +4,16 @@ ### New Features: +- Added the Python ``FrameDiscardPolicy`` enum with ``NoDiscard``, ``Discard``, + and ``DiscardPartial``, enabling access to ``RawMasterFile.frame_discard_policy``. +- Added the Python ``Pedestal`` factory with ``dtype`` selection, matching + ``FastPedestal`` and defaulting to ``float64`` output. +- Added ``FastPedestal`` in C++ and Python for per-pixel running mean + and population standard deviation. It supports exponentially +- Added ``ClusterFile.frames()`` and ``ClusterFile.chunks()`` in C++ and + Python for iteration from the current file position. Frames preserve empty + frames and frame numbers; chunks support an optional positive size override. + Python's default file iteration continues to yield chunks. - Added ``FastPedestal`` in C++ and Python for per-pixel running mean, population variance, and standard deviation. It supports exponentially weighted updates, initialization from files, direct subtraction from NumPy @@ -16,9 +26,32 @@ - Added the ``AARE_TUNE_LOCAL`` CMake option to build with ``-march=native`` and ``-mtune=native`` when supported. Binaries built with this option are specific to the local CPU and may not be portable. +- Added string representator in python for Cluster and Eta +- Added roi slice method in python for easy slicing of numpy arrays ``array[roi.slice()]``. +- added context manager for ``aare.RawMasterFile`` ### API Changes: +- ``FastPedestal`` variance is now a private ``double`` intermediate. Removed + the C++ ``variance()``/``variance_unchecked()`` APIs and Python ``var()``. + Standard deviation is calculated before conversion to the output type, + avoiding overflow of intermediate variance for ``int16`` output. Negative + variance from floating-point roundoff is clamped to zero. +- ``Pedestal`` now always accumulates sums and sums of squares in ``double``, + like ``FastPedestal``. Mean and standard deviation output types + are unchanged. Removed the C++ ``get_sum()``/``get_sum2()`` getters and + Python ``sum``/``sum2`` properties; internal sums are no longer exposed. +- Removed the public ``Pedestal.variance()`` and ``cached_std()`` APIs and + C++ ``update_std()``. Use ``std()`` to calculate the current population + standard deviation; variance is now an internal implementation detail. +- Added ``ClusterFile::read_frame(ClusterVector&)`` for allocation-reusing C++ + reads. It returns ``false`` at a clean end of file. The value-returning C++ + overload now returns ``std::optional>`` and + Python ``read_frame()`` returns ``None`` at end of file. Incomplete frames + still raise an error. +- Added an explicit boolean conversion to the C++ ``FilePtr`` type. +- Removed the public C++ ``ClusterFile::open`` method. Construct a new + ``ClusterFile`` to reopen a file or change its mode. - ``ClusterFinder`` now uses ``FastPedestal``. It must receive 1000 pedestal frames before cluster finding; ``find_clusters()`` raises an error until initialization is complete. Added ``update_threshold()`` to @@ -37,8 +70,62 @@ - ``NDView`` now converts to ``NDView``; ``expand4to8bit`` and ``expand24to32bit`` accept const input views. +- ``RawMasterFile::geometry()`` is deprecetad and returns full detector geometry information including module geometry. Use +``RawMasterFile::module_layout()`` to get num_modules in x an y +- ``RawMasterFile::rois()`` always returns a list of rois (no optional). Per default it returns a list of one ROI element spawing the entire detector +- ``TimingMode::Auto`` changed to ``TimingMode::AUTO_TIMING``, ``TimingMode::Trigger`` changed to ``TimingMode::TRIGGER_EXPOSURE`` + ### Bugfixes: +- Mismatched operators inhibited vectorization in gcc of NDArray math operators +- ``RawFile`` and ``File`` reject raw files with frame padding disabled unless + the frame discard policy is ``discardpartial``. The constructor reports the + master path before opening data subfiles. Legacy ``.raw`` master files now + parse the frame discard policy so unpadded ``discardpartial`` files remain + readable. +- Fixed ``CtbRawFile.read_frame(index)`` and reads after ``seek(index)`` at + subfile boundaries skipping a subfile, returning the wrong frame or raising + ``Subfile index out of range``. +- ``Pedestal`` reports mismatched frame shapes with exceptions in all push + overloads, including Debug builds, instead of aborting on assertions. +- Python ``Pedestal`` constructors reject negative dimensions and sample + counts instead of converting them to large unsigned values. +- ``Pedestal`` clamps negative variance from floating-point roundoff to zero, + preventing NaN standard deviations for nearly constant inputs. +- Python ``Pedestal.push()`` now requires C-contiguous ``uint16`` frames + without implicit conversion. Both ``push()`` and ``push_with_threshold()`` + validate that frames and thresholds are two-dimensional before constructing + views, preventing incorrect results from unsupported array layouts or ranks. + +- Fixed a leaked empty ``ClusterVector`` at the end of Python ``ClusterFile`` + iteration. Chunk iteration now rejects a zero chunk size. +- ``ClusterFile::read_clusters`` and Python iteration now report incomplete + frame headers and cluster records instead of treating truncated files as a + clean end of file, with or without ROI or noise filtering. +- ``ClusterFile::write_frame`` now reports incomplete writes instead of + silently continuing with a truncated file. +- Gain-map application now checks the complete cluster footprint, preventing + out-of-bounds access for cluster sizes larger than 3x3. +- ``RawFile`` now derives its frame count from the shortest selected raw + subfile series across all ROIs. Frame-number reads use the same bounds, and + Python ``len(reader)`` returns the adjusted count. A warning is printed when + subfile counts differ or their minimum differs from the recorded master + count. The expected frame count is not used and master metadata is preserved. + Warning-level logging is now enabled in non-verbose builds. +- Raw frame, batch, ROI, and frame-number read errors now include the attempted + frame index and file path in C++ and Python, preserving subfile error details. + Subfile errors omit redundant master-file context and C++ source locations; + out-of-range errors include the available frame count. + Synchronization errors identify the last raw data file for the affected module. + Top-level frame bounds errors state the total frame count and that indices + are zero-based. +- Fixed ``ClusterVector`` move operations to transfer storage instead of + copying every cluster. +- Validate that ``ClusterVector`` masks are one-dimensional, C-contiguous + Boolean arrays. +- Preserve signed ``ClusterVector`` frame numbers when filtering or reducing + cluster dimensions. - Fixed broken reading of old (pre reordering) Moench03 +- Supports reading all timing modes supported in slsDetectorPackage (auto, trigger, gating, burst_trigger, trigger_gating) ## 2026.7.2 @@ -54,6 +141,7 @@ - ``aare.transfrom.Matterhorn10Transform`` reshapes data such that first dimension is number of counters - Added support for len() for files. Returns the number of frames - Added support for direct subtraction of Pedestal from numpy array +- Added support to read files with disabled udp ports ### Bugfixes: @@ -185,6 +273,3 @@ dhanya.thattil@psi.ch - - - diff --git a/benchmarks/CMakeLists.txt b/benchmarks/CMakeLists.txt index 78031277..e874a003 100644 --- a/benchmarks/CMakeLists.txt +++ b/benchmarks/CMakeLists.txt @@ -18,8 +18,9 @@ FetchContent_MakeAvailable(benchmark) add_executable(benchmarks) target_sources( - benchmarks PRIVATE ndarray_benchmark.cpp ndview_benchmark.cpp - calculateeta_benchmark.cpp reduce_benchmark.cpp) + benchmarks + PRIVATE ndarray_benchmark.cpp ndview_benchmark.cpp calculateeta_benchmark.cpp + reduce_benchmark.cpp clusterfile_benchmark.cpp) # Link Google Benchmark and other necessary libraries target_link_libraries(benchmarks PRIVATE benchmark::benchmark aare_core diff --git a/benchmarks/clusterfile_benchmark.cpp b/benchmarks/clusterfile_benchmark.cpp new file mode 100644 index 00000000..7d757532 --- /dev/null +++ b/benchmarks/clusterfile_benchmark.cpp @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/ClusterFile.hpp" + +#include +#include +#include +#include +#include + +class ClusterFileFixture : public benchmark::Fixture { + using Cluster = aare::Cluster; + using File = aare::ClusterFile; + using Vector = aare::ClusterVector; + + std::filesystem::path m_path; + static constexpr int32_t frame_count = 256; + static constexpr size_t chunk_size = 1000; + + static void observe(Vector &clusters) { + benchmark::DoNotOptimize(clusters.data()); + benchmark::DoNotOptimize(clusters.size()); + } + + public: + void SetUp(const benchmark::State &state) override { + const auto stamp = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_path = std::filesystem::temp_directory_path() / + ("aare-cluster-benchmark-" + std::to_string(stamp) + ".clust"); + File writer(m_path, chunk_size, "w"); + Vector frame(0); + frame.resize(state.range(0)); + for (int32_t number = 0; number < frame_count; ++number) { + frame.set_frame_number(number); + writer.write_frame(frame); + } + } + + void TearDown(const benchmark::State &) override { + std::error_code error; + std::filesystem::remove(m_path, error); + } + + template void read(benchmark::State &state) { + for (auto _ : state) { + File reader(m_path, chunk_size); + if constexpr (Iterate) { + auto consume = [](auto range) { + for (auto &clusters : range) { + observe(clusters); + } + }; + if constexpr (Frames) { + consume(reader.frames()); + } else { + consume(reader.chunks()); + } + } else if constexpr (Frames) { + Vector frame(0); + while (reader.read_frame(frame)) { + observe(frame); + } + } else { + while (true) { + auto chunk = reader.read_clusters(chunk_size); + if (chunk.empty()) { + break; + } + observe(chunk); + } + } + } + state.SetBytesProcessed(state.iterations() * frame_count * + state.range(0) * sizeof(Cluster)); + } +}; + +BENCHMARK_DEFINE_F(ClusterFileFixture, ReadFrames)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, IterateFrames)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, ReadChunks)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, IterateChunks)(benchmark::State &state) { + read(state); +} + +BENCHMARK_REGISTER_F(ClusterFileFixture, ReadFrames)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, IterateFrames)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, ReadChunks)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, IterateChunks)->Arg(64)->Arg(1024); diff --git a/docs/CMakeLists.txt b/docs/CMakeLists.txt index df589419..d83b6f29 100644 --- a/docs/CMakeLists.txt +++ b/docs/CMakeLists.txt @@ -36,7 +36,7 @@ add_custom_target( docs COMMAND ${DOXYGEN_EXECUTABLE} ${DOXYGEN_OUT} COMMAND - ${SPHINX_EXECUTABLE} -a -b html + ${SPHINX_EXECUTABLE} -W -a -b html -Dbreathe_projects.aare=${CMAKE_CURRENT_BINARY_DIR}/xml -c "${SPHINX_BUILD}" ${SPHINX_BUILD}/src ${SPHINX_BUILD}/html COMMENT "Generating documentation with Sphinx") diff --git a/docs/src/ClusterFile.rst b/docs/src/ClusterFile.rst index a2ee162b..d764b687 100644 --- a/docs/src/ClusterFile.rst +++ b/docs/src/ClusterFile.rst @@ -1,8 +1,40 @@ ClusterFile ============= +Use ``frames()`` to iterate over complete frames, including empty frames and +frames whose clusters are all rejected by ROI or noise filtering. Each result +keeps its stored frame number; missing frame numbers are not synthesized. + +.. code-block:: cpp + + using ClusterType = aare::Cluster; + aare::ClusterFile file("clusters.clust"); + for (auto &frame : file.frames()) { + process_frame(frame.frame_number(), frame); + } + +Use ``chunks()`` for the constructor's chunk size, or ``chunks(10000)`` to +request a different size for that traversal. The size must be positive and +counts selected clusters after filtering. Chunks may split or combine frames, +so their frame numbers are not reliable per-cluster metadata. Empty chunks are +not yielded; only the final chunk can contain fewer clusters than requested. +Both ranges apply the configured gain map and report incomplete files as +errors, just like the explicit read methods. + +Ranges borrow the file and consume its current position without rewinding. +Constructing a range does not read; ``begin()`` reads the first result and +increment reads the next. Breaking a loop does not read ahead. Use one +traversal at a time and separate files for independent cursors. Switching to +frame reads after a chunk stops partway through a frame raises an error until +the remaining clusters in that frame have been read. + +The file must outlive its ranges and iterators and must not be moved while +they are in use. Iterators are move only and support C++17 range-based loops, +not algorithms requiring copyable STL input iterators. References to a result +last until advancement or iterator destruction. Move a result out with +``auto retained = std::move(frame)`` to retain it. Frame iteration reuses the +current vector's storage when it has not been moved out. + .. doxygenclass:: aare::ClusterFile :members: :undoc-members: - :private-members: - diff --git a/docs/src/RawFile.rst b/docs/src/RawFile.rst index 4b7e03ae..8ecd30e9 100644 --- a/docs/src/RawFile.rst +++ b/docs/src/RawFile.rst @@ -1,8 +1,48 @@ RawFile =============== +Reading requires frame padding to be enabled +or ``Frame Discard Policy`` to be ``discardpartial``. +The constructor checks the +parsed master metadata before opening data subfiles and throws +``std::runtime_error`` with the master path if this requirement is not met. +This applies to JSON and legacy ``.raw`` masters, including when opened through +``aare::File``. ``RawMasterFile`` can still be used to inspect their metadata. + +``total_frames()`` is the minimum actual frame count across all selected +``RawSubFile`` objects in all ROIs. Each subfile count includes its complete +series of ``.raw`` files and is determined when the file is opened. Disabled +UDP ports are excluded. An empty subfile makes the count zero; opening a file +with no selected subfiles raises an error. + +A warning is logged once on opening if the subfile counts differ or their +minimum differs from ``master().frames_in_file()``. It reports the subfile count +range, the master count, and the count used for reading. The master metadata +is preserved. Its expected frame count does not affect the count used for +reading. + +Frame, batch, ROI, and frame-number reads use this minimum as their bound. +Frame-number synchronization may still fail within this bound if detector +frame numbers differ between subfiles. + +Read errors include the attempted zero-based frame index and the failing file +path. Errors from a data subfile identify that subfile's path and preserve +the underlying failure reason without adding a master-file wrapper or C++ +source location. Out-of-range subfile errors also report the available frame +count across that subfile's series. This applies to frame, batch, ROI, and +frame-number reads, including reads through ``aare::File``. + +The frame index is the position in the file, not the frame number recorded by +the detector. Batch errors report the index of the frame that failed. + +Synchronization errors identify the last ``.raw`` data file read for the module +that ran out of frames. Requests rejected as out of range before accessing a +subfile report the master path, since there is no single failing data file. +The top-level frame bounds error states the number of frames in the file and +that indices are zero-based. + .. doxygenclass:: aare::RawFile :members: :undoc-members: - :private-members: \ No newline at end of file + :private-members: diff --git a/docs/src/python/cluster/pyClusterVector.rst b/docs/src/python/cluster/pyClusterVector.rst index 1aaf1b78..fc9508c3 100644 --- a/docs/src/python/cluster/pyClusterVector.rst +++ b/docs/src/python/cluster/pyClusterVector.rst @@ -3,17 +3,16 @@ ClusterVector ================ -The ClusterVector, holds clusters from the ClusterFinder. Since it is templated -in C++ we use a suffix indicating the type of cluster it holds. The suffix follows -the same pattern as for ClusterFile i.e. ``ClusterVector_Cluster3x3i`` -for a vector holding 3x3 integer clusters. +A ClusterVector stores fixed-size clusters contiguously. Since it is templated +in C++, each bound class has a suffix indicating the cluster type. The suffix +follows the same pattern as ClusterFile; for example, +``ClusterVector_Cluster3x3i`` stores 3x3 clusters with 32-bit integer pixels. -At the moment the functionality from python is limited and it is not supported -to push_back clusters to the vector. The intended use case is to pass it to -C++ functions that support the ClusterVector or to view it as a numpy array. +The intended use case is to pass a ClusterVector to C++ functions that support +it or to view it as a NumPy array. -**View ClusterVector as numpy array** +**View ClusterVector as a NumPy array** .. code:: python @@ -25,7 +24,16 @@ C++ functions that support the ClusterVector or to view it as a numpy array. clusters = np.array(cluster_vector) # Avoid copying the data by passing copy=False - clusters = np.array(cluster_vector, copy = False) + clusters = np.array(cluster_vector, copy=False) + +.. warning:: + + A NumPy array created with ``copy=False`` is a view of the ClusterVector's + current storage. Do not call ``push_back`` or otherwise change the + ClusterVector while using the view. A ``push_back`` that reallocates the + backing buffer leaves existing NumPy views pointing to invalid memory, and + appending without reallocation does not update their shape. Use + ``copy=True`` if the ClusterVector may change after creating the array. .. py:currentmodule:: aare @@ -35,7 +43,8 @@ C++ functions that support the ClusterVector or to view it as a numpy array. :undoc-members: :inherited-members: -Below is the API of the ClusterVector_Cluster3x3i but all variants share the same API. +Below is the API of ``ClusterVector_Cluster3x3i``. All variants share the same +API. .. autoclass:: aare._aare.ClusterVector_Cluster3x3i :special-members: __init__, __call__ @@ -45,14 +54,39 @@ Below is the API of the ClusterVector_Cluster3x3i but all variants share the sam :inherited-members: -**Free Functions:** +**Free Functions:** -.. autofunction:: reduce_to_3x3 +.. py:function:: hitmap(image_size, clusters) :noindex: - Reduce a single Cluster to 3x3 by taking the 3x3 subcluster with highest photon energy. + Count cluster centers into an ``int32`` image. ``image_size`` is given as + ``(rows, columns)``, and output element ``[y, x]`` contains the number of + photon hits at that coordinate. Out-of-bounds hits are ignored. All registered + ClusterVector variants are accepted. -.. autofunction:: reduce_to_2x2 + :param tuple[int, int] image_size: Shape of the output image. + :param ClusterVector clusters: Clusters whose centers are counted. + :return: Hit counts with shape ``image_size``. + :rtype: numpy.ndarray + +.. py:function:: reduce_to_3x3(clustervector) :noindex: - Reduce a single Cluster to 2x2 by taking the 2x2 subcluster with highest photon energy. + Return a new vector containing the central 3x3 block of every input cluster. + Cluster order, coordinates, frame number, and pixel dtype are preserved. + + :param ClusterVector clustervector: Input clusters with both dimensions at + least 3 and at least one dimension greater than 3. + :return: Reduced 3x3 clusters. + :rtype: ClusterVector + +.. py:function:: reduce_to_2x2(clustervector) + :noindex: + + Return a new vector containing the highest-sum center-adjacent 2x2 block of + every input cluster. Cluster order, coordinates, frame number, and pixel + dtype are preserved. + + :param ClusterVector clustervector: Input clusters of size 2x2 or larger. + :return: Reduced 2x2 clusters. + :rtype: ClusterVector diff --git a/docs/src/python/file/pyClusterFile.rst b/docs/src/python/file/pyClusterFile.rst index cd391e0e..b406cab3 100644 --- a/docs/src/python/file/pyClusterFile.rst +++ b/docs/src/python/file/pyClusterFile.rst @@ -1,26 +1,78 @@ ClusterFile -============ +=========== +The :func:`ClusterFile` factory is the main interface for reading and writing +cluster files. Use ``mode="r"`` to read, ``mode="w"`` to truncate and +write, or ``mode="a"`` to append. -The :class:`ClusterFile` class is the main interface to read and write clusters in aare. Unfortunately the -old file format does not include metadata like the cluster size and the data type. This means that the -user has to know this information from other sources. Specifying the wrong cluster size or data type -will lead to garbage data being read. +The format does not store the cluster dimensions, value type, coordinate type, +padding, or byte order. The ``cluster_size`` and ``dtype`` arguments must match +the writer; otherwise the bytes are interpreted incorrectly. + +Use ``frames()`` when frame boundaries and the frame number matter: + +.. code-block:: python + + with ClusterFile("clusters.clust") as file: + for frame in file.frames(): + print(frame.frame_number, frame.size) + +Each stored frame is yielded, including empty frames and frames whose clusters +are all rejected by filtering. Missing frame numbers are not synthesized. +The explicit ``read_frame()`` method returns one frame or ``None`` at EOF. + +Use ``chunks()`` to iterate over chunks of ``chunk_size`` clusters. Per default +the chunk size defined in the constructor is used. Alternatively one can pass +an explicit chunk size to the function for that specific traversal: + +.. code-block:: python + + with ClusterFile("clusters.clust", chunk_size=1000) as file: + for clusters in file.chunks(10000): + process(clusters) + +Chunk sizes must be positive and count selected clusters after filtering. +``file.chunks(10000)`` does not overwrite the constructor argument ``chunk_size`` +. Iterating directly +over ``file`` or calling ``next(file)`` still reads chunks using that default. +Chunks and ``read_clusters(n_clusters)`` may split or combine frames, so their +frame numbers are not reliable per-cluster metadata. Empty chunks are not +yielded. +Chunk reads and iteration raise an error if they encounter an incomplete frame +header or cluster record, including when ROI or noise filtering is enabled. +They return fewer clusters than requested only at a clean end of file. + +Both iterators consume the current file position without rewinding. Creating +an iterator does not read; each ``next()`` reads one result without prefetching +the following result. Use one traversal at a time and separate readers for +independent cursors. Switching to frames after a chunk stops partway through a +frame raises an error until the remaining clusters in that frame have been +read. Exhausted iterators continue to raise ``StopIteration``. + +Iterators keep their file object alive, but explicitly closing the file or +leaving its ``with`` block prevents further reads. Every yielded +``ClusterVector`` owns its storage and can be retained after iteration advances +or the file closes. NumPy views also remain valid across those operations; +as usual, resizing or modifying the allocation of their backing vector can +invalidate them. + +When a gain map is configured, it is applied to every cluster whose complete +footprint lies inside the map. Clusters whose footprint extends beyond the +gain-map boundaries remain in the returned cluster vector, but all their data +values are set to zero. .. py:currentmodule:: aare -.. autoclass:: ClusterFile - :members: - :undoc-members: - :inherited-members: +.. autofunction:: ClusterFile -Below is the API of the ClusterFile_Cluster3x3i but all variants share the same API. +Below is the API of ``ClusterFile_Cluster3x3i``; all compiled variants share the +same API. .. autoclass:: aare._aare.ClusterFile_Cluster3x3i :special-members: __init__ :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/file/pyCtbRawFile.rst b/docs/src/python/file/pyCtbRawFile.rst index 75d83550..f0676e6d 100644 --- a/docs/src/python/file/pyCtbRawFile.rst +++ b/docs/src/python/file/pyCtbRawFile.rst @@ -6,6 +6,10 @@ Read analog, digital and transceiver samples from a raw file containing data from the Chip Test Board. Uses :mod:`aare.transform` to decode the data into a format that the user can work with. +``read_frame(index)`` reads the frame at the zero-based index across the raw +subfiles and advances the position to ``index + 1``. A subsequent +``read_frame()`` reads the following frame, including across subfile boundaries. + .. code:: python import aare @@ -22,4 +26,4 @@ data into a format that the user can work with. :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/file/pyRawFile.rst b/docs/src/python/file/pyRawFile.rst index b81f4138..a128d2fe 100644 --- a/docs/src/python/file/pyRawFile.rst +++ b/docs/src/python/file/pyRawFile.rst @@ -3,8 +3,45 @@ RawFile .. py:currentmodule:: aare +Reading requires frame padding to be enabled +or ``Frame Discard Policy`` to be ``discardpartial``. Construction raises +``RuntimeError`` with the master path before opening data subfiles if this +requirement is not met. This applies to JSON and legacy ``.raw`` masters, +including when opened through ``File``. ``RawMasterFile`` can still be used +to inspect their metadata. + +``total_frames`` and ``len(reader)`` report the minimum actual frame count +across all selected subfiles and ROIs, including each subfile's complete +``.raw`` series. ``read()`` uses this count, and ``read_n()`` and +``read_n_with_roi()`` limit batches to the remaining frames. Counts are +determined when the reader is constructed. +An empty subfile makes the count zero, and construction fails if no subfiles +are selected. + +The C++ logger prints one warning to stderr on opening if the subfile counts +differ or their minimum differs from ``reader.master.frames_in_file``. The +warning includes the count range, the recorded master count, and the count +used for reading. Master metadata is preserved; its expected frame count +does not affect reading bounds. Detector frame-number mismatches can still +cause synchronization errors within these bounds. + +Reading errors raise ``RuntimeError`` with the attempted zero-based frame index +and the failing file path. If a data subfile fails, its path and the underlying +failure reason are reported. Out-of-range subfile errors include the available frame count across +that subfile's series. This applies to ``read_frame()``, ``read_n()``, +and the ROI reading methods. The frame index is the position in the file, not +the frame number recorded by the detector. + +Synchronization errors also identify the last ``.raw`` data file read for the +module that ran out of frames. An out-of-range request rejected before accessing +a subfile reports the master path. + +For a file containing five frames, the valid indices are 0 through 4. Attempting +to read index 5 raises an error stating: ``Frame index 5 is out of range: file +contains 5 frames (indices are zero-based)``. + .. autoclass:: RawFile :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/pedestal/pyFastPedestal.rst b/docs/src/python/pedestal/pyFastPedestal.rst index d442d298..fcc57d33 100644 --- a/docs/src/python/pedestal/pyFastPedestal.rst +++ b/docs/src/python/pedestal/pyFastPedestal.rst @@ -1,7 +1,7 @@ FastPedestal ============ -``FastPedestal`` calculates a running mean, variance and standard deviation for each pixel in a +``FastPedestal`` calculates a running mean and standard deviation for each pixel in a series of frames. The python binding only exposes ``uint16`` input but the underlying C++ class is templated. Initialize it with ``n_samples`` frames using ``add_init_frame()``. Once ``ready`` is true, use ``push_ema()`` to update the exponential @@ -18,7 +18,10 @@ The public factory selects the bound C++ specialization from ``dtype``: * ``numpy.float32`` creates ``FastPedestal_f`` * ``numpy.int16`` creates ``FastPedestal_i16`` -The internal calculations are done with double, but the cached mean and on demand var and std are returned in the specified type. +Internal moments and variance are calculated in double precision. Variance is +private and stays in double precision through the square root; the cached mean +and on-demand standard deviation are returned in the specified type. Negative +variance caused by floating-point roundoff is clamped to zero. Factory ------- diff --git a/docs/src/python/pedestal/pyPedestal.rst b/docs/src/python/pedestal/pyPedestal.rst index 7d6aecb2..00000b19 100644 --- a/docs/src/python/pedestal/pyPedestal.rst +++ b/docs/src/python/pedestal/pyPedestal.rst @@ -1,30 +1,57 @@ Pedestal ======== -``Pedestal`` calculates a running mean and variance for each pixel in a series -of ``uint16`` frames. ``push()`` updates the cached mean immediately. For -faster batch initialization, use ``push_no_update()`` for each frame and call -``update_mean()`` after the batch. +``Pedestal`` calculates a running mean and population standard deviation for +each pixel in a series of ``uint16`` frames. ``push()`` updates the cached mean +immediately; ``std()`` calculates the noise from the current statistics. -Three specializations are available from :mod:`aare`: +``push()`` and ``push_with_threshold()`` require C-contiguous, two-dimensional +NumPy frames with dtype ``uint16`` and shape matching the pedestal. +``push_with_threshold()`` also requires a C-contiguous, two-dimensional +threshold array with the pedestal's output dtype and shape. Noncontiguous +inputs raise ``TypeError``; inputs with the wrong number of dimensions raise +``ValueError``. Mismatched frame or threshold shapes raise ``RuntimeError``. +Use ``numpy.ascontiguousarray()`` to copy a sliced or transposed array into the +required layout when needed. -* ``Pedestal_d`` uses ``float64`` storage -* ``Pedestal_f`` uses ``float32`` storage -* ``Pedestal_i16`` uses ``int16`` storage +Internal sums and sums of squares always use ``float64``. Three +specializations are available from :mod:`aare` for the mean and standard +deviation output types: + +* ``Pedestal_d`` returns ``float64`` +* ``Pedestal_f`` returns ``float32`` +* ``Pedestal_i16`` returns ``int16`` + +The public ``Pedestal`` factory selects the specialization from ``dtype``, +defaulting to ``numpy.float64``. + +Constructor dimensions and ``n_samples`` must be positive integers. Negative +values raise ``TypeError`` and zero values raise ``RuntimeError``. +Internally, negative variance caused by floating-point roundoff is clamped to +zero before taking its square root, keeping the standard deviation finite for +nearly constant inputs. Only the final standard deviation is converted to the +output dtype. + +Factory +------- + +.. py:currentmodule:: aare + +.. autofunction:: Pedestal Example ------- .. code-block:: python - from aare import Pedestal_d + import numpy as np + from aare import Pedestal - pedestal = Pedestal_d(512, 1024, 100) + pedestal = Pedestal(512, 1024, n_samples=100, dtype=np.float32) for frame in initialization_frames: - pedestal.push_no_update(frame) + pedestal.push(frame) - pedestal.update_mean() mean = pedestal.mean() noise = pedestal.std() diff --git a/include/aare/ArrayExpr.hpp b/include/aare/ArrayExpr.hpp index 368f691f..8eab0d2e 100644 --- a/include/aare/ArrayExpr.hpp +++ b/include/aare/ArrayExpr.hpp @@ -29,7 +29,7 @@ class ArrayAdd : public ArrayExpr, Ndim> { ArrayAdd(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { assert(arr1.size() == arr2.size()); } - auto operator[](int i) const { return arr1_[i] + arr2_[i]; } + auto operator[](size_t i) const { return arr1_[i] + arr2_[i]; } size_t size() const { return arr1_.size(); } std::array shape() const { return arr1_.shape(); } }; @@ -43,7 +43,7 @@ class ArraySub : public ArrayExpr, Ndim> { ArraySub(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { assert(arr1.size() == arr2.size()); } - auto operator[](int i) const { return arr1_[i] - arr2_[i]; } + auto operator[](size_t i) const { return arr1_[i] - arr2_[i]; } size_t size() const { return arr1_.size(); } std::array shape() const { return arr1_.shape(); } }; @@ -57,7 +57,7 @@ class ArrayMul : public ArrayExpr, Ndim> { ArrayMul(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { assert(arr1.size() == arr2.size()); } - auto operator[](int i) const { return arr1_[i] * arr2_[i]; } + auto operator[](size_t i) const { return arr1_[i] * arr2_[i]; } size_t size() const { return arr1_.size(); } std::array shape() const { return arr1_.shape(); } }; @@ -71,7 +71,7 @@ class ArrayDiv : public ArrayExpr, Ndim> { ArrayDiv(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { assert(arr1.size() == arr2.size()); } - auto operator[](int i) const { return arr1_[i] / arr2_[i]; } + auto operator[](size_t i) const { return arr1_[i] / arr2_[i]; } size_t size() const { return arr1_.size(); } std::array shape() const { return arr1_.shape(); } }; diff --git a/include/aare/Cluster.hpp b/include/aare/Cluster.hpp index 117e587d..a601b85a 100755 --- a/include/aare/Cluster.hpp +++ b/include/aare/Cluster.hpp @@ -52,9 +52,9 @@ struct Cluster { // TODO: handle 1 dimensional clusters /** - * @brief sum of 2x2 subcluster with highest energy - * @return photon energy of subcluster, 2x2 subcluster index relative to - * cluster center + * @brief Find the highest-sum center-adjacent 2x2 subcluster. + * @return Sum and corner index of the selected 2x2 subcluster relative to + * the cluster center */ Sum_index_pair max_sum_2x2() const { @@ -117,13 +117,10 @@ struct Cluster { }; /** - * @brief Reduce a cluster to a 2x2 cluster by selecting the 2x2 block with the - * highest sum. + * @brief Reduce a cluster to its highest-sum center-adjacent 2x2 block. * @param c Cluster to reduce - * @return reduced cluster + * @return Reduced cluster with the input coordinates * @note The cluster is filled using row major ordering starting at the top-left - * (thus for a max subcluster in the top left cornern the photon hit is at - * the fourth position) */ template @@ -194,17 +191,21 @@ Cluster reduce_to_2x2(const Cluster &c) { } /** - * @brief Reduce a cluster to a 3x3 cluster + * @brief Reduce a cluster to the 3x3 block around its center index. * @param c Cluster to reduce - * @return reduced cluster + * @pre ClusterSizeX and ClusterSizeY must both be at least 3, and at least one + * must be greater than 3. + * @return Reduced cluster with the input coordinates are preserved. */ template Cluster reduce_to_3x3(const Cluster &c) { - static_assert(ClusterSizeX >= 3 && ClusterSizeY >= 3, - "Cluster sizes must be at least 3x3 for reduction to 3x3"); + static_assert(ClusterSizeX >= 3 && ClusterSizeY >= 3 && + (ClusterSizeX > 3 || ClusterSizeY > 3), + "Cluster sizes must both be at least 3, and at least one " + "must be greater than 3 for reduction to 3x3"); Cluster result{}; diff --git a/include/aare/ClusterFile.hpp b/include/aare/ClusterFile.hpp index 4e416c23..507e2c9c 100644 --- a/include/aare/ClusterFile.hpp +++ b/include/aare/ClusterFile.hpp @@ -3,46 +3,39 @@ #include "aare/Cluster.hpp" #include "aare/ClusterVector.hpp" +#include "aare/FilePtr.hpp" #include "aare/GainMap.hpp" #include "aare/NDArray.hpp" -#include "aare/defs.hpp" +#include "aare/ROI.hpp" #include "aare/logger.hpp" #include #include #include +#include +#include namespace aare { -/* -Binary cluster file. Expects data to be laid out as: -int32_t frame_number -uint32_t number_of_clusters -int16_t x, int16_t y, int32_t data[9] x number_of_clusters -int32_t frame_number -uint32_t number_of_clusters -.... -*/ - -// TODO: change to support any type of clusters, e.g. header line with -// clsuter_size_x, cluster_size_y, /** - * @brief Class to read and write cluster files - * Expects data to be laid out as: + * @brief Read and write legacy binary cluster files. * + * Each frame is stored as: * * int32_t frame_number * uint32_t number_of_clusters - * int16_t x, int16_t y, int32_t data[9] * number_of_clusters - * int32_t frame_number - * uint32_t number_of_clusters - * etc. + * ClusterType clusters[number_of_clusters] + * + * The format stores clusters as their native in-memory representation and has + * no metadata describing the cluster dimensions, value type, coordinate type, + * padding, or byte order. Readers must therefore use the same ClusterType and + * a compatible platform ABI as the writer. */ template >> class ClusterFile { - FILE *fp{}; - const std::string m_filename{}; + FilePtr m_fp; + std::string m_filename{}; uint32_t m_num_left{}; /*Number of photons left in frame*/ size_t m_chunk_size{}; /*Number of clusters to read at a time*/ std::string m_mode; /*Mode to open the file in*/ @@ -54,48 +47,150 @@ class ClusterFile { public: /** - * @brief Construct a new Cluster File object - * @param fname path to the file - * @param chunk_size number of clusters to read at a time when iterating - * over the file - * @param mode mode to open the file in. "r" for reading, "w" for writing, - * "a" for appending - * @throws std::runtime_error if the file could not be opened + * @brief Single-pass range over frames or chunks of selected clusters. + * + * The range borrows the file and shares its current position with all + * other reads. Use only one traversal at a time. The file must outlive the + * range and its iterators and must not be moved while they are in use. + * Constructing a range does not read; begin() reads the first result. + * Calling begin() again starts at the file's then-current position. + */ + template class ReadRange { + ClusterFile *m_file; + size_t m_chunk_size; + + friend class ClusterFile; + ReadRange(ClusterFile &file, size_t chunk_size) + : m_file(&file), m_chunk_size(chunk_size) {} + + public: + struct Sentinel {}; + + /** + * @brief Move-only iterator for C++17 range-based loops. + * + * References to the current vector are valid until the iterator is + * advanced or destroyed. Move the vector out to retain its storage. + * Frame iteration reuses storage when the vector is not moved out. + * This is not a copyable STL input iterator. + */ + class Iterator { + ClusterFile *m_file; + size_t m_chunk_size; + ClusterVector m_clusters{0}; + + friend class ReadRange; + Iterator(ClusterFile &file, size_t chunk_size) + : m_file(&file), m_chunk_size(chunk_size) { + ++*this; + } + + public: + using value_type = ClusterVector; + + Iterator(const Iterator &) = delete; + Iterator &operator=(const Iterator &) = delete; + Iterator(Iterator &&) noexcept = default; + Iterator &operator=(Iterator &&) noexcept = default; + ~Iterator() = default; + + value_type &operator*() { return m_clusters; } + value_type *operator->() { return &m_clusters; } + + Iterator &operator++() { + if (m_file) { + if constexpr (ByFrame) { + if (!m_file->read_frame(m_clusters)) { + m_file = nullptr; + } + } else { + m_clusters = m_file->read_clusters(m_chunk_size); + if (m_clusters.empty()) { + m_file = nullptr; + } + } + } + return *this; + } + + void operator++(int) { ++*this; } + + friend bool operator==(const Iterator &it, Sentinel /*end*/) { + return it.m_file == nullptr; + } + friend bool operator!=(const Iterator &it, Sentinel end) { + return !(it == end); + } + friend bool operator==(Sentinel end, const Iterator &it) { + return it == end; + } + friend bool operator!=(Sentinel end, const Iterator &it) { + return !(it == end); + } + }; + + Iterator begin() const { return Iterator(*m_file, m_chunk_size); } + Sentinel end() const { return {}; } + }; + + using FrameRange = ReadRange; + using ChunkRange = ReadRange; + + /** + * @brief Open a cluster file. + * @param fname Path to the file. + * @param chunk_size Maximum number of selected clusters per chunk step. + * @param mode File mode: "r" to read, "w" to truncate and write, or "a" + * to append. + * @throws std::runtime_error If the mode is unsupported or the file cannot + * be opened. */ ClusterFile(const std::filesystem::path &fname, size_t chunk_size = 1000, const std::string &mode = "r") - : m_filename(fname.string()), m_chunk_size(chunk_size), m_mode(mode) { - - if (mode == "r") { - fp = fopen(m_filename.c_str(), "rb"); - if (!fp) { - throw std::runtime_error("Could not open file for reading: " + - m_filename); - } - } else if (mode == "w") { - fp = fopen(m_filename.c_str(), "wb"); - if (!fp) { - throw std::runtime_error("Could not open file for writing: " + - m_filename); - } - } else if (mode == "a") { - fp = fopen(m_filename.c_str(), "ab"); - if (!fp) { - throw std::runtime_error("Could not open file for appending: " + - m_filename); - } - } else { - throw std::runtime_error("Unsupported mode: " + mode); - } + : m_filename(fname.string()), m_chunk_size(chunk_size) { + open(mode); } - ~ClusterFile() { close(); } + /** + * @brief Iterate over complete frames from the current file position. + * + * Empty and fully filtered frames are yielded with their stored frame + * numbers. Reading stops only at a clean end of file. Advancing uses + * read_frame(), including its error for a prior partial-frame read. + */ + FrameRange frames() & { return FrameRange(*this, 0); } + FrameRange frames() && = delete; + + /** @brief Iterate using the chunk size supplied to the constructor. */ + ChunkRange chunks() & { return chunks(m_chunk_size); } + ChunkRange chunks() && = delete; /** - * @brief Read n_clusters clusters from the file discarding - * frame numbers. If EOF is reached the returned vector will - * have less than n_clusters clusters + * @brief Iterate over chunks from the current file position. + * @param chunk_size Maximum number of selected clusters per step. + * @throws std::invalid_argument If chunk_size is zero. + * @note Chunks can span frames; their frame numbers are not per-cluster + * metadata. Advancing uses read_clusters() and propagates its errors. + * Only the final chunk can be short. Empty chunks are not yielded. + */ + ChunkRange chunks(size_t chunk_size) & { + if (chunk_size == 0) { + throw std::invalid_argument("Chunk size must be greater than zero"); + } + return ChunkRange(*this, chunk_size); + } + ChunkRange chunks(size_t) && = delete; + + /** + * @brief Read up to n_clusters without preserving frame boundaries. + * @param n_clusters Maximum number of selected clusters to return. + * @return A cluster vector that may contain fewer clusters at a clean end + * of file. + * @note The returned vector may combine data from several frames, so its + * frame number must not be used as per-cluster metadata. + * @throws std::runtime_error If the file is not open for reading, an I/O + * error occurs, or an incomplete frame header or cluster record is read. */ ClusterVector read_clusters(size_t n_clusters) { if (m_mode != "r") { @@ -109,76 +204,116 @@ class ClusterFile { } /** - * @brief Read a single frame from the file and return the - * clusters. The cluster vector will have the frame number - * set. - * @throws std::runtime_error if the file is not opened for - * reading or the file pointer not at the beginning of a - * frame + * @brief Read the next complete frame. + * @return The selected clusters with the stored frame number set, or + * std::nullopt at a clean end of file before the next frame. + * @throws std::runtime_error If the file is not open for reading, a prior + * partial-frame read left clusters unread, or the frame is incomplete. + * @note A complete frame produces an engaged optional even when it contains + * no clusters or all of its clusters are removed by the configured filters. */ - ClusterVector read_frame() { + std::optional> read_frame() { + ClusterVector clusters(0); + if (!read_frame(clusters)) { + return std::nullopt; + } + return std::optional>{std::move(clusters)}; + } + + /** + * @brief Read the next complete frame into an existing cluster vector. + * @param clusters Destination whose storage is reused when large enough. + * Existing clusters are replaced after a frame header is read and remain + * unchanged at a clean end of file. + * @return true when a complete frame was read, or false at a clean end of + * file before the next frame. + * @throws std::runtime_error If the file is not open for reading, a prior + * partial-frame read left clusters unread, or a frame is incomplete. + * @note A complete frame is a successful read even when it contains no + * clusters or all of its clusters are removed by the configured filters. + */ + bool read_frame(ClusterVector &clusters) { if (m_mode != "r") { throw std::runtime_error(LOCATION + "File not opened for reading"); } - if (m_noise_map || m_roi) { - return read_frame_with_cut(); - } else { - return read_frame_without_cut(); + if (m_num_left) { + throw std::runtime_error( + LOCATION + "There are still clusters left in the last frame"); } + + if (m_noise_map || m_roi) { + return read_frame_with_cut(clusters); + } + return read_frame_without_cut(clusters); } + /** + * @brief Write one frame to the file. + * @param clusters Clusters to write, including their frame number. + * @throws std::runtime_error If the file is not open for writing or any + * part of the frame cannot be written completely. + */ void write_frame(const ClusterVector &clusters) { if (m_mode != "w" && m_mode != "a") { throw std::runtime_error("File not opened for writing"); } int32_t frame_number = clusters.frame_number(); - fwrite(&frame_number, sizeof(frame_number), 1, fp); + if (fwrite(&frame_number, sizeof(frame_number), 1, m_fp.get()) != 1) { + throw std::runtime_error(LOCATION + "Could not write frame number"); + } + uint32_t n_clusters = clusters.size(); - fwrite(&n_clusters, sizeof(n_clusters), 1, fp); - fwrite(clusters.data(), clusters.item_size(), clusters.size(), fp); + if (fwrite(&n_clusters, sizeof(n_clusters), 1, m_fp.get()) != 1) { + throw std::runtime_error(LOCATION + + "Could not write number of clusters"); + } + + if (fwrite(clusters.data(), clusters.item_size(), clusters.size(), + m_fp.get()) != clusters.size()) { + throw std::runtime_error(LOCATION + "Could not write clusters"); + } } /** - * @brief Return the chunk size + * @brief Return the default number of selected clusters per chunk step. */ size_t chunk_size() const { return m_chunk_size; } /** - * @brief Estimate the number of clusters in the file from its size + * @brief Estimate the number of clusters in the file from its size. * - * Frame headers are included in the estimate, so the result may be slightly - * larger than the actual number of clusters. The file position is not - * changed. - * - * @throws std::runtime_error if the file is not opened for reading + * Frame-header bytes are included in the estimate, so it may exceed the + * actual number of clusters. The file position is not changed. */ size_t estimate_n_clusters() const { return std::filesystem::file_size(m_filename) / sizeof(ClusterType); } /** - * @brief Set the region of interest to use when reading - * clusters. If set only clusters within the ROI will be - * read. + * @brief Select clusters by their center coordinate when reading. + * @param roi Half-open region of interest: [xmin, xmax) x [ymin, ymax). */ void set_roi(ROI roi) { m_roi = roi; } /** - * @brief Set the noise map to use when reading clusters. If - * set clusters below the noise level will be discarded. - * Selection criteria one of: Central pixel above noise, - * highest 2x2 sum above 2 * noise, total sum above 3 * - * noise. + * @brief Discard clusters that do not pass the noise thresholds. + * @param noise_map Per-pixel noise indexed as [y, x]. The map is copied. + * A cluster is retained only when its central pixel exceeds the local + * noise, its highest 2x2 sum exceeds twice the noise, and its total sum + * exceeds three times the noise. + * @warning The map must cover every cluster center coordinate in the file. */ void set_noise_map(const NDView noise_map) { m_noise_map = NDArray(noise_map); } /** - * @brief Set the gain map to use when reading clusters. If set the gain map - * will be applied to the clusters that pass ROI and noise_map selection. - * The gain map is expected to be in ADU/energy. + * @brief Apply a gain map to clusters selected while reading. + * @param gain_map Per-pixel gain in ADU/energy, indexed as [y, x]. The map + * is copied and inverted internally. + * @note Clusters whose complete footprint extends beyond the gain map are + * retained with all cluster data values set to zero. */ void set_gain_map(const NDView gain_map) { m_gain_map = InvertedGainMap(gain_map); @@ -193,65 +328,53 @@ class ClusterFile { } /** - * @brief Close the file. If not closed the file will be - * closed in the destructor + * @brief Close the file. + * + * Calling close more than once is safe. The destructor closes an open file + * automatically. */ void close() { - if (fp) { - fclose(fp); - fp = nullptr; - } + m_fp = FilePtr{}; + m_mode = ""; } /** - * @brief Return the current position in the file (bytes) + * @brief Return the current byte position in the file. + * @throws std::runtime_error If the file is closed or its position cannot + * be determined. */ int64_t tell() { - if (!fp) { + if (!m_fp.get()) { throw std::runtime_error(LOCATION + "File not opened"); } - return ftell(fp); + return m_fp.tell(); } + private: /** @brief Open the file in specific mode * */ void open(const std::string &mode) { - if (fp) { - close(); - } + close(); if (mode == "r") { - fp = fopen(m_filename.c_str(), "rb"); - if (!fp) { - throw std::runtime_error("Could not open file for reading: " + - m_filename); - } + m_fp = FilePtr(m_filename, "rb"); m_mode = "r"; } else if (mode == "w") { - fp = fopen(m_filename.c_str(), "wb"); - if (!fp) { - throw std::runtime_error("Could not open file for writing: " + - m_filename); - } + m_fp = FilePtr(m_filename, "wb"); m_mode = "w"; } else if (mode == "a") { - fp = fopen(m_filename.c_str(), "ab"); - if (!fp) { - throw std::runtime_error("Could not open file for appending: " + - m_filename); - } + m_fp = FilePtr(m_filename, "ab"); m_mode = "a"; } else { throw std::runtime_error("Unsupported mode: " + mode); } } - - private: ClusterVector read_clusters_with_cut(size_t n_clusters); ClusterVector read_clusters_without_cut(size_t n_clusters); - ClusterVector read_frame_with_cut(); - ClusterVector read_frame_without_cut(); + bool read_frame_header(int32_t &frame_number, uint32_t &n_clusters); + bool read_frame_with_cut(ClusterVector &clusters); + bool read_frame_without_cut(ClusterVector &clusters); bool is_selected(ClusterType &cl); ClusterType read_one_cluster(); }; @@ -281,25 +404,31 @@ ClusterFile::read_clusters_without_cut(size_t n_clusters) { } else { nn = nph; } - nph_read += fread((buf + nph_read), clusters.item_size(), nn, fp); + const auto count = + fread((buf + nph_read), clusters.item_size(), nn, m_fp.get()); + if (count != nn) { + throw std::runtime_error(LOCATION + "Could not read clusters"); + } + nph_read += count; m_num_left = nph - nn; // write back the number of photons left } if (nph_read < n_clusters) { // keep on reading frames and photons until reaching n_clusters - while (fread(&iframe, sizeof(iframe), 1, fp)) { + while (read_frame_header(iframe, nph)) { clusters.set_frame_number(iframe); - // read number of clusters in frame - if (fread(&nph, sizeof(nph), 1, fp)) { - if (nph > (n_clusters - nph_read)) - nn = n_clusters - nph_read; - else - nn = nph; + if (nph > (n_clusters - nph_read)) + nn = n_clusters - nph_read; + else + nn = nph; - nph_read += - fread((buf + nph_read), clusters.item_size(), nn, fp); - m_num_left = nph - nn; + const auto count = + fread((buf + nph_read), clusters.item_size(), nn, m_fp.get()); + if (count != nn) { + throw std::runtime_error(LOCATION + "Could not read clusters"); } + nph_read += count; + m_num_left = nph - nn; if (nph_read >= n_clusters) break; } @@ -339,16 +468,15 @@ ClusterFile::read_clusters_with_cut(size_t n_clusters) { } int32_t frame_number = 0; // frame number needs to be 4 bytes! - while (fread(&frame_number, sizeof(frame_number), 1, fp)) { - if (fread(&m_num_left, sizeof(m_num_left), 1, fp)) { - clusters.set_frame_number( - frame_number); // cluster vector will hold the last - // frame number - while (m_num_left && clusters.size() < n_clusters) { - ClusterType c = read_one_cluster(); - if (is_selected(c)) { - clusters.push_back(c); - } + uint32_t n_clusters_in_frame = 0; + while (read_frame_header(frame_number, n_clusters_in_frame)) { + m_num_left = n_clusters_in_frame; + clusters.set_frame_number( + frame_number); // cluster vector will hold the last frame number + while (m_num_left && clusters.size() < n_clusters) { + ClusterType c = read_one_cluster(); + if (is_selected(c)) { + clusters.push_back(c); } } @@ -366,7 +494,7 @@ ClusterFile::read_clusters_with_cut(size_t n_clusters) { template ClusterType ClusterFile::read_one_cluster() { ClusterType c; - auto rc = fread(&c, sizeof(c), 1, fp); + auto rc = fread(&c, sizeof(c), 1, m_fp.get()); if (rc != 1) { throw std::runtime_error(LOCATION + "Could not read cluster"); } @@ -375,73 +503,70 @@ ClusterType ClusterFile::read_one_cluster() { } template -ClusterVector -ClusterFile::read_frame_without_cut() { - if (m_mode != "r") { - throw std::runtime_error(LOCATION + "File not opened for reading"); +bool ClusterFile::read_frame_header(int32_t &frame_number, + uint32_t &n_clusters) { + const auto frame_number_bytes = + fread(&frame_number, 1, sizeof(frame_number), m_fp.get()); + if (frame_number_bytes == 0 && feof(m_fp.get()) && !ferror(m_fp.get())) { + return false; } - if (m_num_left) { - throw std::runtime_error( - LOCATION + "There are still photons left in the last frame"); - } - int32_t frame_number; - if (fread(&frame_number, sizeof(frame_number), 1, fp) != 1) { - if (feof(fp)) - throw std::runtime_error(LOCATION + "Unexpected end of file"); - else if (ferror(fp)) + if (frame_number_bytes != sizeof(frame_number)) { + if (ferror(m_fp.get())) { throw std::runtime_error(LOCATION + "Error reading from file"); - - throw std::runtime_error( - LOCATION + - "Unexpected error (not feof or ferror) when reading frame number"); + } + throw std::runtime_error(LOCATION + "Incomplete frame number"); } + const auto cluster_count_bytes = + fread(&n_clusters, 1, sizeof(n_clusters), m_fp.get()); + if (cluster_count_bytes != sizeof(n_clusters)) { + if (ferror(m_fp.get())) { + throw std::runtime_error(LOCATION + "Error reading from file"); + } + throw std::runtime_error(LOCATION + "Incomplete number of clusters"); + } + return true; +} + +template +bool ClusterFile::read_frame_without_cut( + ClusterVector &clusters) { + int32_t frame_number; uint32_t n_clusters; - if (fread(&n_clusters, sizeof(n_clusters), 1, fp) != 1) { - throw std::runtime_error(LOCATION + - "Could not read number of clusters"); + if (!read_frame_header(frame_number, n_clusters)) { + return false; } LOG(logDEBUG1) << "Reading " << n_clusters << " clusters from frame " << frame_number; - ClusterVector clusters(n_clusters); clusters.set_frame_number(frame_number); clusters.resize(n_clusters); LOG(logDEBUG1) << "clusters.item_size(): " << clusters.item_size(); - if (fread(clusters.data(), clusters.item_size(), n_clusters, fp) != + if (fread(clusters.data(), clusters.item_size(), n_clusters, m_fp.get()) != static_cast(n_clusters)) { throw std::runtime_error(LOCATION + "Could not read clusters"); } if (m_gain_map) m_gain_map->apply_gain_map(clusters); - return clusters; + return true; } template -ClusterVector -ClusterFile::read_frame_with_cut() { - if (m_mode != "r") { - throw std::runtime_error("File not opened for reading"); - } - if (m_num_left) { - throw std::runtime_error( - "There are still photons left in the last frame"); - } +bool ClusterFile::read_frame_with_cut( + ClusterVector &clusters) { int32_t frame_number; - if (fread(&frame_number, sizeof(frame_number), 1, fp) != 1) { - throw std::runtime_error("Could not read frame number"); + uint32_t n_clusters; + if (!read_frame_header(frame_number, n_clusters)) { + return false; } - if (fread(&m_num_left, sizeof(m_num_left), 1, fp) != 1) { - throw std::runtime_error("Could not read number of clusters"); - } - - ClusterVector clusters; - clusters.reserve(m_num_left); + m_num_left = n_clusters; + clusters.resize(0); + clusters.reserve(n_clusters); clusters.set_frame_number(frame_number); while (m_num_left) { ClusterType c = read_one_cluster(); @@ -451,7 +576,7 @@ ClusterFile::read_frame_with_cut() { } if (m_gain_map) m_gain_map->apply_gain_map(clusters); - return clusters; + return true; } template diff --git a/include/aare/ClusterFinder.hpp b/include/aare/ClusterFinder.hpp index 55c099f9..39b2fb25 100644 --- a/include/aare/ClusterFinder.hpp +++ b/include/aare/ClusterFinder.hpp @@ -44,18 +44,21 @@ class ClusterFinder { * @param image_size Image shape as (rows, columns). * @param nSigma Per-pixel noise threshold multiplier. * @param capacity Initial cluster-vector capacity. + * @param min_pedestal_samples Minimum number of pedestal frames to + * accumulate to get reasonable statistics */ ClusterFinder(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, - size_t capacity = 1000000) + size_t capacity = 1000000, size_t min_pedestal_samples = 1000) : m_image_size(image_size), m_nSigma(nSigma), c2(sqrt((ClusterSizeY + 1) / 2 * (ClusterSizeX + 1) / 2)), c3(sqrt(ClusterSizeX * ClusterSizeY)), - m_pedestal(image_size[0], image_size[1]), m_clusters(capacity), - m_threshold({image_size[0], image_size[1]}, 0), + m_pedestal(image_size[0], image_size[1], min_pedestal_samples), + m_clusters(capacity), m_threshold({image_size[0], image_size[1]}, 0), m_pd_corrected_frame({image_size[0], image_size[1]}, 0) { LOG(logDEBUG) << "ClusterFinder: " << "image_size: " << image_size[0] << "x" << image_size[1] - << ", nSigma: " << nSigma << ", capacity: " << capacity; + << ", nSigma: " << nSigma << ", capacity: " << capacity + << ", min_pedestal_samples: " << min_pedestal_samples; } /** diff --git a/include/aare/ClusterFinderMT.hpp b/include/aare/ClusterFinderMT.hpp index 5b61e8b9..7f1f3e01 100644 --- a/include/aare/ClusterFinderMT.hpp +++ b/include/aare/ClusterFinderMT.hpp @@ -131,6 +131,15 @@ class ClusterFinderMT { } } + bool output_queues_are_empty() const { + for (auto &q : m_output_queues) { + if (!q->isEmpty()) { + return false; + } + } + return true; + } + /** * @brief Collect all the clusters from the output queues and write them to * the sink @@ -191,16 +200,19 @@ class ClusterFinderMT { * allocated once and recycled, so the total resident frame memory is * n_threads * queue_depth * frame size. Keeping the in flight data below * the L3 size keeps the per frame copy cheap. + * @param min_pedestal_samples minimum number of pedestal samples to + * accumulate before using the pedestal */ ClusterFinderMT(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, size_t capacity = 2000, size_t n_threads = 3, - size_t queue_depth = 16) + size_t queue_depth = 16, size_t min_pedestal_samples = 1000) : m_n_threads(n_threads) { LOG(logDEBUG1) << "ClusterFinderMT: " << "image_size: " << image_size[0] << "x" << image_size[1] << ", nSigma: " << nSigma << ", capacity: " << capacity + << ", min_pedestal_samples: " << min_pedestal_samples << ", n_threads: " << n_threads << ", queue_depth: " << queue_depth; @@ -208,7 +220,7 @@ class ClusterFinderMT { m_cluster_finders.push_back( std::make_unique< ClusterFinder>( - image_size, nSigma, capacity)); + image_size, nSigma, capacity, min_pedestal_samples)); } for (size_t i = 0; i < n_threads; i++) { m_frame_pools.emplace_back( diff --git a/include/aare/ClusterVector.hpp b/include/aare/ClusterVector.hpp index a4141857..37956933 100644 --- a/include/aare/ClusterVector.hpp +++ b/include/aare/ClusterVector.hpp @@ -20,14 +20,19 @@ template > { /** * @brief Construct a new ClusterVector object - * @param capacity initial capacity of the buffer in number of clusters + * @param capacity minimum initial capacity in number of clusters * @param frame_number frame number of the clusters. Default is 0, which is * also used to indicate that the clusters come from many frames */ - ClusterVector(size_t capacity = 1024, uint64_t frame_number = 0) + ClusterVector(size_t capacity = 1024, int32_t frame_number = 0) : m_frame_number(frame_number) { m_data.reserve(capacity); } @@ -57,18 +62,24 @@ class ClusterVector> { ClusterVector &operator=(ClusterVector &&other) noexcept = default; /** - * @brief Create a copy of the clustervector by filtering clusters in the - * ClusterVector using a boolean mask. - * @param mask boolean 1d mask - * @return ClusterVector containing only the clusters where the mask is - * true + * @brief Return a filtered copy selected by a one-dimensional Boolean mask. + * @param mask boolean 1d mask true if element in ClusterVector will be + * included + * @return ClusterVector containing the selected clusters in their original + * order and with the original frame number + * @throws std::runtime_error if the mask length differs from size() */ ClusterVector operator()(NDView mask) { if (static_cast(mask.size()) != m_data.size()) { throw std::runtime_error( LOCATION + "Mask size does not match number of clusters"); } - ClusterVector result(capacity(), frame_number()); + if (m_data.empty()) { + return ClusterVector(0, frame_number()); + } + const auto selected = + static_cast(std::count(mask.begin(), mask.end(), true)); + ClusterVector result(selected, frame_number()); for (size_t i = 0; i < m_data.size(); ++i) { if (mask(i)) { result.push_back(m_data[i]); @@ -77,20 +88,9 @@ class ClusterVector> { return result; } - // // Move assignment operator - // ClusterVector &operator=(ClusterVector &&other) noexcept { - // if (this != &other) { - // m_data = other.m_data; - // m_frame_number = other.m_frame_number; - // other.m_data.clear(); - // other.m_frame_number = 0; - // } - // return *this; - // } - /** - * @brief Sum the pixels in each cluster - * @return std::vector vector of sums for each cluster + * @brief Sum the pixels in each cluster. + * @return One sum for every cluster, in container order */ std::vector sum() { std::vector sums(m_data.size()); @@ -103,9 +103,10 @@ class ClusterVector> { } /** - * @brief Sum the pixels in the 2x2 subcluster with the biggest pixel sum in - * each cluster - * @return vector of sums index pairs for each cluster + * @brief Find the highest-sum center-adjacent 2x2 subcluster in each + * cluster. + * @return One sum and corner-index pair for every cluster, in container + * order */ std::vector> sum_2x2() { std::vector> sums_2x2(m_data.size()); @@ -125,10 +126,29 @@ class ClusterVector> { */ void reserve(size_t capacity) { m_data.reserve(capacity); } + /** + * @brief Change the number of stored clusters. + * @param size new number of clusters + * @note Growing the vector value-initializes new clusters and can + * invalidate pointers, references, and iterators. + */ void resize(size_t size) { m_data.resize(size); } + /** + * @brief Append a cluster to the vector. + * @param cluster cluster to append + * @note Reallocation invalidates pointers, references, iterators, and + * zero-copy NumPy views of the storage. + */ void push_back(const ClusterType &cluster) { m_data.push_back(cluster); } + /** + * @brief Append all clusters from another vector. + * @param other vector whose clusters are appended + * @return Reference to this vector + * @note The frame number of this vector is unchanged. + * @warning other must not refer to this vector. + */ ClusterVector &operator+=(const ClusterVector &other) { m_data.insert(m_data.end(), other.begin(), other.end()); @@ -145,8 +165,10 @@ class ClusterVector> { */ bool empty() const { return m_data.empty(); } + /** @brief Return the cluster size in the x dimension. */ uint8_t cluster_size_x() const { return ClusterSizeX; } + /** @brief Return the cluster size in the y dimension. */ uint8_t cluster_size_y() const { return ClusterSizeY; } /** @@ -161,7 +183,7 @@ class ClusterVector> { auto end() const { return m_data.end(); } /** - * @brief Return the size in bytes of a single cluster + * @brief Return the size in bytes of one stored cluster, including padding. */ size_t item_size() const { return sizeof(ClusterType); // 2 * sizeof(CoordType) + ClusterSizeX * @@ -172,8 +194,8 @@ class ClusterVector> { ClusterType const *data() const { return m_data.data(); } /** - * @brief Return a reference to the i-th cluster casted to type V - * @tparam V type of the cluster + * @brief Return a reference to the i-th cluster without bounds checking. + * @param i zero-based cluster index */ ClusterType &operator[](size_t i) { return m_data[i]; } @@ -185,16 +207,20 @@ class ClusterVector> { */ int32_t frame_number() const { return m_frame_number; } + /** + * @brief Set the signed 32-bit frame number associated with the clusters. + * @param frame_number frame number, or 0 for clusters from multiple frames + */ void set_frame_number(int32_t frame_number) { m_frame_number = frame_number; } }; /** - * @brief Reduce a cluster to a 2x2 cluster by selecting the 2x2 block with the - * highest sum. - * @param cv Clustervector containing clusters to reduce - * @return Clustervector with reduced clusters + * @brief Reduce every cluster to its highest-sum center-adjacent 2x2 block. + * @param cv ClusterVector containing clusters to reduce + * @return ClusterVector of 2x2 clusters in the original order and with the + * original frame number * @note The cluster is filled using row major ordering starting at the top-left * (thus for a max subcluster in the top left cornern the photon hit is at * the fourth position) @@ -204,7 +230,8 @@ template > reduce_to_2x2( const ClusterVector> &cv) { - ClusterVector> result; + ClusterVector> result(cv.size(), + cv.frame_number()); for (const auto &c : cv) { result.push_back(reduce_to_2x2(c)); } @@ -212,20 +239,25 @@ ClusterVector> reduce_to_2x2( } /** - * @brief Reduce a cluster to a 3x3 cluster - * @param cv Clustervector containing clusters to reduce - * @return Clustervector with reduced clusters + * @brief Reduce every cluster to the 3x3 block around its center index. + * @param cv ClusterVector containing clusters to reduce + * @pre ClusterSizeX and ClusterSizeY must both be at least 3, and at least one + * must be greater than 3. + * @return ClusterVector of 3x3 clusters in the original order and with the + * original frame number + * @note Coordinates are preserved. */ template ClusterVector> reduce_to_3x3( const ClusterVector> &cv) { - ClusterVector> result; + ClusterVector> result(cv.size(), + cv.frame_number()); for (const auto &c : cv) { result.push_back(reduce_to_3x3(c)); } return result; } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/DetectorGeometry.hpp b/include/aare/DetectorGeometry.hpp index 8aa60783..e6718ffd 100644 --- a/include/aare/DetectorGeometry.hpp +++ b/include/aare/DetectorGeometry.hpp @@ -1,8 +1,7 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once +#include "aare/ROI.hpp" #include "aare/ROIGeometry.hpp" -#include "aare/RawMasterFile.hpp" //ROI refactor away -#include "aare/defs.hpp" #include namespace aare { @@ -97,6 +96,8 @@ class DetectorGeometry { const xy udp_interfaces_per_module = xy{1, 1}, const bool quad = false); + DetectorGeometry() = default; + ~DetectorGeometry() = default; /** @@ -114,6 +115,8 @@ class DetectorGeometry { size_t modules_x() const; size_t modules_y() const; + xy udp_interfaces_per_module() const; + const std::vector &get_module_geometries() const; const ModuleGeometry &get_module_geometries(const size_t index) const; @@ -125,6 +128,7 @@ class DetectorGeometry { size_t m_modules_y{}; size_t m_pixels_x{}; size_t m_pixels_y{}; + xy m_udp_interfaces_per_module{}; static constexpr ModuleConfig cfg{0, 0}; // TODO: maybe remove - should be a member in ROIGeometry - in particular diff --git a/include/aare/FastPedestal.hpp b/include/aare/FastPedestal.hpp index 48aec4bf..1a3579e8 100644 --- a/include/aare/FastPedestal.hpp +++ b/include/aare/FastPedestal.hpp @@ -4,20 +4,20 @@ #include "aare/Frame.hpp" #include "aare/NDArray.hpp" #include "aare/NDView.hpp" +#include #include #include namespace aare { /** - * @brief Maintain per-pixel mean, population variance, and standard deviation. + * @brief Maintain per-pixel mean and population standard deviation. * * Initialization accumulates exactly n_samples frames. Subsequent push_ema() * update the exponential moving average initialized with the mean using * a smoothing factor of (1/ n_samples). Internal moments are stored - * in double precision. - * @tparam PEDESTAL_TYPE Type returned for the mean, variance, and standard - * deviation. + * in double precision. Variance is a private double-precision intermediate. + * @tparam PEDESTAL_TYPE Type returned for the mean and standard deviation. */ template class FastPedestal { @@ -93,7 +93,7 @@ template class FastPedestal { * @brief Return a copy of the cached mean. * @throws std::runtime_error if ready() is false. */ - NDArray mean() { + NDArray mean() const { if (!ready()) { throw std::runtime_error( "Pedestal is not ready, cannot return mean"); @@ -127,58 +127,12 @@ template class FastPedestal { */ PEDESTAL_TYPE mean_unchecked(ssize_t index) const { return m_mean[index]; } - /** - * @brief Calculate and return the population variance of every pixel. - * @throws std::runtime_error if ready() is false. - * @note The result is normalized by n_samples. - */ - NDArray variance() { - if (!ready()) { - throw std::runtime_error( - "Pedestal is not ready, cannot return variance"); - } - NDArray res({m_rows, m_cols}); - for (ssize_t i = 0; i < m_sum.size(); ++i) { - res[i] = variance_unchecked(i); - } - return res; - } - - /** - * @brief Calculate the population variance at (row, col). - * @throws std::runtime_error if ready() is false or either index is out of - * range. - */ - PEDESTAL_TYPE variance(const uint32_t row, const uint32_t col) const { - if (!ready()) { - throw std::runtime_error( - "Pedestal is not ready, cannot return variance"); - } - if (row >= m_rows || col >= m_cols) { - throw std::runtime_error(fmt::format( - "Invalid indices for FastPedestal variance: row={}, " - "col={} must be in [0, {}), [0, {})", - row, col, m_rows, m_cols)); - } - return variance_unchecked(rc_to_index(row, col)); - } - - /** - * @brief Calculate the population variance at a flat row-major index. - * @pre ready() is true and index is valid; the index is not checked. - */ - PEDESTAL_TYPE variance_unchecked(ssize_t index) const { - const auto &entry = m_sum[index]; - const auto m = entry.sum * m_inv_samples; - return std::fma(-m, m, entry.sum2 * m_inv_samples); - } - /** * @brief Calculate and return the population standard deviation of every * pixel. * @throws std::runtime_error if ready() is false. */ - NDArray std() { + NDArray std() const { if (!ready()) { throw std::runtime_error( "Pedestal is not ready, cannot return std"); @@ -206,7 +160,7 @@ template class FastPedestal { "col={} must be in [0, {}), [0, {})", row, col, m_rows, m_cols)); } - return std::sqrt(variance(row, col)); + return std_unchecked(rc_to_index(row, col)); } /** @@ -410,6 +364,14 @@ template class FastPedestal { uint32_t n_samples() const { return m_samples; } private: + double variance_unchecked(ssize_t index) const { + const auto &entry = m_sum[index]; + const auto mean = entry.sum * m_inv_samples; + const auto variance = std::fma(-mean, mean, entry.sum2 * m_inv_samples); + // Roundoff in the moments can make a near-zero variance negative. + return std::max(variance, 0.0); + } + /** * @brief Write the cached mean after the final add_init_frame. All other * (non initialization) pushes update the cached mean immediately. diff --git a/include/aare/FilePtr.hpp b/include/aare/FilePtr.hpp index 2cd08490..2e84b87c 100644 --- a/include/aare/FilePtr.hpp +++ b/include/aare/FilePtr.hpp @@ -18,6 +18,7 @@ class FilePtr { FilePtr &operator=(const FilePtr &) = delete; // since we handle a resource FilePtr(FilePtr &&other); FilePtr &operator=(FilePtr &&other); + explicit operator bool() const noexcept; FILE *get(); ssize_t tell(); void seek(ssize_t offset, int whence = SEEK_SET) { @@ -28,4 +29,4 @@ class FilePtr { ~FilePtr(); }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/GainMap.hpp b/include/aare/GainMap.hpp index ff888915..3cc63077 100644 --- a/include/aare/GainMap.hpp +++ b/include/aare/GainMap.hpp @@ -34,29 +34,40 @@ class InvertedGainMap { template >> void apply_gain_map(ClusterVector &clustervec) { - // in principle we need to know the size of the image for this lookup - size_t ClusterSizeX = clustervec.cluster_size_x(); - size_t ClusterSizeY = clustervec.cluster_size_y(); - using T = typename ClusterVector::value_type; - int64_t index_cluster_center_x = ClusterSizeX / 2; - int64_t index_cluster_center_y = ClusterSizeY / 2; + constexpr size_t cluster_size_x = ClusterType::cluster_size_x; + constexpr size_t cluster_size_y = ClusterType::cluster_size_y; + constexpr ssize_t pixel_left_from_center = cluster_size_x / 2; + constexpr ssize_t pixels_right_from_center = + cluster_size_x - pixel_left_from_center - 1; + constexpr ssize_t pixels_top_from_center = cluster_size_y / 2; + constexpr ssize_t pixels_bottom_from_center = + cluster_size_y - pixels_top_from_center - 1; + for (size_t i = 0; i < clustervec.size(); i++) { auto &cl = clustervec[i]; + const auto center_x = static_cast(cl.x); + const auto center_y = static_cast(cl.y); - if (cl.x > 0 && cl.y > 0 && cl.x < m_gain_map.shape(1) - 1 && - cl.y < m_gain_map.shape(0) - 1) { - for (size_t j = 0; j < ClusterSizeX * ClusterSizeY; j++) { - size_t x = cl.x + j % ClusterSizeX - index_cluster_center_x; - size_t y = cl.y + j / ClusterSizeX - index_cluster_center_y; + if (center_x >= pixel_left_from_center && + center_y >= pixels_top_from_center && + center_x < m_gain_map.shape(1) - pixels_right_from_center && + center_y < m_gain_map.shape(0) - pixels_bottom_from_center) { + for (size_t j = 0; j < cluster_size_x * cluster_size_y; j++) { + const auto x = center_x + + static_cast(j % cluster_size_x) - + pixel_left_from_center; + const auto y = center_y + + static_cast(j / cluster_size_x) - + pixels_top_from_center; cl.data[j] = static_cast( static_cast(cl.data[j]) * m_gain_map( y, x)); // cast after conversion to keep precision } } else { - // clear edge clusters + // Clear clusters whose footprint extends beyond the gain map. cl.data.fill(0); } } @@ -66,4 +77,4 @@ class InvertedGainMap { NDArray m_gain_map{}; }; -} // end of namespace aare \ No newline at end of file +} // end of namespace aare diff --git a/include/aare/Pedestal.hpp b/include/aare/Pedestal.hpp index 4e68384f..33ad77e9 100644 --- a/include/aare/Pedestal.hpp +++ b/include/aare/Pedestal.hpp @@ -3,15 +3,19 @@ #include "aare/Frame.hpp" #include "aare/NDArray.hpp" #include "aare/NDView.hpp" +#include #include namespace aare { /** - * @brief Calculate the pedestal of a series of frames. Can be used as - * standalone but mostly used in the ClusterFinder. + * @brief Maintain per-pixel mean and population standard deviation. * - * @tparam SUM_TYPE type of the sum + * Each pixel accumulates its first n_samples values. Subsequent pushes update + * the exponential moving average with a smoothing factor of 1 / n_samples. + * Statistics are available during initialization and are zero for empty pixels. + * Internal moments and the private variance intermediate use double precision. + * @tparam SUM_TYPE Type returned for the mean and standard deviation. */ template class Pedestal { uint32_t m_rows; @@ -20,98 +24,110 @@ template class Pedestal { uint32_t m_samples; NDArray m_cur_samples; - // TODO! in case of int needs to be changed to uint64_t - NDArray m_sum; - NDArray m_sum2; + NDArray m_sum; + NDArray m_sum2; // Cache mean since it is used over and over in the ClusterFinder // This optimization is related to the access pattern of the ClusterFinder // Relies on having more reads than pushes to the pedestal NDArray m_mean; - // Cache std. Only refreshed via update_std() to keep push() cheap. - NDArray m_std; - public: + /** + * @brief Construct an empty pedestal with zero mean and standard deviation. + * @param rows Number of image rows. + * @param cols Number of image columns. + * @param n_samples Number of initialization samples per pixel and + * reciprocal of the weight assigned to each subsequent value. + * @throws std::runtime_error if rows, cols, or n_samples is zero. + */ Pedestal(uint32_t rows, uint32_t cols, uint32_t n_samples = 1000) : m_rows(rows), m_cols(cols), m_samples(n_samples), m_cur_samples(NDArray({rows, cols}, 0)), - m_sum(NDArray({rows, cols})), - m_sum2(NDArray({rows, cols})), - m_mean(NDArray({rows, cols})), - m_std(NDArray({rows, cols})) { - assert(rows > 0 && cols > 0 && n_samples > 0); - m_sum = 0; - m_sum2 = 0; - m_mean = 0; - m_std = 0; + m_sum(NDArray({rows, cols}, 0.0)), + m_sum2(NDArray({rows, cols}, 0.0)), + m_mean(NDArray({rows, cols}, SUM_TYPE(0))) { + if (!(rows > 0 && cols > 0 && n_samples > 0)) { + throw std::runtime_error( + fmt::format("Invalid parameters for Pedestal: rows={}, " + "cols={}, n_samples={} need to be positive", + rows, cols, n_samples)); + } } + ~Pedestal() = default; - NDArray mean() { return m_mean; } - + /** + * @brief Return a non-owning view of the cached mean. + * @note The caller must treat the data as read-only and must not retain the + * view after this object is destroyed, moved, or assigned. + */ NDView view() const { return m_mean.view(); } + /** + * @brief Return the cached mean at (row, col), or zero for an empty pixel. + * @pre row and col are valid pixel indices. + */ SUM_TYPE mean(const uint32_t row, const uint32_t col) const { return m_mean(row, col); } - NDArray cached_std() { return m_std; } - - SUM_TYPE cached_std(const uint32_t row, const uint32_t col) const { - return m_std(row, col); - } + /** @brief Return a copy of the cached mean. */ + NDArray mean() const { return m_mean; } + /** + * @brief Calculate the population standard deviation at (row, col). + * @pre row and col are valid pixel indices. + * @note Variance is normalized by the pixel's current sample count. Empty + * pixels return zero, and negative variance from roundoff is clamped to + * zero. + */ SUM_TYPE std(const uint32_t row, const uint32_t col) const { return std::sqrt(variance(row, col)); } - SUM_TYPE variance(const uint32_t row, const uint32_t col) const { - if (m_cur_samples(row, col) == 0) { - return 0.0; + /** + * @brief Calculate and return the population standard deviation of every + * pixel. Empty pixels return zero. + */ + NDArray std() const { + NDArray res({m_rows, m_cols}); + for (uint32_t row = 0; row < m_rows; ++row) { + for (uint32_t col = 0; col < m_cols; ++col) { + res(row, col) = std(row, col); + } } - return m_sum2(row, col) / m_cur_samples(row, col) - - mean(row, col) * mean(row, col); - } - - NDArray variance() { - NDArray variance_array({m_rows, m_cols}); - for (uint32_t i = 0; i < m_rows * m_cols; i++) { - variance_array(i / m_cols, i % m_cols) = - variance(i / m_cols, i % m_cols); - } - return variance_array; - } - - NDArray std() { - NDArray standard_deviation_array({m_rows, m_cols}); - for (uint32_t i = 0; i < m_rows * m_cols; i++) { - standard_deviation_array(i / m_cols, i % m_cols) = - std(i / m_cols, i % m_cols); - } - - return standard_deviation_array; + return res; } + /** + * @brief Zero the moments, cached mean, and sample counts of every pixel. + */ void clear() { m_sum = 0; m_sum2 = 0; m_cur_samples = 0; m_mean = 0; - m_std = 0; } + /** + * @brief Zero the moments, cached mean, and sample count at (row, col). + * @pre row and col are valid pixel indices. + */ void clear(const uint32_t row, const uint32_t col) { m_sum(row, col) = 0; m_sum2(row, col) = 0; m_cur_samples(row, col) = 0; m_mean(row, col) = 0; - m_std(row, col) = 0; } + /** + * @brief Accumulate or exponentially update every pixel and its cached + * mean. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs. + */ template void push(NDView frame) { - assert(frame.size() == m_rows * m_cols); - // TODO! move away from m_rows, m_cols if (frame.shape() != std::array{m_rows, m_cols}) { throw std::runtime_error( @@ -125,11 +141,18 @@ template class Pedestal { } } + /** + * @brief Push only pixels whose absolute difference from the cached mean is + * strictly less than their threshold. + * @param frame Frame whose shape must exactly match the pedestal. + * @param threshold Per-pixel thresholds with the same shape as the + * pedestal. + * @throws std::runtime_error if either shape differs. + * @note Rejected pixels keep their statistics and sample counts unchanged. + */ template void push_with_threshold(const NDView frame, const NDView threshold) { - assert(frame.size() == m_rows * m_cols); - // TODO! move away from m_rows, m_cols if (frame.shape() != std::array{m_rows, m_cols}) { throw std::runtime_error( @@ -152,45 +175,43 @@ template class Pedestal { } /** - * Push but don't update the cached mean. Speeds up the process - * when initializing the pedestal. - * + * @brief Accumulate or exponentially update every pixel from a Frame. + * @tparam T Actual pixel type stored in frame; this is not runtime-checked. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs. */ - template void push_no_update(NDView frame) { - assert(frame.size() == m_rows * m_cols); + template void push(Frame &frame) { push(frame.view()); } - // TODO! move away from m_rows, m_cols - if (frame.shape() != std::array{m_rows, m_cols}) { - throw std::runtime_error( - "Frame shape does not match pedestal shape"); - } - - for (size_t row = 0; row < m_rows; row++) { - for (size_t col = 0; col < m_cols; col++) { - push_no_update(row, col, frame(row, col)); - } - } - } - - template void push(Frame &frame) { - assert(frame.rows() == static_cast(m_rows) && - frame.cols() == static_cast(m_cols)); - push(frame.view()); - } - - // getter functions + /** @brief Return the number of image rows. */ uint32_t rows() const { return m_rows; } - uint32_t cols() const { return m_cols; } - uint32_t n_samples() const { return m_samples; } - NDArray cur_samples() const { return m_cur_samples; } - NDArray get_sum() const { return m_sum; } - NDArray get_sum2() const { return m_sum2; } - // pixel level operations (should be refactored to allow users to implement - // their own pixel level operations) + /** @brief Return the number of image columns. */ + uint32_t cols() const { return m_cols; } + + /** + * @brief Return the initialization sample count per pixel and steady-state + * update-weight denominator. + */ + uint32_t n_samples() const { return m_samples; } + + /** + * @brief Return a copy of the per-pixel initialization sample counts. + * @note Each count is in [0, n_samples] and does not change during + * steady-state pushes. Thresholded pushes can leave counts unequal. + */ + NDArray cur_samples() const { return m_cur_samples; } + + /** + * @brief Accumulate or exponentially update one pixel and its cached mean. + * @param row Pixel row. + * @param col Pixel column. + * @param val_ New pixel value, with weight 1 / n_samples after + * initialization. + * @pre row and col are valid pixel indices. + */ template void push(const uint32_t row, const uint32_t col, const T val_) { - SUM_TYPE val = static_cast(val_); + const auto val = static_cast(val_); if (m_cur_samples(row, col) < m_samples) { m_sum(row, col) += val; m_sum2(row, col) += val * val; @@ -204,45 +225,16 @@ template class Pedestal { m_mean(row, col) = m_sum(row, col) / m_cur_samples(row, col); } - template - void push_no_update(const uint32_t row, const uint32_t col, const T val_) { - SUM_TYPE val = static_cast(val_); - if (m_cur_samples(row, col) < m_samples) { - m_sum(row, col) += val; - m_sum2(row, col) += val * val; - m_cur_samples(row, col)++; - } else { - m_sum(row, col) += val - m_sum(row, col) / m_cur_samples(row, col); - m_sum2(row, col) += - val * val - m_sum2(row, col) / m_cur_samples(row, col); + private: + double variance(const uint32_t row, const uint32_t col) const { + if (m_cur_samples(row, col) == 0) { + return 0.0; } - } - - /** - * @brief Update the mean of the pedestal. This is used after having done - * push_no_update. It is not necessary to call this function after push. - */ - void update_mean() { m_mean = m_sum / m_cur_samples; } - - /** - * @brief Refresh the cached std for all pixels from the current sums. - * Kept separate from push() so pushes stay cheap; call before reading - * cached_std() (analogous to update_mean()). - */ - void update_std() { - for (uint32_t i = 0; i < m_rows * m_cols; i++) { - m_std(i / m_cols, i % m_cols) = std(i / m_cols, i % m_cols); - } - } - - template - void push_fast(const uint32_t row, const uint32_t col, const T val_) { - // Assume we reached the steady state where all pixels have - // m_samples samples - SUM_TYPE val = static_cast(val_); - m_sum(row, col) += val - m_sum(row, col) / m_samples; - m_sum2(row, col) += val * val - m_sum2(row, col) / m_samples; - m_mean(row, col) = m_sum(row, col) / m_samples; + const auto mean = m_sum(row, col) / m_cur_samples(row, col); + const auto var = + m_sum2(row, col) / m_cur_samples(row, col) - mean * mean; + // Roundoff in the moments can make a near-zero variance negative. + return std::max(var, 0.0); } }; } // namespace aare diff --git a/include/aare/ROI.hpp b/include/aare/ROI.hpp new file mode 100644 index 00000000..81c2a84e --- /dev/null +++ b/include/aare/ROI.hpp @@ -0,0 +1,132 @@ +#pragma once +#include "aare/defs.hpp" +#include +#include +#include +#include +#include +#include + +namespace aare { + +class ROIGeometry; // forward declaration to avoid circular dependency +class DetectorGeometry; // forward declaration to avoid circular dependency + +struct ROI { + ssize_t xmin{}; + ssize_t xmax{}; + ssize_t ymin{}; + ssize_t ymax{}; + + ssize_t height() const { return ymax - ymin; } + ssize_t width() const { return xmax - xmin; } + bool contains(ssize_t x, ssize_t y) const { + return x >= xmin && x < xmax && y >= ymin && y < ymax; + } + + bool operator==(const ROI &other) const { + return xmin == other.xmin && xmax == other.xmax && ymin == other.ymin && + ymax == other.ymax; + } +}; + +/** + * @brief Merge all consecutive ROIs in a vector into a single ROI. + * @param rois vector of ROIs to merge + * @return vector of merged ROIs + * @tparam horizontally_aligned true if the ROIs are horizontally aligned, false + * otherwise + * @tparam vertically_aligned true if the ROIs are vertically aligned, false + * otherwise + */ +template +std::vector merge_consecutive_rois(std::vector &rois) { + + if constexpr (horizontally_aligned && vertically_aligned) { + throw std::runtime_error( + LOCATION + "Vector of the same ROI? Cannot merge ROIs both " + "horizontally and vertically at the same time."); + } + + if (rois.empty()) { + return {}; + } + if (rois.size() == 1) { + return rois; + } + + auto merge_along_x = [](std::vector in_rois) { + std::sort(in_rois.begin(), in_rois.end(), + [](const ROI &a, const ROI &b) { + return (a.ymin != b.ymin) ? (a.ymin < b.ymin) + : (a.xmin < b.xmin); + }); // N log (N) + + std::vector merged_rois; + merged_rois.reserve(in_rois.size()); + merged_rois.push_back(in_rois[0]); + + for (size_t i = 1; i < in_rois.size(); ++i) { + auto &last = merged_rois.back(); + const auto ¤t = in_rois[i]; + if (last.ymin == current.ymin && last.ymax == current.ymax && + last.xmax == current.xmin) { + // merge + last.xmax = current.xmax; + } else { + merged_rois.push_back(current); + } + } + return merged_rois; + }; + + auto merge_along_y = [](std::vector in_rois) { + std::sort(in_rois.begin(), in_rois.end(), + [](const ROI &a, const ROI &b) { + return (a.xmin != b.xmin) ? (a.xmin < b.xmin) + : (a.ymin < b.ymin); + }); + + std::vector merged_rois; + merged_rois.reserve(in_rois.size()); + merged_rois.push_back(in_rois[0]); + + for (size_t i = 1; i < in_rois.size(); ++i) { + auto &last = merged_rois.back(); + const auto ¤t = in_rois[i]; + if (last.xmin == current.xmin && last.xmax == current.xmax && + last.ymax == current.ymin) { + last.ymax = current.ymax; + } else { + merged_rois.push_back(current); + } + } + return merged_rois; + }; + + if constexpr (horizontally_aligned) { + return merge_along_y(rois); // one sort + one pass + } else if constexpr (vertically_aligned) { + return merge_along_x(rois); // one sort + one pass + } else { + return merge_along_y(merge_along_x(rois)); // generic case: two passes + } +} + +/** + * @brief Check if the ROI covers the entire detector geometry + * @param roi Region of interest + * @param geometry Detector geometry + * @return true if the ROI covers the entire detector geometry, false otherwise + */ +bool complete_ROI(const ROI &roi, const DetectorGeometry &geometry); + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry); + +bool complete_ROI(const ROIGeometry &roi, const DetectorGeometry &geometry); + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry); + +} // namespace aare \ No newline at end of file diff --git a/include/aare/ROIGeometry.hpp b/include/aare/ROIGeometry.hpp index 05450df5..739b24c3 100644 --- a/include/aare/ROIGeometry.hpp +++ b/include/aare/ROIGeometry.hpp @@ -1,6 +1,8 @@ #pragma once #include "aare/DetectorGeometry.hpp" -#include "aare/defs.hpp" +#include "aare/ROI.hpp" +#include +#include namespace aare { @@ -17,11 +19,6 @@ class ROIGeometry { */ ROIGeometry(const ROI &roi, DetectorGeometry &geometry); - /** @brief Constructor for ROI geometry expanding over full detector - * @param geometry general detector geometry - */ - ROIGeometry(DetectorGeometry &geometry); - /// @brief Get number of modules in the ROI size_t num_modules_in_roi() const; diff --git a/include/aare/RawFile.hpp b/include/aare/RawFile.hpp index 999b737f..4ef92962 100644 --- a/include/aare/RawFile.hpp +++ b/include/aare/RawFile.hpp @@ -1,6 +1,5 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once -#include "aare/DetectorGeometry.hpp" #include "aare/FileInterface.hpp" #include "aare/Frame.hpp" #include "aare/NDArray.hpp" //for pixel map @@ -30,17 +29,18 @@ class RawFile : public FileInterface { RawMasterFile m_master; size_t m_current_frame{}; - DetectorGeometry m_geometry; - - /// @brief Geometries e.g. number of modules, size etc. for each ROI std::vector m_ROI_geometries; + /// @brief Minimum frame count across the selected raw subfile series. + size_t m_frames_in_file{}; + public: /** * @brief RawFile constructor * @param fname path to the master file (.json) * @param mode file mode (only "r" is supported at the moment) - + * @throws std::runtime_error if frame padding is disabled and the frame + * discard policy is not DiscardPartial. */ RawFile(const std::filesystem::path &fname, const std::string &mode = "r"); virtual ~RawFile() override = default; @@ -104,6 +104,7 @@ class RawFile : public FileInterface { size_t bytes_per_pixel() const; void seek(size_t frame_index) override; size_t tell() override; + /// @brief Minimum actual frame count across all subfiles and ROIs. size_t total_frames() const override; size_t rows() const override; /** @@ -122,7 +123,7 @@ class RawFile : public FileInterface { size_t n_modules() const; /** - * @brief number of ROIs defined + * @brief number of ROIs defined (always 1 for complete ROI) */ size_t num_rois() const; @@ -150,6 +151,9 @@ class RawFile : public FileInterface { static DetectorHeader read_header(const std::filesystem::path &fname); private: + std::runtime_error frame_error(size_t frame_index, + const std::string &message) const; + /** * @brief read the frame at the given frame index into the image buffer * @param frame_index frame number to read diff --git a/include/aare/RawMasterFile.hpp b/include/aare/RawMasterFile.hpp index d7b13a42..26ea3b41 100644 --- a/include/aare/RawMasterFile.hpp +++ b/include/aare/RawMasterFile.hpp @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once -#include "aare/defs.hpp" +#include "aare/DetectorGeometry.hpp" +#include "aare/ROI.hpp" #include #include #include @@ -88,7 +89,8 @@ class RawMasterFile { std::optional m_exptime; std::chrono::nanoseconds m_period{0}; - xy m_geometry{}; + /// @brief modules in x and y direction + xy m_detector_layout{}; xy m_udp_interfaces_per_module{1, 1}; size_t m_max_frames_per_file{}; @@ -109,12 +111,28 @@ class RawMasterFile { std::optional m_number_of_rows; std::optional m_counter_mask; - std::optional> m_rois; + /// @brief index of disabled UDP ports - index relative to UDP_port_types + std::vector m_disabled_udp_ports{}; + + /// @brief udp port types + std::optional> m_udp_port_types{}; + + /// @brief ROIs defined in master file or derived from disabled UDP ports + std::vector m_rois; + + /// @brief Detector geometry - geometry for each module + DetectorGeometry m_geometry{}; + + /// @brief ROI geometries + std::vector m_ROI_geometries; public: RawMasterFile(const std::filesystem::path &fpath); RawMasterFile(std::istream &is, const std::string &fname); // for testing + /// @brief Get the filename including path of the master file. + std::filesystem::path master_fname() const { return m_fnc.master_fname(); } + std::filesystem::path data_fname(size_t mod_id, size_t file_id) const; const std::string &version() const; //!< For example "7.2" @@ -131,10 +149,14 @@ class RawMasterFile { const FrameDiscardPolicy &frame_discard_policy() const; size_t total_frames_expected() const; - xy geometry() const; + xy detector_layout() const; size_t n_modules() const; uint8_t quad() const; + const DetectorGeometry &geometry() const; + + const std::vector &roi_geometries() const; + ReadoutMode get_reading_mode() const; std::optional analog_samples() const; @@ -143,9 +165,21 @@ class RawMasterFile { std::optional number_of_rows() const; std::optional counter_mask() const; - std::optional> rois() const; + /// @brief Get the types of UDP ports + /// @return Optional vector of UDP port types as strings (only present for + /// masterfile version >= 8.1) + std::optional> udp_port_types() const; - std::optional roi() const; + /// @brief Get the indices of disabled UDP ports + /// @return vector of indices of disabled UDP ports (empty if none are + /// disabled) + std::vector disabled_udp_ports() const; + + std::vector rois() const; + + /// @brief get roi for the case of a single ROI + /// @return ROI object (complete ROI if no roi present in master file) + ROI roi() const; ScanParameters scan_parameters() const; @@ -157,7 +191,8 @@ class RawMasterFile { private: void parse_json(std::istream &is); void parse_raw(std::istream &is); + void update_rois_from_disabled_udp_ports(); void retrieve_geometry(); }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/RawSubFile.hpp b/include/aare/RawSubFile.hpp index 50db61fa..deff6fb7 100644 --- a/include/aare/RawSubFile.hpp +++ b/include/aare/RawSubFile.hpp @@ -84,7 +84,13 @@ class RawSubFile { size_t frames_in_file() const { return m_total_frames; } + /// @brief Path of the current data file in the raw subfile series. + std::filesystem::path current_path() const; + private: + std::runtime_error frame_error(size_t frame_index, + const std::string &message) const; + template void read_with_map(std::byte *image_buf); void parse_fname(const std::filesystem::path &fname); @@ -93,4 +99,4 @@ class RawSubFile { std::filesystem::path fpath(size_t file_index) const; }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/defs.hpp b/include/aare/defs.hpp index f810f411..b0701912 100644 --- a/include/aare/defs.hpp +++ b/include/aare/defs.hpp @@ -94,19 +94,6 @@ template struct t_xy { }; using xy = t_xy; -struct ROI { - ssize_t xmin{}; - ssize_t xmax{}; - ssize_t ymin{}; - ssize_t ymax{}; - - ssize_t height() const { return ymax - ymin; } - ssize_t width() const { return xmax - xmin; } - bool contains(ssize_t x, ssize_t y) const { - return x >= xmin && x < xmax && y >= ymin && y < ymax; - } -}; - /// @brief Chip specifications for Matterhorn1 struct Matterhorn10 { constexpr static size_t nRows = 256; @@ -315,7 +302,21 @@ enum class corner : int { cBottomRight = 3 }; -enum class TimingMode { Auto, Trigger }; +enum class UDPPortPosition : uint8_t { + LEFT = 0, + RIGHT = 1, + TOP = 2, + BOTTOM = 3 +}; + +enum class TimingMode : uint8_t { + AUTO_TIMING = 0, + TRIGGER_EXPOSURE = 1, + GATED = 2, + BURST_TRIGGER = 3, + TRIGGER_GATED = 4 +}; + enum class FrameDiscardPolicy { NoDiscard, Discard, DiscardPartial }; using DataTypeVariants = std::variant; diff --git a/src/to_string.hpp b/include/aare/to_string.hpp similarity index 83% rename from src/to_string.hpp rename to include/aare/to_string.hpp index 0bb10ada..dbb0c09f 100644 --- a/src/to_string.hpp +++ b/include/aare/to_string.hpp @@ -55,6 +55,14 @@ template T string_to(const std::string &arg) { */ template <> DetectorType string_to(const std::string &arg); +/** + * @brief Convert a string to UDPPortPosition + * @param name string representation of the UDPPortPosition + * @return UDPPortPosition + * @throw runtime_error if the string does not match any UDPPortPosition + */ +template <> UDPPortPosition string_to(const std::string &arg); + /** * @brief Convert a string to TimingMode * @param mode string representation of the TimingMode @@ -79,4 +87,12 @@ template <> FrameDiscardPolicy string_to(const std::string &arg); */ template <> DACIndex string_to(const std::string &arg); +/** + * @brief Convert a corner enum to string + * @param arg corner enum value + * @return string representation of the corner enum + * @throw runtime_error if the corner value is invalid + */ +std::string to_string(const corner &arg); + } // namespace aare \ No newline at end of file diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt index d79a3e8e..c1bc0bab 100644 --- a/python/CMakeLists.txt +++ b/python/CMakeLists.txt @@ -69,6 +69,7 @@ set(PYTHON_FILES aare/ClusterVector.py aare/Cluster.py aare/FastPedestal.py + aare/Pedestal.py aare/calibration.py aare/factory.py aare/experimental.py diff --git a/python/aare/ClusterFinder.py b/python/aare/ClusterFinder.py index ca770324..f3eab2e3 100644 --- a/python/aare/ClusterFinder.py +++ b/python/aare/ClusterFinder.py @@ -18,17 +18,33 @@ def _get_class(name, cluster_size, dtype): -def ClusterFinder(image_size, cluster_size=(3,3), n_sigma=5, dtype = np.int32, capacity = 1024): +def ClusterFinder(image_size, *, cluster_size=(3,3), n_sigma=5, dtype = np.int32, capacity = 1024, min_pedestal_samples = 1000): """ Factory function to create a ClusterFinder object. Provides a cleaner syntax for the templated ClusterFinder in C++. + + Parameters + ---------- + image_size : tuple + The size of the image as a tuple (height, width). + cluster_size : tuple, optional + The size of the cluster to find as a tuple (height, width). Default is (3,3). + n_sigma : int, optional + Multiplier of the standard deviation used as a threshold to identify potential photon pixels. Default is 5. + dtype : data-type, optional + The data type of the image. Default is np.int32. + capacity : int, optional + The maximum number of clusters than can be stored before reallocating. Default is 1024. + min_pedestal_samples : int, optional + The minimum number of pedestal samples to accumulate before using the pedestal. Default is 1000. """ cls = _get_class("ClusterFinder", cluster_size, dtype) - return cls(image_size, n_sigma=n_sigma, capacity=capacity) + return cls(image_size, n_sigma=n_sigma, capacity=capacity, min_pedestal_samples=min_pedestal_samples) -def ClusterFinderFrozen(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, capacity=1024): +def ClusterFinderFrozen(image_size, *, cluster_size=(3,3), n_sigma=5, dtype=np.int32, + capacity=1024, min_pedestal_samples=1000): """ Factory function to create a ClusterFinderFrozen object. @@ -37,19 +53,40 @@ def ClusterFinderFrozen(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int3 pedestal updates are deferred to the end of the frame). This mirrors the CUDA kernel's per-frame update model, so running it against ClusterFinderCUDA isolates pedestal-update timing as the sole variable. + + min_pedestal_samples matches ClusterFinder so the twins warm up identically; + a comparison run must pass the same value to both. """ cls = _get_class("ClusterFinderFrozen", cluster_size, dtype) - return cls(image_size, n_sigma=n_sigma, capacity=capacity) + return cls(image_size, n_sigma=n_sigma, capacity=capacity, + min_pedestal_samples=min_pedestal_samples) -def ClusterFinderMT(image_size, cluster_size = (3,3), dtype=np.int32, n_sigma=5, capacity = 1024, n_threads = 3): +def ClusterFinderMT(image_size, *, cluster_size = (3,3), dtype=np.int32, n_sigma=5, capacity = 1024, n_threads = 3, min_pedestal_samples = 1000): """ Factory function to create a ClusterFinderMT object. Provides a cleaner syntax for the templated ClusterFinderMT in C++. + + Parameters + ---------- + image_size : tuple + The size of the image as a tuple (height, width). + cluster_size : tuple, optional + The size of the cluster to find as a tuple (height, width). Default is (3,3). + n_sigma : int, optional + Multiplier of the standard deviation used as a threshold to identify potential photon pixels. Default is 5. + dtype : data-type, optional + The data type of the image. Default is np.int32. + capacity : int, optional + The maximum number of clusters than can be stored before reallocating. Default is 1024. + n_threads : int, optional + The number of threads to use for processing. Default is 3. + min_pedestal_samples : int, optional + The minimum number of pedestal samples to accumulate before using the pedestal. Default is 1000. """ cls = _get_class("ClusterFinderMT", cluster_size, dtype) - return cls(image_size, n_sigma=n_sigma, capacity=capacity, n_threads=n_threads) + return cls(image_size, n_sigma=n_sigma, capacity=capacity, min_pedestal_samples=min_pedestal_samples, n_threads=n_threads) def _cuda_available(): @@ -63,9 +100,10 @@ def _cuda_available(): _CUDA_ALGORITHMS = {"fixed_window": ""} -def ClusterFinderCUDA(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, +def ClusterFinderCUDA(image_size, *, cluster_size=(3,3), n_sigma=5, dtype=np.int32, max_clusters_per_frame=2048, n_streams=4, - time_kernels=False, algorithm="fixed_window"): + min_pedestal_samples=1000, algorithm="fixed_window", + time_kernels=False): """ Factory function to create a ClusterFinderCUDA object. Provides a cleaner syntax for the templated ClusterFinderCUDA in C++. API mirrors @@ -87,6 +125,14 @@ def ClusterFinderCUDA(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, but as tight as possible to minimize PCIe traffic. Default 2048. n_streams : int, optional Number of CUDA streams for H2D/kernel/D2H pipelining. Default 4. + min_pedestal_samples : int, optional + Frames the host pedestal accumulates before it is ready, and the + reciprocal of the steady-state update weight. Matches ClusterFinder so + a CPU/GPU comparison warms up identically. Default 1000. + algorithm : str, optional + GPU algorithm. "fixed_window" (default): a cluster_size window around + each pixel with local-maximum suppression. Memory and streams are + managed internally whichever algorithm is chosen. time_kernels : bool, optional Enable per-frame CUDA-event kernel timing, exposed via avg_kernel_time_ms(). Off by default because it adds two event records @@ -95,10 +141,6 @@ def ClusterFinderCUDA(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, contention the events measure queue wait, not execution, and over-read by up to ~3.5x. Use Nsight Systems for exclusive kernel times. When disabled, avg_kernel_time_ms() returns NaN. - algorithm : str, optional - GPU algorithm. "fixed_window" (default): a cluster_size window around - each pixel with local-maximum suppression. Memory and streams are - managed internally whichever algorithm is chosen. Example ------- @@ -137,6 +179,7 @@ def ClusterFinderCUDA(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, n_sigma=n_sigma, max_clusters_per_frame=max_clusters_per_frame, n_streams=n_streams, + min_pedestal_samples=min_pedestal_samples, time_kernels=time_kernels) def find_cluster_views_batched_iter(cf, frames, first_frame=0, chunk=None): @@ -202,8 +245,9 @@ def find_cluster_views_batched_iter(cf, frames, first_frame=0, chunk=None): view.release() -def ClusterFinderCUDAGraph(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.int32, - max_clusters_per_frame=2048, n_streams=4): +def ClusterFinderCUDAGraph(image_size, *, cluster_size=(3,3), n_sigma=5, dtype=np.int32, + max_clusters_per_frame=2048, n_streams=4, + min_pedestal_samples=1000): """ Factory function to create a ClusterFinderCUDAGraph object. Uses pre-recorded CUDA Graphs to reduce per-frame CPU API overhead (~23 µs vs ~31 µs for the @@ -224,6 +268,8 @@ def ClusterFinderCUDAGraph(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.i Hard upper bound on clusters per frame. Default 2048. n_streams : int, optional Number of CUDA streams (one graph per stream). Default 4. + min_pedestal_samples : int, optional + Frames the host pedestal accumulates before it is ready. Default 1000. Note ---- @@ -240,7 +286,8 @@ def ClusterFinderCUDAGraph(image_size, cluster_size=(3,3), n_sigma=5, dtype=np.i return cls(image_size, n_sigma=n_sigma, max_clusters_per_frame=max_clusters_per_frame, - n_streams=n_streams) + n_streams=n_streams, + min_pedestal_samples=min_pedestal_samples) def ClusterCollector(clusterfindermt, dtype=np.int32): @@ -263,19 +310,55 @@ def ClusterFileSink(clusterfindermt, cluster_file, dtype=np.int32): def ClusterFile(fname, cluster_size=(3,3), dtype=np.int32, chunk_size = 1000, mode = "r"): - """ - Factory function to create a ClusterFile object. Provides a cleaner syntax for - the templated ClusterFile in C++. + """Create a reader or writer for a binary cluster file. + + Parameters + ---------- + fname : path-like + Cluster file to open. + cluster_size : tuple[int, int], default=(3, 3) + Cluster dimensions stored in the file. + dtype : numpy dtype, default=numpy.int32 + Data type of the cluster values stored in the file. + chunk_size : int, default=1000 + Maximum number of selected clusters returned by ``chunks()`` and + default iteration. Must be positive when iterating over chunks. + mode : {"r", "w", "a"}, default="r" + Open for reading, truncate and write, or append, respectively. + + Returns + ------- + ClusterFile + The compiled ClusterFile specialization matching ``cluster_size`` and + ``dtype``. + + Notes + ----- + The file format contains no cluster shape or data-type metadata. Supplying + values that do not match the file causes its bytes to be interpreted + incorrectly. Use ``frames()`` to iterate over complete frames, including + empty or fully filtered frames with their stored frame numbers. Use + ``chunks()`` or ``chunks(chunk_size)`` to iterate over selected clusters + in batches. Chunks may combine frames, so their frame number is not + reliable per-cluster metadata. + + Iterators consume the current file position without rewinding; use one + traversal at a time. Each result owns its storage and remains valid after + advancing the iterator or closing the file. + + Examples + -------- .. code-block:: python from aare import ClusterFile - - with ClusterFile("clusters.clust", cluster_size=(3,3), dtype=np.int32) as cf: - # cf is now a ClusterFile_Cluster3x3i object but you don't need to know that. - for clusters in cf: - # Loop over clusters in chunks of 1000 - # The type of clusters will be a ClusterVector_Cluster3x3i in this case + + with ClusterFile( + "clusters.clust", cluster_size=(3, 3), dtype=np.int32 + ) as cf: + for clusters in cf.chunks(): + # Process clusters in chunks of at most 1000. + ... """ diff --git a/python/aare/ClusterVector.py b/python/aare/ClusterVector.py index a294bb55..6c4a65f9 100644 --- a/python/aare/ClusterVector.py +++ b/python/aare/ClusterVector.py @@ -7,16 +7,33 @@ from .ClusterFinder import _get_class def ClusterVector(cluster_size=(3,3), dtype = np.int32): """ - Factory function to create a ClusterVector object. Provides a cleaner syntax for - the templated ClusterVector in C++. + Create an empty ClusterVector for a supported cluster size and pixel dtype. - .. code-block:: python + Parameters + ---------- + cluster_size : tuple[int, int], default=(3, 3) + Cluster dimensions in the x and y directions. + dtype : numpy.dtype, default=numpy.int32 + Pixel storage type. The cluster size and dtype combination must have a + compiled binding. - from aare import ClusterVector - - ClusterVector(cluster_size=(3,3), dtype=np.float64) + Returns + ------- + ClusterVector + An empty vector with frame number 0 and space reserved for at least + 1024 clusters. + + Raises + ------ + ValueError + If the requested size and dtype combination is unavailable. + + Examples + -------- + >>> import numpy as np + >>> from aare import ClusterVector + >>> clusters = ClusterVector(cluster_size=(3, 3), dtype=np.float64) """ cls = _get_class("ClusterVector", cluster_size, dtype) return cls() - diff --git a/python/aare/FastPedestal.py b/python/aare/FastPedestal.py index 38bbeb5b..3e28a205 100644 --- a/python/aare/FastPedestal.py +++ b/python/aare/FastPedestal.py @@ -22,7 +22,7 @@ def FastPedestal(rows, cols, n_samples=1000, dtype=np.float64): cols: Number of image columns. n_samples: Initialization frame count and steady-state update-weight denominator. - dtype: Output dtype for the mean, variance, and standard deviation. + dtype: Output dtype for the mean and standard deviation. Supported values are ``np.float64``, ``np.float32``, and ``np.int16``. """ @@ -41,7 +41,7 @@ def from_file(filename, n_samples=1000, skip_first=0, dtype=np.float64): filename: Input image file. n_samples: Number of frames used for initialization. skip_first: Number of leading frames to ignore. - dtype: Output dtype for the mean, variance, and standard deviation. + dtype: Output dtype for the mean and standard deviation. Raises: RuntimeError: If fewer than ``n_samples`` frames remain after diff --git a/python/aare/Pedestal.py b/python/aare/Pedestal.py new file mode 100644 index 00000000..d89d6012 --- /dev/null +++ b/python/aare/Pedestal.py @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: MPL-2.0 + +import numpy as np + +from .factory import _get_typed_class + + +def Pedestal(rows, cols, n_samples=1000, dtype=np.float64): + """Create an empty per-pixel running pedestal. + + This factory hides the dtype suffix used by the templated C++ bindings. + Call ``push()`` to update the statistics and cached mean for each frame. + Statistics are available during initialization and are zero for empty pixels. + Internal moments and variance always use double precision. + + Args: + rows: Number of image rows. + cols: Number of image columns. + n_samples: Number of samples accumulated before switching to + steady-state updates with weight ``1 / n_samples``. + dtype: Output dtype for the mean and standard deviation. + Supported values are ``np.float64``, ``np.float32``, and + ``np.int16``. + """ + cls = _get_typed_class("Pedestal", dtype) + return cls(rows, cols, n_samples) diff --git a/python/aare/RawFile.py b/python/aare/RawFile.py index 988ff3d5..253fcc53 100644 --- a/python/aare/RawFile.py +++ b/python/aare/RawFile.py @@ -43,7 +43,7 @@ class RawFile(_aare.RawFile): Returns: int: Number of frames in file. """ - return super().frames_in_file + return self.total_frames def __enter__(self): return self diff --git a/python/aare/__init__.py b/python/aare/__init__.py index 77495c54..d6aeeb1e 100644 --- a/python/aare/__init__.py +++ b/python/aare/__init__.py @@ -39,15 +39,18 @@ from ._aare import ( RawSubFile, ) from ._aare import Pedestal_d, Pedestal_f, ClusterFinder_Cluster3x3i, VarClusterFinder -from ._aare import DetectorType, ReadoutMode +from ._aare import DetectorType, FrameDiscardPolicy, ReadoutMode from ._aare import hitmap from ._aare import ROI from ._aare import corner +from ._aare import UDPPortPosition + # from ._aare import ClusterFinderMT, ClusterCollector, ClusterFileSink, ClusterVector_i from ._version import __version__ from .FastPedestal import FastPedestal +from .Pedestal import Pedestal from .ClusterFinder import (ClusterFinder, ClusterFinderFrozen, ClusterCollector, ClusterFinderMT, ClusterFileSink, ClusterFile, ClusterFinderCUDA, ClusterFinderCUDAGraph, _cuda_available, find_cluster_views_batched_iter) from .ClusterVector import ClusterVector diff --git a/python/aare/calibration.py b/python/aare/calibration.py index b7a23e74..20f8c77b 100644 --- a/python/aare/calibration.py +++ b/python/aare/calibration.py @@ -5,9 +5,12 @@ def load_calibration(fname, hg0=False): """ Load calibration data from a file. - Parameters: - fname (str): Path to the calibration file. - hg0 (bool): If True, load HG0 calibration data instead of G0. + Parameters + ---------- + fname : str + Path to the calibration file. + hg0 : bool + If True, load HG0 calibration data instead of G0. """ gains = 3 diff --git a/python/src/bind_Cluster.hpp b/python/src/bind_Cluster.hpp index 0c64eb6d..c168929e 100644 --- a/python/src/bind_Cluster.hpp +++ b/python/src/bind_Cluster.hpp @@ -37,7 +37,6 @@ void define_Cluster(py::module &m, const std::string &typestr) { return cluster; })) - // TODO! Review if to keep or not .def_property_readonly( "data", [](Cluster &c) @@ -60,6 +59,26 @@ void define_Cluster(py::module &m, const std::string &typestr) { .def_readonly("y", &Cluster::y) + .def("__repr__", + [class_name](const Cluster &self) { + fmt::memory_buffer data; + fmt::format_to(std::back_inserter(data), "[["); + for (size_t i = 0; i < self.data.size(); ++i) { + if (i != 0) { + fmt::format_to(std::back_inserter(data), + i % ClusterSizeX == 0 ? "], [" : ", "); + } + fmt::format_to(std::back_inserter(data), "{}", + self.data[i]); + } + fmt::format_to(std::back_inserter(data), "]]"); + + return fmt::format("{}(x={}, y={}, data={})", class_name, + self.x, self.y, + fmt::string_view(data.data(), data.size())); + }) + .def( "max_sum_2x2", [](Cluster &self) { @@ -67,8 +86,16 @@ void define_Cluster(py::module &m, const std::string &typestr) { return py::make_tuple(max_sum.sum, static_cast(max_sum.index)); }, - R"(calculates sum of 2x2 subcluster with highest energy and index relative to cluster center 0: top_left, 1: top_right, 2: bottom_left, 3: bottom_right - )"); + R"doc( + Return the highest-sum center-adjacent 2x2 subcluster. + + Returns + ------- + tuple + ``(sum, index)``, where index is 0 for top-left, 1 for + top-right, 2 for bottom-left, or 3 for bottom-right relative to + the cluster center. + )doc"); } template &cl) { return reduce_to_3x3(cl); }, - py::return_value_policy::move, R"(Reduce cluster to 3x3 subcluster)"); + py::return_value_policy::move, py::arg("cluster"), R"doc( + Return the 3x3 block around the cluster's center index. + + Both input dimensions must be at least 3, and at least one must be + greater than 3. + The input coordinates are preserved and output data is stored in + row-major order. + )doc"); } template &cl) { return reduce_to_2x2(cl); }, - py::return_value_policy::move, - R"( - Reduce cluster to 2x2 subcluster by taking the 2x2 subcluster with - the highest photon energy. + py::return_value_policy::move, py::arg("cluster"), R"doc( + Return the highest-sum center-adjacent 2x2 block. - RETURN: - - reduced cluster (cluster is filled in row major ordering starting at the top left. Thus for a max subcluster in the top left corner the photon hit is at the fourth position.) - - )"); + The input coordinates are preserved and output data is stored in + row-major order. + )doc"); } -#pragma GCC diagnostic pop \ No newline at end of file +#pragma GCC diagnostic pop diff --git a/python/src/bind_ClusterFile.hpp b/python/src/bind_ClusterFile.hpp index 0831d64f..09fc649b 100644 --- a/python/src/bind_ClusterFile.hpp +++ b/python/src/bind_ClusterFile.hpp @@ -11,6 +11,7 @@ #include #include #include +#include // Disable warnings for unused parameters, as we ignore some // in the __exit__ method @@ -20,18 +21,90 @@ namespace py = pybind11; using namespace ::aare; +template class ClusterFileIterator { + Range m_range; + std::optional m_iterator; + bool m_done{false}; + + public: + explicit ClusterFileIterator(Range range) : m_range(std::move(range)) {} + + typename Range::Iterator::value_type next() { + if (m_done) { + throw py::stop_iteration(); + } + // Advance before returning the next result, never after yielding it. + if (m_iterator) { + ++*m_iterator; + } else { + m_iterator.emplace(m_range.begin()); + } + if (*m_iterator == m_range.end()) { + m_done = true; + throw py::stop_iteration(); + } + return std::move(**m_iterator); + } +}; + +template +void define_cluster_file_iterator(py::module &m, const std::string &name) { + py::class_(m, name.c_str()) + .def( + "__iter__", [](Iterator &self) -> Iterator & { return self; }, + py::return_value_policy::reference_internal) + .def("__next__", &Iterator::next); +} + template void define_ClusterFile(py::module &m, const std::string &typestr) { using ClusterType = Cluster; + using File = ClusterFile; + using FrameIterator = ClusterFileIterator; + using ChunkIterator = ClusterFileIterator; auto class_name = fmt::format("ClusterFile_{}", typestr); + define_cluster_file_iterator(m, + class_name + "_FrameIterator"); + define_cluster_file_iterator(m, + class_name + "_ChunkIterator"); - py::class_>(m, class_name.c_str()) + py::class_>( + m, class_name.c_str(), + "Read and write legacy binary cluster files. The format contains no " + "cluster type or shape metadata, so this class must match the file.") .def(py::init(), - py::arg(), py::arg("chunk_size") = 1000, py::arg("mode") = "r") + py::arg("fname"), py::arg("chunk_size") = 1000, + py::arg("mode") = "r", + "Open a cluster file. Mode must be 'r' to read, 'w' to truncate " + "and write, or 'a' to append.") + .def( + "frames", [](File &self) { return FrameIterator(self.frames()); }, + py::keep_alive<0, 1>(), + "Iterate over complete frames from the current position, including " + "empty and fully filtered frames with their stored frame numbers. " + "Results own their storage. The iterator keeps the file alive but " + "shares its cursor; use only one traversal at a time.") + .def( + "chunks", [](File &self) { return ChunkIterator(self.chunks()); }, + py::keep_alive<0, 1>(), + "Iterate from the current position using the constructor's chunk " + "size, which must be positive. Results own their storage. The " + "iterator keeps the file alive but shares its cursor; use only " + "one traversal at a time.") + .def( + "chunks", + [](File &self, size_t chunk_size) { + return ChunkIterator(self.chunks(chunk_size)); + }, + py::arg("chunk_size"), py::keep_alive<0, 1>(), + "Iterate over chunks of up to chunk_size selected clusters. The " + "size must be positive and does not change the constructor's " + "default. Chunks may span frames; their frame numbers are not " + "per-cluster metadata. Only the final chunk can be short.") .def( "read_clusters", [](ClusterFile &self, size_t n_clusters) { @@ -39,32 +112,60 @@ void define_ClusterFile(py::module &m, const std::string &typestr) { self.read_clusters(n_clusters)); return v; }, - py::return_value_policy::take_ownership, py::arg("n_clusters")) - .def("read_frame", - [](ClusterFile &self) { - auto v = new ClusterVector(self.read_frame()); - return v; - }) - .def("set_roi", &ClusterFile::set_roi, py::arg("roi")) - .def("tell", &ClusterFile::tell) + py::return_value_policy::take_ownership, py::arg("n_clusters"), + "Read up to n_clusters without preserving frame boundaries. The " + "result may combine frames, so its frame number is not reliable " + "per-cluster metadata.") + .def( + "read_frame", + [](ClusterFile &self) -> py::object { + auto clusters = self.read_frame(); + if (!clusters) { + return py::none(); + } + return py::cast( + new ClusterVector(std::move(*clusters)), + py::return_value_policy::take_ownership); + }, + "Read and return the next complete frame with its frame number, " + "or None at end of file.") + .def("set_roi", &ClusterFile::set_roi, py::arg("roi"), + "Select clusters whose centers lie within the half-open ROI.") + .def("tell", &ClusterFile::tell, + "Return the current byte position in the file.") .def("estimate_n_clusters", - &ClusterFile::estimate_n_clusters) + &ClusterFile::estimate_n_clusters, + "Estimate the number of clusters from the file size. Frame " + "headers can make this larger than the actual count.") .def( "set_noise_map", - [](ClusterFile &self, py::array_t noise_map) { + [](ClusterFile &self, + py::array_t noise_map) { auto view = make_view_2d(noise_map); self.set_noise_map(view); }, - py::arg("noise_map")) + py::arg("noise_map"), + "Set a two-dimensional, C-contiguous int32 noise map indexed as " + "[y, x]. The map must cover every cluster center coordinate.") - .def("set_gain_map", - [](ClusterFile &self, py::array_t gain_map) { - auto view = make_view_2d(gain_map); - self.set_gain_map(view); - }) + .def( + "set_gain_map", + [](ClusterFile &self, + py::array_t gain_map) { + auto view = make_view_2d(gain_map); + self.set_gain_map(view); + }, + py::arg("gain_map"), + "Set a two-dimensional, C-contiguous float64 gain map in " + "ADU/energy, indexed as [y, x]. Clusters whose complete footprint " + "extends beyond the map are retained with all data values set to " + "zero.") - .def("close", &ClusterFile::close) - .def("write_frame", &ClusterFile::write_frame) + .def("close", &ClusterFile::close, + "Close the file. Calling close more than once is safe.") + .def("write_frame", &ClusterFile::write_frame, + py::arg("clusters"), + "Write one ClusterVector, including its frame number.") .def("__enter__", [](ClusterFile &self) { return &self; }) .def("__exit__", [](ClusterFile &self, @@ -74,14 +175,8 @@ void define_ClusterFile(py::module &m, const std::string &typestr) { self.close(); }) .def("__iter__", [](ClusterFile &self) { return &self; }) - .def("__next__", [](ClusterFile &self) { - auto v = new ClusterVector( - self.read_clusters(self.chunk_size())); - if (v->size() == 0) { - throw py::stop_iteration(); - } - return v; - }); + .def("__next__", + [](File &self) { return ChunkIterator(self.chunks()).next(); }); } #pragma GCC diagnostic pop diff --git a/python/src/bind_ClusterFinder.hpp b/python/src/bind_ClusterFinder.hpp index c13d3305..2e0bf8b3 100644 --- a/python/src/bind_ClusterFinder.hpp +++ b/python/src/bind_ClusterFinder.hpp @@ -32,8 +32,10 @@ void define_ClusterFinder(py::module &m, const std::string &typestr) { py::class_>( m, class_name.c_str()) - .def(py::init, pd_type, size_t>(), py::arg("image_size"), - py::arg("n_sigma") = 5.0, py::arg("capacity") = 1'000'000) + .def(py::init, pd_type, size_t, size_t>(), + py::arg("image_size"), py::arg("n_sigma") = 5.0, + py::arg("capacity") = 1'000'000, + py::arg("min_pedestal_samples") = 1000) .def_property( "nSigma", diff --git a/python/src/bind_ClusterFinderMT.hpp b/python/src/bind_ClusterFinderMT.hpp index 252cc2f8..37ab8d47 100644 --- a/python/src/bind_ClusterFinderMT.hpp +++ b/python/src/bind_ClusterFinderMT.hpp @@ -32,10 +32,11 @@ void define_ClusterFinderMT(py::module &m, const std::string &typestr) { py::class_>( m, class_name.c_str()) - .def(py::init, pd_type, size_t, size_t, size_t>(), + .def(py::init, pd_type, size_t, size_t, size_t, size_t>(), py::arg("image_size"), py::arg("n_sigma") = 5.0, py::arg("capacity") = 2048, py::arg("n_threads") = 3, - py::arg("queue_depth") = 16) + py::arg("queue_depth") = 16, + py::arg("min_pedestal_samples") = 1000) .def("push_pedestal_frame", [](ClusterFinderMT &self, py::array_t frame) { diff --git a/python/src/bind_ClusterVector.hpp b/python/src/bind_ClusterVector.hpp index 8a5ef5aa..4f2eebf1 100644 --- a/python/src/bind_ClusterVector.hpp +++ b/python/src/bind_ClusterVector.hpp @@ -29,37 +29,73 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { py::class_, void>>( - m, class_name.c_str(), py::buffer_protocol(), - py::module_local()) + m, class_name.c_str(), R"doc( + A contiguous, move-only container of fixed-size clusters. - .def(py::init()) // TODO change!!! + The class supports the Python buffer protocol, so ``numpy.array`` can + either copy its data or create a zero-copy view. A zero-copy view is + valid only while the ClusterVector's underlying allocation and size + remain unchanged. + )doc", + py::buffer_protocol(), py::module_local()) + + .def(py::init(), R"doc( + Create an empty ClusterVector with frame number 0 and space reserved + for at least 1024 clusters. + )doc") .def( "__call__", - [](ClusterVector &self, py::array_t mask) { + [](ClusterVector &self, + py::array_t mask) { return self(make_view_1d(mask)); }, - py::arg("mask"), R"( - Create a copy of the clustervector and apply a boolean mask to the ClusterVector. + py::arg("mask").noconvert(), R"doc( + Return a filtered copy of this ClusterVector. Parameters ---------- + mask : numpy.ndarray + One-dimensional, writable, C-contiguous array with dtype + ``numpy.bool_`` and one element per cluster. - mask : 1d boolean numpy array - Mask to apply to the ClusterVector. Must be the same length as the number of clusters in the ClusterVector. + Returns + ------- + ClusterVector + Selected clusters in their original order. The frame number is + preserved. + )doc") - )") + .def( + "push_back", + [](ClusterVector &self, const ClusterType &cluster) { + self.push_back(cluster); + }, + py::arg("cluster"), R"doc( + Append one cluster. - .def("push_back", - [](ClusterVector &self, const ClusterType &cluster) { - self.push_back(cluster); - }) + Notes + ----- + Do not call this method while a zero-copy NumPy view of the + ClusterVector exists. Reallocation invalidates the view, while an + append without reallocation leaves its shape unchanged. + )doc") - .def("sum", - [](ClusterVector &self) { - auto *vec = new std::vector(self.sum()); - return return_vector(vec); - }) + .def( + "sum", + [](ClusterVector &self) { + auto *vec = new std::vector(self.sum()); + return return_vector(vec); + }, + R"doc( + Return the sum of all pixels in each cluster. + + Returns + ------- + numpy.ndarray + One value per cluster, in container order and with the cluster + pixel dtype. + )doc") .def( "sum_2x2", [](ClusterVector &self) { @@ -68,24 +104,45 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { return return_vector(vec); }, - R"(calculates sum of 2x2 subcluster with highest energy and index relative to cluster center 0: top_left, 1: top_right, 2: bottom_left, 3: bottom_right - )") - .def_property_readonly("size", &ClusterVector::size) - .def("empty", &ClusterVector::empty) - .def("item_size", &ClusterVector::item_size) - .def_property_readonly("fmt", - [typestr](ClusterVector &self) { - return fmt_format; - }) + R"doc( + Return the highest-sum center-adjacent 2x2 subcluster for each + cluster. + + Returns + ------- + numpy.ndarray + Structured array with ``sum`` and ``index`` fields. Indices are + 0 for top-left, 1 for top-right, 2 for bottom-left, and 3 for + bottom-right, relative to the cluster center. + )doc") + .def_property_readonly("size", &ClusterVector::size, + "Number of stored clusters.") + .def("empty", &ClusterVector::empty, + "Return True when no clusters are stored.") + .def("item_size", &ClusterVector::item_size, + "Return the size in bytes of one stored cluster, including " + "padding.") + .def_property_readonly( + "fmt", + [typestr](ClusterVector &self) { + return fmt_format; + }, + "PEP 3118 format string for one stored cluster.") .def_property_readonly("cluster_size_x", - &ClusterVector::cluster_size_x) + &ClusterVector::cluster_size_x, + "Cluster size in the x dimension.") .def_property_readonly("cluster_size_y", - &ClusterVector::cluster_size_y) + &ClusterVector::cluster_size_y, + "Cluster size in the y dimension.") .def_property_readonly("capacity", - &ClusterVector::capacity) + &ClusterVector::capacity, + "Number of clusters that fit without " + "reallocation.") .def_property("frame_number", &ClusterVector::frame_number, - &ClusterVector::set_frame_number) + &ClusterVector::set_frame_number, + "Signed 32-bit frame number; 0 can indicate clusters " + "from multiple frames.") .def_buffer( [typestr](ClusterVector &self) -> py::buffer_info { return py::buffer_info( @@ -99,31 +156,38 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { }); // Free functions using ClusterVector - m.def("hitmap", - [](std::array image_size, ClusterVector &cv) { - // Create a numpy array to hold the hitmap - // The shape of the array is (image_size[0], image_size[1]) - // note that the python array is passed as [row, col] which - // is the opposite of the clusters [x,y] - py::array_t hitmap(image_size); - auto r = hitmap.mutable_unchecked<2>(); + m.def( + "hitmap", + [](std::array image_size, ClusterVector &cv) { + // Create a numpy array to hold the hitmap + // The shape of the array is (image_size[0], image_size[1]) + // note that the python array is passed as [row, col] which + // is the opposite of the clusters [x,y] + py::array_t hitmap(image_size); + auto r = hitmap.mutable_unchecked<2>(); - // Initialize hitmap to 0 - for (py::ssize_t i = 0; i < r.shape(0); i++) - for (py::ssize_t j = 0; j < r.shape(1); j++) - r(i, j) = 0; + // Initialize hitmap to 0 + for (py::ssize_t i = 0; i < r.shape(0); i++) + for (py::ssize_t j = 0; j < r.shape(1); j++) + r(i, j) = 0; - // Loop over the clusters and increment the hitmap - // Skip out of bound clusters - for (const auto &cluster : cv) { - auto x = cluster.x; - auto y = cluster.y; - if (x < image_size[1] && y < image_size[0]) - r(cluster.y, cluster.x) += 1; - } + // Loop over the clusters and increment the hitmap + // Skip out of bound clusters + for (const auto &cluster : cv) { + auto x = cluster.x; + auto y = cluster.y; + if (x < image_size[1] && y < image_size[0]) + r(cluster.y, cluster.x) += 1; + } - return hitmap; - }); + return hitmap; + }, + py::arg("image_size"), py::arg("clusters"), R"doc( + Count cluster centers into an ``int32`` image whose shape is given by + ``image_size`` as ``(rows, columns)``. Element ``[y, x]`` contains the + number of cluster centers at that coordinate. Out-of-bounds centers are + ignored. + )doc"); } template >( reduce_to_2x2(cv)); }, - R"( - Reduce cluster to 2x2 subcluster by taking the 2x2 subcluster with - the highest photon energy. - - Parameters - - cv : ClusterVector (clusters are filled in row-major ordering starting at the top left. Thus for a max subcluster in the top left corner the photon hit is at the fourth position.) - - )", + R"doc( + Reduce every cluster to its highest-sum center-adjacent 2x2 block. + Returns a new ClusterVector; cluster order, coordinates, and frame + number are preserved. Input pixel data is interpreted in row-major + order. + )doc", py::arg("clustervector")); } @@ -159,13 +220,13 @@ void define_3x3_reduction(py::module &m) { return new ClusterVector>( reduce_to_3x3(cv)); }, - R"( - Reduce cluster to 3x3 subcluster - - Parameters - - cv : ClusterVector - )", + R"doc( + Reduce every cluster to the 3x3 block around its center index. + Both input dimensions must be at least 3, and at least one must be + greater than 3. + Returns a new ClusterVector; cluster order, coordinates, and frame + number are preserved. + )doc", py::arg("clustervector")); } diff --git a/python/src/bind_Defs.hpp b/python/src/bind_Defs.hpp index f0917884..c6e7d185 100644 --- a/python/src/bind_Defs.hpp +++ b/python/src/bind_Defs.hpp @@ -27,4 +27,77 @@ void define_defs_bindings(py::module &m) { moench05.attr("nRows") = Moench05::nRows; moench05.attr("nCols") = Moench05::nCols; moench05.attr("adcNumbers") = Moench05::adcNumbers; + + py::class_(m, "ROI") + .def(py::init<>()) + .def(py::init(), py::arg("xmin"), + py::arg("xmax"), py::arg("ymin"), py::arg("ymax")) + .def_readwrite("xmin", &ROI::xmin) + .def_readwrite("xmax", &ROI::xmax) + .def_readwrite("ymin", &ROI::ymin) + .def_readwrite("ymax", &ROI::ymax) + + .def( + "slice", + [](const ROI &self) { + return std::make_tuple(py::slice(self.ymin, self.ymax, 1), + py::slice(self.xmin, self.xmax, 1)); + }, + R"doc( + The slice can be used to index into a 2D array to extract the region of interest. + + Returns + ------- + tuple of slice + A tuple containing two slice objects for the x and y dimensions. + )doc") + + .def("__str__", + [](const ROI &self) { + return fmt::format("ROI: xmin: {} xmax: {} ymin: {} ymax: {}", + self.xmin, self.xmax, self.ymin, self.ymax); + }) + .def("__repr__", + [](const ROI &self) { + return fmt::format( + "", self.xmin, + self.xmax, self.ymin, self.ymax); + }) + .def("__iter__", + [](const ROI &self) { + return py::make_iterator(&self.xmin, &self.ymax + 1); // NOLINT + }) + + .def("__eq__", [](const ROI &self, const ROI &other) { + return self.xmin == other.xmin && self.xmax == other.xmax && + self.ymin == other.ymin && self.ymax == other.ymax; + }); + + py::enum_(m, "DetectorType") + .value("Jungfrau", DetectorType::Jungfrau) + .value("Eiger", DetectorType::Eiger) + .value("Mythen3", DetectorType::Mythen3) + .value("Moench", DetectorType::Moench) + .value("Moench03", DetectorType::Moench03) + .value("Moench03_old", DetectorType::Moench03_old) + .value("ChipTestBoard", DetectorType::ChipTestBoard) + .value("Unknown", DetectorType::Unknown); + + py::enum_(m, "UDPPortPosition") + .value("LEFT", UDPPortPosition::LEFT) + .value("RIGHT", UDPPortPosition::RIGHT) + .value("TOP", UDPPortPosition::TOP) + .value("BOTTOM", UDPPortPosition::BOTTOM); + + py::enum_(m, "TimingMode") + .value("AUTO_TIMING", TimingMode::AUTO_TIMING) + .value("TRIGGER_EXPOSURE", TimingMode::TRIGGER_EXPOSURE) + .value("GATED", TimingMode::GATED) + .value("BURST_TRIGGER", TimingMode::BURST_TRIGGER) + .value("TRIGGER_GATED", TimingMode::TRIGGER_GATED); + + py::enum_(m, "FrameDiscardPolicy") + .value("NoDiscard", FrameDiscardPolicy::NoDiscard) + .value("Discard", FrameDiscardPolicy::Discard) + .value("DiscardPartial", FrameDiscardPolicy::DiscardPartial); } diff --git a/python/src/bind_Eta.hpp b/python/src/bind_Eta.hpp index c27012f7..716b0e72 100644 --- a/python/src/bind_Eta.hpp +++ b/python/src/bind_Eta.hpp @@ -1,6 +1,8 @@ #include "aare/CalculateEta.hpp" +#include "aare/to_string.hpp" #include +#include // #include only for version 3 #include @@ -18,7 +20,11 @@ void define_eta(py::module &m, const std::string &typestr) { .def_readwrite("c", &Eta2::c, "eta corner value cTopLeft, cTopRight, " "cBottomLeft, cBottomRight") - .def_readwrite("sum", &Eta2::sum, "photon energy of cluster"); + .def_readwrite("sum", &Eta2::sum, "photon energy of cluster") + .def("__repr__", [class_name](const Eta2 &self) { + return fmt::format("{}(x={}, y={}, c={}, sum={})", class_name, + self.x, self.y, to_string(self.c), self.sum); + }); } void define_corner_enum(py::module &m) { @@ -101,4 +107,4 @@ void register_calculate_3x3eta(py::module &m) { }, R"(calculates eta3x3 taking into account cross pixels in cluster)", py::arg("cluster")); -} \ No newline at end of file +} diff --git a/python/src/bind_FastPedestal.hpp b/python/src/bind_FastPedestal.hpp index c1911e52..b2b4fb8f 100644 --- a/python/src/bind_FastPedestal.hpp +++ b/python/src/bind_FastPedestal.hpp @@ -15,8 +15,7 @@ void define_fast_pedestal_bindings(py::module &m, const std::string &name) { py::class_>( m, name.c_str(), - "Maintain a per-pixel running mean, population variance, and " - "standard deviation.", + "Maintain a per-pixel running mean and population standard deviation.", py::buffer_protocol()) .def(py::init(), py::arg("rows"), py::arg("cols"), py::arg("n_samples"), @@ -33,15 +32,6 @@ void define_fast_pedestal_bindings(py::module &m, const std::string &name) { return return_image_data(mean); }, "Return a copy of the cached mean. The pedestal must be ready.") - .def( - "var", - [](FastPedestal &self) { - auto variance = new NDArray{}; - *variance = self.variance(); - return return_image_data(variance); - }, - "Return the population variance, normalized by n_samples, as a " - "NumPy array. The pedestal must be ready.") .def( "std", [](FastPedestal &self) { diff --git a/python/src/bind_Pedestal.hpp b/python/src/bind_Pedestal.hpp new file mode 100644 index 00000000..602ee13e --- /dev/null +++ b/python/src/bind_Pedestal.hpp @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/Pedestal.hpp" +#include "np_helper.hpp" + +#include +#include +#include +#include +#include + +namespace py = pybind11; + +template +void define_pedestal_bindings(py::module &m, const std::string &name) { + + py::class_>( + m, name.c_str(), + "Maintain a per-pixel running mean and population standard deviation. " + "Statistics are available during initialization and are zero for empty " + "pixels.", + py::buffer_protocol()) + .def(py::init(), py::arg("rows"), + py::arg("cols"), py::arg("n_samples"), + "Construct an empty pedestal. Each pixel accumulates n_samples " + "values before switching to exponential updates.") + .def(py::init(), py::arg("rows"), py::arg("cols"), + "Construct an empty pedestal with n_samples=1000.") + .def( + "mean", + [](Pedestal &self) { + auto mea = new NDArray{}; + *mea = self.mean(); + return return_image_data(mea); + }, + "Return a copy of the cached mean. Empty pixels return zero.") + .def( + "view", + [](py::object self_py) { + return py::module_::import("numpy").attr("asarray")(self_py); + }, + "Return a non-owning, non-writable NumPy view of the cached mean.") + .def( + "std", + [](Pedestal &self) { + auto std = new NDArray{}; + *std = self.std(); + return return_image_data(std); + }, + "Return the population standard deviation as a NumPy array. " + "Empty pixels return zero.") + .def( + "__array_ufunc__", + [](py::object self, py::object ufunc, const std::string &method, + py::args inputs, py::kwargs kwargs) -> py::object { + if (method != "__call__" || inputs.size() != 2 || + inputs[1].ptr() != self.ptr() || + py::cast(ufunc.attr("__name__")) != + "subtract") { + return py::reinterpret_borrow( + Py_NotImplemented); + } + + auto mean = + py::module_::import("builtins").attr("memoryview")(self); + return ufunc(inputs[0], mean, **kwargs); + }, + "Support subtracting a Pedestal from a NumPy array.") + .def("clear", py::overload_cast<>(&Pedestal::clear), + "Reset all statistics and per-pixel sample counts to zero.") + .def_property_readonly("rows", &Pedestal::rows, + "Number of image rows.") + .def_property_readonly("cols", &Pedestal::cols, + "Number of image columns.") + .def_property_readonly( + "n_samples", &Pedestal::n_samples, + "Initialization sample count per pixel and steady-state " + "update-weight denominator.") + .def( + "clone", + [&](Pedestal &pedestal) { + return Pedestal(pedestal); + }, + "Return an independent copy of the pedestal and its state.") + // TODO! add push for other data types + .def( + "push", + [](Pedestal &pedestal, + py::array_t &f) { + if (f.ndim() != 2) { + throw py::value_error("Frame must be 2-dimensional"); + } + auto v = make_view_2d(f); + pedestal.push(v); + }, + py::arg("frame").noconvert(), + "Accumulate or exponentially update every pixel from a " + "C-contiguous uint16 frame matching the pedestal shape. After " + "n_samples values per pixel, new values have weight 1 / n_samples.") + .def( + "push_with_threshold", + [](Pedestal &pedestal, + py::array_t &f, + py::array_t &threshold) { + if (f.ndim() != 2) { + throw py::value_error("Frame must be 2-dimensional"); + } + if (threshold.ndim() != 2) { + throw py::value_error("Threshold must be 2-dimensional"); + } + auto frame_view = make_view_2d(f); + auto threshold_view = make_view_2d(threshold); + pedestal.push_with_threshold(frame_view, threshold_view); + }, + py::arg("frame").noconvert(), py::arg("threshold").noconvert(), + "Push only pixels where abs(frame - mean) is strictly less than " + "threshold. Both arrays must be C-contiguous with the pedestal " + "shape; frame must be uint16 and threshold must use the output " + "dtype. Rejected pixels keep their statistics and sample counts.") + .def_buffer([](Pedestal &self) { + auto mean = self.view(); + return py::buffer_info( + const_cast(mean.data()), sizeof(SUM_TYPE), + py::format_descriptor::format(), 2, + {static_cast(mean.shape(0)), + static_cast(mean.shape(1))}, + {static_cast(mean.strides()[0] * sizeof(SUM_TYPE)), + static_cast(mean.strides()[1] * + sizeof(SUM_TYPE))}, + true); + }); +} diff --git a/python/src/bind_RawFile.hpp b/python/src/bind_RawFile.hpp index 85ef155f..2c4e20c8 100644 --- a/python/src/bind_RawFile.hpp +++ b/python/src/bind_RawFile.hpp @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -21,14 +22,23 @@ namespace py = pybind11; using namespace ::aare; +namespace { +std::runtime_error raw_file_read_error(RawFile &file, + const std::string &message) { + return std::runtime_error(fmt::format( + "Error reading frame index {} from file '{}': {}", file.tell(), + file.master().master_fname().string(), message)); +} +} // namespace + void define_raw_file_io_bindings(py::module &m) { py::class_(m, "RawFile") .def(py::init()) .def("read_frame", [](RawFile &self) { if (self.n_modules_in_roi().size() > 1) { - throw std::runtime_error( - "File contains multiple ROIs - use read_ROIs()"); + throw raw_file_read_error( + self, "File contains multiple ROIs - use read_ROIs()"); } std::vector shape; @@ -51,16 +61,19 @@ void define_raw_file_io_bindings(py::module &m) { "read_n", [](RawFile &self, size_t n_frames) { if (self.n_modules_in_roi().size() > 1) { - throw std::runtime_error( + throw raw_file_read_error( + self, "File contains multiple ROIs - use read_n_ROIs() to " "read a specific ROI or use read_ROIs and " "read one frame at a time."); } // adjust for actual frames left in the file n_frames = - std::min(n_frames, self.total_frames() - self.tell()); + std::min(n_frames, self.tell() < self.total_frames() + ? self.total_frames() - self.tell() + : size_t{0}); if (n_frames == 0) { - throw std::runtime_error("No frames left in file"); + throw raw_file_read_error(self, "No frames left in file"); } std::vector shape{n_frames, self.rows(), self.cols()}; @@ -89,12 +102,13 @@ void define_raw_file_io_bindings(py::module &m) { "read_roi", [](RawFile &self, const size_t roi_index) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } if (roi_index >= self.num_rois()) { - throw std::runtime_error(LOCATION + - "ROI index out of range."); + throw raw_file_read_error( + self, LOCATION + "ROI index out of range."); } // return headers from all subfiles @@ -141,7 +155,8 @@ void define_raw_file_io_bindings(py::module &m) { "read_rois", [](RawFile &self) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } size_t number_of_ROIs = self.num_rois(); @@ -196,19 +211,22 @@ void define_raw_file_io_bindings(py::module &m) { "read_n_with_roi", [](RawFile &self, const size_t num_frames, const size_t roi_index) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } if (roi_index >= self.num_rois()) { - throw std::runtime_error(LOCATION + - "ROI index out of range."); + throw raw_file_read_error( + self, LOCATION + "ROI index out of range."); } // adjust for actual frames left in the file size_t n_frames = - std::min(num_frames, self.total_frames() - self.tell()); + std::min(num_frames, self.tell() < self.total_frames() + ? self.total_frames() - self.tell() + : size_t{0}); if (n_frames == 0) { - throw std::runtime_error("No frames left in file"); + throw raw_file_read_error(self, "No frames left in file"); } std::vector shape{ n_frames, self.roi_geometries(roi_index).pixels_y(), @@ -306,4 +324,4 @@ void define_raw_file_io_bindings(py::module &m) { .def_property_readonly("n_modules", &RawFile::n_modules) .def_property_readonly("n_modules_in_roi", &RawFile::n_modules_in_roi) .def_property_readonly("num_rois", &RawFile::num_rois); -} \ No newline at end of file +} diff --git a/python/src/bind_calibration.hpp b/python/src/bind_calibration.hpp index a4a3f160..c9479e8a 100644 --- a/python/src/bind_calibration.hpp +++ b/python/src/bind_calibration.hpp @@ -73,10 +73,27 @@ py::array_t pybind_calculate_pedestal_g0( } void bind_calibration(py::module &m) { + py::options options; + options.disable_function_signatures(); + m.def("apply_calibration", &pybind_apply_calibration, py::arg("raw_data").noconvert(), py::kw_only(), py::arg("pd").noconvert(), py::arg("cal").noconvert(), - py::arg("n_threads") = 4); + py::arg("n_threads") = 4, + R"( + Apply the calibration to the raw data and return the result as a 3D array of doubles. + + Parameters + ---------- + raw_data : array_like + 3D array of shape ``(frames, rows, cols)``. + pd : array_like + Pedestal array. + cal : array_like + Calibration array. + n_threads : int, optional + Number of threads to use. Defaults to 4. + )"); m.def("apply_calibration", &pybind_apply_calibration, py::arg("raw_data").noconvert(), py::kw_only(), diff --git a/python/src/bind_testing.hpp b/python/src/bind_testing.hpp new file mode 100644 index 00000000..2a4e8760 --- /dev/null +++ b/python/src/bind_testing.hpp @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: MPL-2.0 +// Probes used by the python test-suite to exercise the helpers in +// np_helper.hpp. Internal, no API guarantees. +#pragma once + +#include "aare/NDArray.hpp" +#include "aare/NDView.hpp" +#include "np_helper.hpp" + +#include +#include +#include +#include +#include +#include + +namespace py = pybind11; +using namespace ::aare; + +// Arrays are taken with the default flags (no c_style) so that nothing is +// made contiguous by pybind11 before it reaches make_view +template +void define_make_view_probes(py::module &m, const std::string &suffix) { + + // Copy what C++ sees through the view back to python + m.def(("view_roundtrip_" + suffix).c_str(), + [](py::array_t arr) { + auto view = make_view(arr); + return return_image_data(new NDArray(view)); + }, + py::arg("arr").noconvert()); + + // Same as above but through a read only view + m.def(("const_view_roundtrip_" + suffix).c_str(), + [](py::array_t arr) { + auto view = make_const_view(arr); + static_assert( + std::is_same_v>); + auto out = new NDArray(view.shape()); + std::copy(view.begin(), view.end(), out->begin()); + return return_image_data(out); + }, + py::arg("arr").noconvert()); + + // Write through the view + m.def(("view_fill_" + suffix).c_str(), + [](py::array_t arr, T value) { + auto view = make_view(arr); + view = value; + }, + py::arg("arr").noconvert(), py::arg("value")); +} + +void define_testing_bindings(py::module &m) { + auto testing = m.def_submodule( + "testing", "Internal helpers for the test-suite, no API guarantees"); + + // noconvert + overloads gives dispatch on dtype + define_make_view_probes(testing, "1d"); + define_make_view_probes(testing, "2d"); + define_make_view_probes(testing, "3d"); + define_make_view_probes(testing, "1d"); + define_make_view_probes(testing, "2d"); + define_make_view_probes(testing, "3d"); +} diff --git a/python/src/file.hpp b/python/src/file.hpp index ccb62d1f..a493c910 100644 --- a/python/src/file.hpp +++ b/python/src/file.hpp @@ -2,6 +2,7 @@ #include "aare/CtbRawFile.hpp" #include "aare/File.hpp" #include "aare/Frame.hpp" +#include "aare/ROI.hpp" #include "aare/RawFile.hpp" #include "aare/RawMasterFile.hpp" #include "aare/RawSubFile.hpp" @@ -27,16 +28,6 @@ using namespace ::aare; void define_file_io_bindings(py::module &m) { - py::enum_(m, "DetectorType") - .value("Jungfrau", DetectorType::Jungfrau) - .value("Eiger", DetectorType::Eiger) - .value("Mythen3", DetectorType::Mythen3) - .value("Moench", DetectorType::Moench) - .value("Moench03", DetectorType::Moench03) - .value("Moench03_old", DetectorType::Moench03_old) - .value("ChipTestBoard", DetectorType::ChipTestBoard) - .value("Unknown", DetectorType::Unknown); - PYBIND11_NUMPY_DTYPE(DetectorHeader, frameNumber, expLength, packetNumber, bunchId, timestamp, modId, row, column, reserved, debug, roundRNumber, detType, version, packetMask); @@ -170,28 +161,5 @@ void define_file_io_bindings(py::module &m) { .def_property_readonly("stop", &ScanParameters::stop) .def_property_readonly("step", &ScanParameters::step); - py::class_(m, "ROI") - .def(py::init<>()) - .def(py::init(), py::arg("xmin"), - py::arg("xmax"), py::arg("ymin"), py::arg("ymax")) - .def_readwrite("xmin", &ROI::xmin) - .def_readwrite("xmax", &ROI::xmax) - .def_readwrite("ymin", &ROI::ymin) - .def_readwrite("ymax", &ROI::ymax) - .def("__str__", - [](const ROI &self) { - return fmt::format("ROI: xmin: {} xmax: {} ymin: {} ymax: {}", - self.xmin, self.xmax, self.ymin, self.ymax); - }) - .def("__repr__", - [](const ROI &self) { - return fmt::format( - "", self.xmin, - self.xmax, self.ymin, self.ymax); - }) - .def("__iter__", [](const ROI &self) { - return py::make_iterator(&self.xmin, &self.ymax + 1); // NOLINT - }); - #pragma GCC diagnostic pop } \ No newline at end of file diff --git a/python/src/module.cpp b/python/src/module.cpp index 3908855f..1c2eff8e 100644 --- a/python/src/module.cpp +++ b/python/src/module.cpp @@ -17,18 +17,19 @@ #include "bind_FastPedestal.hpp" #include "bind_Interpolator.hpp" #include "bind_MultiThreadedFileReader.hpp" +#include "bind_Pedestal.hpp" #include "bind_PedestalTrackingPixelHistogram.hpp" #include "bind_PixelHistogram.hpp" #include "bind_PixelMap.hpp" #include "bind_RawFile.hpp" #include "bind_calibration.hpp" +#include "bind_testing.hpp" // TODO! migrate the other names #include "ctb_raw_file.hpp" #include "file.hpp" #include "fit.hpp" #include "jungfrau_data_file.hpp" -#include "pedestal.hpp" #include "raw_master_file.hpp" #include "raw_sub_file.hpp" #include "var_cluster.hpp" @@ -90,6 +91,7 @@ PYBIND11_MODULE(_aare, m) { define_jungfrau_data_file_io_bindings(m); bind_calibration(m); + define_testing_bindings(m); DEFINE_CLUSTER_BINDINGS(int, 3, 3, uint16_t, i); DEFINE_CLUSTER_BINDINGS(double, 3, 3, uint16_t, d); @@ -130,9 +132,6 @@ PYBIND11_MODULE(_aare, m) { DEFINE_BINDINGS_CLUSTERFINDER(double, 9, 9, uint16_t, d); DEFINE_BINDINGS_CLUSTERFINDER(float, 9, 9, uint16_t, f); - define_3x3_reduction(m); - define_3x3_reduction(m); - define_3x3_reduction(m); define_3x3_reduction(m); define_3x3_reduction(m); define_3x3_reduction(m); @@ -143,9 +142,6 @@ PYBIND11_MODULE(_aare, m) { define_3x3_reduction(m); define_3x3_reduction(m); - reduce_to_3x3(m); - reduce_to_3x3(m); - reduce_to_3x3(m); reduce_to_3x3(m); reduce_to_3x3(m); reduce_to_3x3(m); diff --git a/python/src/np_helper.hpp b/python/src/np_helper.hpp index c0f5c3fd..d21aceff 100644 --- a/python/src/np_helper.hpp +++ b/python/src/np_helper.hpp @@ -41,31 +41,60 @@ template py::array return_vector(std::vector *vec) { free_when_done); // numpy array references this parent } -// todo rewrite generic -template -auto get_shape_3d(const py::array_t &arr) { - return aare::Shape<3>{arr.shape(0), arr.shape(1), arr.shape(2)}; +// Create a NDView of a numpy array. NDView assumes C-order strides so we +// reject anything that is not Ndim dimensional and C-contiguous instead of +// silently reading the data in the wrong order. +template +auto get_checked_shape(const py::array_t &arr) { + if (arr.ndim() != Ndim) { + throw py::value_error( + fmt::format("Expected {}D array, got {}D", Ndim, arr.ndim())); + } + if (!(arr.flags() & py::array::c_style)) { + throw py::value_error( + "Array is not C-contiguous. Use np.ascontiguousarray(arr)"); + } + aare::Shape shape{}; + for (ssize_t i = 0; i < Ndim; ++i) { + shape[i] = arr.shape(i); + } + return shape; +} + +template +auto make_view(py::array_t &arr) { + auto shape = get_checked_shape(arr); + return aare::NDView(arr.mutable_data(), shape); +} + +// Read only view, also works for numpy arrays that are not writeable +template +auto make_const_view(const py::array_t &arr) { + auto shape = get_checked_shape(arr); + return aare::NDView(arr.data(), shape); } template auto make_view_3d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_3d(arr)); + return make_view<3>(arr); } - -template -auto get_shape_2d(const py::array_t &arr) { - return aare::Shape<2>{arr.shape(0), arr.shape(1)}; -} - -template -auto get_shape_1d(const py::array_t &arr) { - return aare::Shape<1>{arr.shape(0)}; -} - template auto make_view_2d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_2d(arr)); + return make_view<2>(arr); } template auto make_view_1d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_1d(arr)); + return make_view<1>(arr); +} + +template +auto make_const_view_3d(const py::array_t &arr) { + return make_const_view<3>(arr); +} +template +auto make_const_view_2d(const py::array_t &arr) { + return make_const_view<2>(arr); +} +template +auto make_const_view_1d(const py::array_t &arr) { + return make_const_view<1>(arr); } template struct fmt_format_trait; // forward declaration diff --git a/python/src/pedestal.hpp b/python/src/pedestal.hpp deleted file mode 100644 index 229f3adf..00000000 --- a/python/src/pedestal.hpp +++ /dev/null @@ -1,125 +0,0 @@ -// SPDX-License-Identifier: MPL-2.0 - -#include "aare/Pedestal.hpp" -#include "np_helper.hpp" - -#include -#include -#include -#include -#include - -namespace py = pybind11; - -template -void define_pedestal_bindings(py::module &m, const std::string &name) { - - py::class_>(m, name.c_str(), py::buffer_protocol()) - .def(py::init()) - .def(py::init()) - .def("mean", - [](Pedestal &self) { - auto mea = new NDArray{}; - *mea = self.mean(); - return return_image_data(mea); - }) - .def("view", - [](py::object self_py) { - auto &self = self_py.cast &>(); - auto v = self.view(); - std::array shape{ - static_cast(v.shape(0)), - static_cast(v.shape(1))}; - std::array byte_strides{ - static_cast(v.strides()[0]) * - static_cast(sizeof(SUM_TYPE)), - static_cast(v.strides()[1]) * - static_cast(sizeof(SUM_TYPE))}; - auto arr = py::array_t(shape, byte_strides, v.data(), - self_py); - arr.attr("setflags")(py::arg("write") = false); - return arr; - }) - .def("variance", - [](Pedestal &self) { - auto var = new NDArray{}; - *var = self.variance(); - return return_image_data(var); - }) - .def("std", - [](Pedestal &self) { - auto std = new NDArray{}; - *std = self.std(); - return return_image_data(std); - }) - .def("cached_std", - [](Pedestal &self) { - auto standard_deviation = new NDArray{}; - *standard_deviation = self.cached_std(); - return return_image_data(standard_deviation); - }) - .def( - "__array_ufunc__", - [](py::object self, py::object ufunc, const std::string &method, - py::args inputs, py::kwargs kwargs) -> py::object { - if (method != "__call__" || inputs.size() != 2 || - inputs[1].ptr() != self.ptr() || - py::cast(ufunc.attr("__name__")) != - "subtract") { - return py::reinterpret_borrow( - Py_NotImplemented); - } - - auto mean = - py::module_::import("builtins").attr("memoryview")(self); - return ufunc(inputs[0], mean, **kwargs); - }, - "Support subtracting a Pedestal from a NumPy array.") - .def("clear", py::overload_cast<>(&Pedestal::clear)) - .def_property_readonly("rows", &Pedestal::rows) - .def_property_readonly("cols", &Pedestal::cols) - .def_property_readonly("n_samples", &Pedestal::n_samples) - .def_property_readonly("sum", &Pedestal::get_sum) - .def_property_readonly("sum2", &Pedestal::get_sum2) - .def("clone", - [&](Pedestal &pedestal) { - return Pedestal(pedestal); - }) - // TODO! add push for other data types - .def("push", - [](Pedestal &pedestal, py::array_t &f) { - auto v = make_view_2d(f); - pedestal.push(v); - }) - .def( - "push_with_threshold", - [](Pedestal &pedestal, - py::array_t &f, - py::array_t &threshold) { - auto frame_view = make_view_2d(f); - auto threshold_view = make_view_2d(threshold); - pedestal.push_with_threshold(frame_view, threshold_view); - }, - py::arg("frame").noconvert(), py::arg("threshold").noconvert()) - .def( - "push_no_update", - [](Pedestal &pedestal, - py::array_t &f) { - auto v = make_view_2d(f); - pedestal.push_no_update(v); - }, - py::arg().noconvert()) - .def("update_mean", &Pedestal::update_mean) - .def_buffer([](Pedestal &self) { - auto mean = self.view(); - return py::buffer_info( - const_cast(mean.data()), sizeof(SUM_TYPE), - py::format_descriptor::format(), 2, - {static_cast(mean.shape(0)), - static_cast(mean.shape(1))}, - {static_cast(mean.strides()[0] * sizeof(SUM_TYPE)), - static_cast(mean.strides()[1] * - sizeof(SUM_TYPE))}, - true); - }); -} diff --git a/python/src/raw_master_file.hpp b/python/src/raw_master_file.hpp index 7f1837c0..40e942a9 100644 --- a/python/src/raw_master_file.hpp +++ b/python/src/raw_master_file.hpp @@ -66,7 +66,8 @@ void define_raw_master_file_bindings(py::module &m) { .def_property_readonly("total_frames_expected", &RawMasterFile::total_frames_expected) - .def_property_readonly("geometry", &RawMasterFile::geometry) + .def_property_readonly("detector_layout", + &RawMasterFile::detector_layout) .def_property_readonly("udp_interfaces_per_module", &RawMasterFile::udp_interfaces_per_module) .def_property_readonly("analog_samples", &RawMasterFile::analog_samples, @@ -107,9 +108,46 @@ void define_raw_master_file_bindings(py::module &m) { return std::nullopt; } }) - .def_property_readonly("period", [](RawMasterFile &self) { - double seconds = - std::chrono::duration(self.period()).count(); - return seconds; - }); + .def_property_readonly("rois", &RawMasterFile::rois, R"( + Get the ROIs defined in the master file + + Returns + ---------- + List[ROI] + List of ROIs (default complete ROI) + )") + .def_property_readonly("udp_port_types", &RawMasterFile::udp_port_types, + R"( + Get the types of UDP ports + + Returns + ---------- + Optional[List[UDPPortPosition]] + Optional vector of UDP port types as strings (only present for + masterfile version >= 8.1) + )") + .def_property_readonly("disabled_udp_ports", + &RawMasterFile::disabled_udp_ports, R"( + Get the indices of disabled UDP ports + + Returns + ---------- + List[int] + Vector of disabled UDP port indices relative to UDP port types (empty if none are disabled) + )") + .def_property_readonly( + "period", + [](RawMasterFile &self) { + double seconds = + std::chrono::duration(self.period()).count(); + return seconds; + }) + + .def("__enter__", + [](RawMasterFile &self) -> RawMasterFile & { return self; }) + + .def("__exit__", + []([[maybe_unused]] RawMasterFile &self, + [[maybe_unused]] py::object, [[maybe_unused]] py::object, + [[maybe_unused]] py::object) { return; }); } diff --git a/python/tests/test_Cluster.py b/python/tests/test_Cluster.py index fece7273..027cc203 100644 --- a/python/tests/test_Cluster.py +++ b/python/tests/test_Cluster.py @@ -3,7 +3,7 @@ import pytest import numpy as np from aare import _aare #import the C++ module -from aare import corner +from aare import ROI, corner from conftest import test_data_path @@ -91,6 +91,22 @@ def test_calculate_eta(): assert eta2.sum == 4 +def test_cluster_representations(): + """Test Cluster representation""" + + cluster = _aare.Cluster2x2i(5, 6, np.array([1, 2, 3, 4], dtype=np.int32)) + assert repr(cluster) == "Cluster2x2i(x=5, y=6, data=[[1, 2], [3, 4]])" + +def test_eta_representations(): + """Test Eta representation""" + eta = _aare.Etai() + eta.x = 0.25 + eta.y = 0.75 + eta.c = corner.cBottomRight + eta.sum = 42 + assert repr(eta) == "Etai(x=0.25, y=0.75, c=BottomRight, sum=42)" + + def test_max_sum(): """Max 2x2 Sum""" cluster = _aare.Cluster3x3i(5,5,np.array([1, 1, 1, 2, 3, 1, 2, 2, 1], dtype=np.int32)) @@ -146,4 +162,3 @@ def test_3x3_reduction(): - diff --git a/python/tests/test_ClusterFile.py b/python/tests/test_ClusterFile.py index 3e2fb81d..6ce8a11b 100644 --- a/python/tests/test_ClusterFile.py +++ b/python/tests/test_ClusterFile.py @@ -6,10 +6,279 @@ import boost_histogram as bh import time from pathlib import Path import pickle +import struct +import gc +import weakref -from aare import ClusterFile +from aare import ClusterFile, ROI from conftest import test_data_path + +@pytest.fixture +def iterator_file(tmp_path): + fname = tmp_path / "iteration.clust" + records = [ + (-11, []), + (42, [(5, 6, 10), (7, 8, 20)]), + (103, []), + (104, [(5, 6, 30), (7, 8, 40)]), + (105, []), + ] + fname.write_bytes(b"".join( + struct.pack("@iI", number, len(clusters)) + + b"".join(struct.pack("@HH9i", x, y, *([value] * 9)) + for x, y, value in clusters) + for number, clusters in records + )) + return fname + + +def test_frames_preserve_empty_frames_and_frame_numbers(iterator_file): + with ClusterFile(iterator_file) as reader: + frames = reader.frames() + assert iter(frames) is frames + assert reader.tell() == 0 + results = list(frames) + assert [frame.frame_number for frame in results] == [-11, 42, 103, 104, 105] + assert [frame.size for frame in results] == [0, 2, 0, 2, 0] + for _ in range(2): + with pytest.raises(StopIteration): + next(frames) + assert list(reader.frames()) == [] + + +@pytest.mark.parametrize("chunk_size, sizes", [(1, [1, 1, 1, 1]), (2, [2, 2]), + (3, [3, 1]), (10, [4])]) +@pytest.mark.parametrize("explicit", [False, True]) +def test_chunks_cross_frames(iterator_file, chunk_size, sizes, explicit): + with ClusterFile(iterator_file, chunk_size=chunk_size) as reader: + chunks = reader.chunks(chunk_size=chunk_size) if explicit else reader.chunks() + assert iter(chunks) is chunks + assert reader.tell() == 0 + results = list(chunks) + assert [chunk.size for chunk in results] == sizes + data = np.concatenate([np.asarray(chunk) for chunk in results]) + np.testing.assert_array_equal(data["x"], [5, 7, 5, 7]) + np.testing.assert_array_equal(data["data"][:, 0, 0], [10, 20, 30, 40]) + for _ in range(2): + with pytest.raises(StopIteration): + next(chunks) + + +def test_chunk_override_preserves_default_iteration(iterator_file): + with ClusterFile(iterator_file, chunk_size=2) as reader: + assert iter(reader) is reader + assert next(reader.chunks(1)).size == 1 + assert [chunk.size for chunk in reader] == [2, 1] + + +@pytest.mark.parametrize("use_roi", [False, True]) +@pytest.mark.parametrize("use_noise", [False, True]) +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_apply_filters_and_gain(iterator_file, use_roi, use_noise, method): + with ClusterFile(iterator_file, chunk_size=3) as reader: + if use_roi: + reader.set_roi(ROI(0, 6, 0, 12)) + if use_noise: + reader.set_noise_map(np.full((12, 12), 15, dtype=np.int32)) + reader.set_gain_map(np.full((12, 12), 2.0)) + results = list(getattr(reader, method)()) + if method == "frames": + assert [frame.frame_number for frame in results] == [-11, 42, 103, 104, 105] + expected = [value / 2 for x, value in [(5, 10), (7, 20), (5, 30), (7, 40)] + if (not use_roi or x == 5) and (not use_noise or value > 15)] + data = np.concatenate([np.asarray(result) for result in results]) + np.testing.assert_array_equal(data["data"][:, 0, 0], expected) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_handle_all_rejected_clusters(iterator_file, method): + with ClusterFile(iterator_file) as reader: + reader.set_roi(ROI(0, 1, 0, 1)) + results = list(getattr(reader, method)()) + assert len(results) == (5 if method == "frames" else 0) + assert all(result.size == 0 for result in results) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterator_keeps_file_alive(iterator_file, method): + reader = ClusterFile(iterator_file) + owner = weakref.ref(reader) + iterator = getattr(reader, method)() + del reader + gc.collect() + assert owner() is not None + results = list(iterator) + assert sum(result.size for result in results) == 4 + del iterator + gc.collect() + assert owner() is None + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_retained_results_and_numpy_views_own_storage(iterator_file, method): + with ClusterFile(iterator_file, chunk_size=2) as reader: + reader.read_frame() # Skip the initial empty frame. + iterator = getattr(reader, method)() + first = next(iterator) + array = np.asarray(first) + expected = array.copy() + results = list(iterator) + np.testing.assert_array_equal(np.asarray(first), expected) + del first, results, iterator, reader + gc.collect() + np.testing.assert_array_equal(array, expected) + + +def test_frame_iteration_resumes_without_reading_ahead(iterator_file): + with ClusterFile(iterator_file) as reader: + reader.read_frame() + for frame in reader.frames(): + assert frame.frame_number == 42 + break + assert reader.read_frame().frame_number == 103 + assert [frame.frame_number for frame in reader.frames()] == [104, 105] + + +def test_partial_chunk_must_be_completed_before_frame_iteration(iterator_file): + with ClusterFile(iterator_file) as reader: + for chunk in reader.chunks(1): + assert chunk.size == 1 + break + with pytest.raises(RuntimeError, match="clusters left"): + next(reader.frames()) + assert reader.read_clusters(1).size == 1 + assert [frame.frame_number for frame in reader.frames()] == [103, 104, 105] + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_empty_iterators(tmp_path, method): + fname = tmp_path / "empty.clust" + fname.touch() + with ClusterFile(fname) as reader: + iterator = getattr(reader, method)() + for _ in range(2): + with pytest.raises(StopIteration): + next(iterator) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +@pytest.mark.parametrize("started", [False, True]) +def test_iterators_reject_closed_files(iterator_file, method, started): + reader = ClusterFile(iterator_file, chunk_size=1) + iterator = getattr(reader, method)() + if started: + next(iterator) + reader.close() + with pytest.raises(RuntimeError, match="not opened for reading"): + next(iterator) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +@pytest.mark.parametrize("mode", ["w", "a"]) +def test_iterators_reject_writing_modes(tmp_path, method, mode): + with ClusterFile(tmp_path / "output.clust", mode=mode) as reader: + with pytest.raises(RuntimeError, match="not opened for reading"): + next(getattr(reader, method)()) + + +def test_chunk_sizes_must_be_positive(iterator_file): + with ClusterFile(iterator_file, chunk_size=0) as reader: + with pytest.raises(ValueError, match="greater than zero"): + reader.chunks() + with pytest.raises(ValueError, match="greater than zero"): + reader.chunks(0) + with pytest.raises(ValueError, match="greater than zero"): + next(reader) + with pytest.raises(TypeError): + reader.chunks(-1) + assert reader.tell() == 0 + assert reader.read_clusters(0).size == 0 + assert len(list(reader.frames())) == 5 + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_defer_read_errors_until_next_result(tmp_path, method): + fname = tmp_path / "truncated.clust" + cluster = struct.pack("@HH9i", 5, 6, *range(9)) + frame = struct.pack("@iI", 42, 1) + cluster + fname.write_bytes(frame + frame[:-1]) + with ClusterFile(fname, chunk_size=1) as reader: + iterator = getattr(reader, method)() + assert next(iterator).size == 1 + assert reader.tell() == len(frame) + with pytest.raises(RuntimeError): + next(iterator) + + +@pytest.mark.parametrize( + "shape, dtype", + [(shape, dtype) + for shape in [(2, 2), (3, 3), (5, 5), (7, 7), (9, 9)] + for dtype in [np.int32, np.float32, np.float64]] + + [((3, 3), np.int16)], +) +def test_iterators_are_bound_for_each_cluster_type(tmp_path, shape, dtype): + fname = tmp_path / "typed_frames.clust" + values = list(range(shape[0] * shape[1])) + record = struct.pack("@HH" + np.dtype(dtype).char * len(values), 5, 6, *values) + fname.write_bytes((struct.pack("@iI", -42, 1) + record) * 3) + with ClusterFile(fname, cluster_size=shape, dtype=dtype) as reader: + frame = next(reader.frames()) + assert frame.frame_number == -42 + np.testing.assert_array_equal(np.asarray(frame)["data"].reshape(-1), values) + chunks = list(reader.chunks()) + assert len(chunks) == 1 + assert chunks[0].size == 2 + np.testing.assert_array_equal(np.asarray(chunks[0])["data"].reshape(-1), values * 2) + + +def test_read_frame_returns_none_at_eof(tmp_path): + fname = tmp_path / "empty.clust" + fname.touch() + + with ClusterFile(fname) as f: + assert f.read_frame() is None + + +def test_read_frame_raises_for_malformed_file(tmp_path): + fname = tmp_path / "malformed.clust" + fname.write_bytes(b"\x00") + + with ClusterFile(fname) as f, pytest.raises(RuntimeError): + f.read_frame() + + +@pytest.mark.parametrize("use_roi", [False, True]) +@pytest.mark.parametrize("method", ["read_clusters", "default", "frames", "chunks"]) +@pytest.mark.parametrize( + "size", + [ + pytest.param(3, id="partial-frame-number"), + pytest.param(4, id="missing-cluster-count"), + pytest.param(7, id="partial-cluster-count"), + pytest.param(48, id="missing-cluster-record"), + pytest.param(87, id="partial-cluster-record"), + ], +) +def test_chunk_reads_reject_incomplete_frames(tmp_path, use_roi, method, size): + fname = tmp_path / "incomplete.clust" + cluster = struct.pack("@HH9i", 5, 6, *range(9)) + frame = struct.pack("@iI", 42, 2) + cluster * 2 + fname.write_bytes(frame[:size]) + + with ClusterFile(fname) as reader: + if use_roi: + reader.set_roi(ROI(0, 10, 0, 10)) + with pytest.raises(RuntimeError): + if method == "default": + next(reader) + elif method == "read_clusters": + reader.read_clusters(10) + else: + next(getattr(reader, method)()) + + @pytest.mark.withdata def test_cluster_file(test_data_path): """Test ClusterFile""" diff --git a/python/tests/test_ClusterVector.py b/python/tests/test_ClusterVector.py index 30f934fa..dd75366e 100644 --- a/python/tests/test_ClusterVector.py +++ b/python/tests/test_ClusterVector.py @@ -83,12 +83,15 @@ def test_make_a_hitmap_from_cluster_vector(): def test_2x2_reduction(): cv = ClusterVector((3,3)) + cv.frame_number = -135 cv.push_back(_aare.Cluster3x3i(5, 5, np.array([1, 1, 1, 2, 3, 1, 2, 2, 1], dtype=np.int32))) cv.push_back(_aare.Cluster3x3i(5, 5, np.array([2, 2, 1, 2, 3, 1, 1, 1, 1], dtype=np.int32))) - reduced_cv = np.array(_aare.reduce_to_2x2(cv), copy=False) + reduced = _aare.reduce_to_2x2(cv) + reduced_cv = np.array(reduced, copy=False) + assert reduced.frame_number == cv.frame_number assert reduced_cv.size == 2 assert reduced_cv[0]["x"] == 5 assert reduced_cv[0]["y"] == 5 @@ -100,14 +103,17 @@ def test_2x2_reduction(): def test_3x3_reduction(): cv = _aare.ClusterVector_Cluster5x5d() + cv.frame_number = 246 cv.push_back(_aare.Cluster5x5d(5,5,np.array([1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 2.0, 2.0, 3.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], dtype=np.double))) cv.push_back(_aare.Cluster5x5d(5,5,np.array([1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 2.0, 2.0, 3.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], dtype=np.double))) - reduced_cv = np.array(_aare.reduce_to_3x3(cv), copy=False) + reduced = _aare.reduce_to_3x3(cv) + reduced_cv = np.array(reduced, copy=False) + assert reduced.frame_number == cv.frame_number assert reduced_cv.size == 2 assert reduced_cv[0]["x"] == 5 assert reduced_cv[0]["y"] == 5 @@ -130,3 +136,26 @@ def test_masking(): assert cv_masked_array[0]["x"] == 1 assert cv_masked_array[0]["y"] == 2 assert (cv_masked_array[0]["data"] == np.ones((3,3),dtype=np.int32)).all() + + +def test_masking_requires_c_contiguous_array(): + cv = _aare.ClusterVector_Cluster3x3i() + cv.push_back(_aare.Cluster3x3i(1, 2, np.ones(9, dtype=np.int32))) + cv.push_back(_aare.Cluster3x3i(3, 4, np.ones(9, dtype=np.int32))) + + mask = np.array([True, False, True, False], dtype=bool)[::2] + assert not mask.flags.c_contiguous + + with pytest.raises(TypeError): + cv(mask) + + +def test_masking_requires_one_dimension(): + cv = _aare.ClusterVector_Cluster3x3i() + cv.push_back(_aare.Cluster3x3i(1, 2, np.ones(9, dtype=np.int32))) + cv.push_back(_aare.Cluster3x3i(3, 4, np.ones(9, dtype=np.int32))) + + mask = np.array([[True, False]], dtype=bool) + + with pytest.raises(ValueError): + cv(mask) diff --git a/python/tests/test_CtbRawFile.py b/python/tests/test_CtbRawFile.py new file mode 100644 index 00000000..6485b076 --- /dev/null +++ b/python/tests/test_CtbRawFile.py @@ -0,0 +1,58 @@ + + +import pytest +import numpy as np +from aare import CtbRawFile + + +@pytest.mark.withdata +def test_basic_properites(test_data_path): + f = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + assert f.total_frames == 47 + assert f.image_size_in_bytes == 640 + assert f.master.analog_samples == 10 + assert f.master.digital_samples is None + assert f.master.transceiver_samples is None + + +@pytest.mark.withdata +def test_seek_and_read(test_data_path): + f = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f.seek(0) + header, raw_data = f.read_frame() + assert header['frameNumber'] == 48 #we know that the first frame in this file is 48 + + #Seek a multiple of frames in file + f.seek(9) + header, raw_data = f.read_frame() + assert header['frameNumber'] == 57 + +@pytest.mark.withdata +def test_read_all_frames_direct_and_iterate(test_data_path): + """ + We want to make sure that iterating and reading give the same result. + """ + f0 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f1 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + + for h0, d0 in f0: + h1, d1 = f1.read_frame() + assert h0 == h1 + assert np.all(d0 == d1) + +@pytest.mark.withdata +def test_read_all_frames_with_index_and_iterate(test_data_path): + """ + We want to make sure that iterating and reading give the same result. + """ + f0 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f1 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + + frames_read = 0 + for i,(h0, d0) in enumerate(f0): + h1, d1 = f1.read_frame(i) + assert h0 == h1 + assert np.all(d0 == d1) + frames_read += 1 + + assert frames_read == f0.total_frames diff --git a/python/tests/test_FastPedestal.py b/python/tests/test_FastPedestal.py index 24c115fc..f93560cb 100644 --- a/python/tests/test_FastPedestal.py +++ b/python/tests/test_FastPedestal.py @@ -143,3 +143,17 @@ def test_fast_pedestal_rejects_wrong_shape(): with pytest.raises(RuntimeError, match="shape"): pedestal.add_init_frame(np.zeros((2, 2), dtype=np.uint16)) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +def test_fast_pedestal_std_uses_double_variance(dtype): + pedestal = FastPedestal(1, 1, n_samples=2, dtype=dtype) + pedestal.add_init_frame(np.array([[0]], dtype=np.uint16)) + pedestal.add_init_frame(np.array([[1000]], dtype=np.uint16)) + + assert pedestal.std().dtype == dtype + np.testing.assert_array_equal(pedestal.std(), [[500]]) + + pedestal.push_ema(np.array([[1000]], dtype=np.uint16)) + expected = np.array([[np.sqrt(187500.0)]], dtype=dtype) + np.testing.assert_array_equal(pedestal.std(), expected) diff --git a/python/tests/test_Pedestal.py b/python/tests/test_Pedestal.py index 2c94811e..27f9897a 100644 --- a/python/tests/test_Pedestal.py +++ b/python/tests/test_Pedestal.py @@ -1,7 +1,106 @@ import numpy as np import pytest -from aare import Pedestal_d, Pedestal_f +from aare import Pedestal, Pedestal_d, Pedestal_f, Pedestal_i16 + + +@pytest.mark.parametrize( + ("dtype", "pedestal_type"), + [(np.float64, Pedestal_d), (np.float32, Pedestal_f), (np.int16, Pedestal_i16)], +) +def test_pedestal_factory(dtype, pedestal_type): + pedestal = Pedestal(rows=2, cols=3, n_samples=4, dtype=dtype) + + assert isinstance(pedestal, pedestal_type) + assert pedestal.rows == 2 + assert pedestal.cols == 3 + assert pedestal.n_samples == 4 + + +def test_pedestal_factory_defaults_to_double(): + pedestal = Pedestal(2, 3) + + assert isinstance(pedestal, Pedestal_d) + assert pedestal.n_samples == 1000 + + +def test_pedestal_factory_rejects_unbound_dtype(): + with pytest.raises(ValueError, match="Unsupported dtype for Pedestal"): + Pedestal(2, 3, dtype=np.int32) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("name", ["rows", "cols", "n_samples"]) +def test_pedestal_factory_rejects_negative_parameters(dtype, name): + parameters = dict(rows=1, cols=1, n_samples=10) + parameters[name] = -1 + with pytest.raises(TypeError): + Pedestal(**parameters, dtype=dtype) + + +@pytest.mark.parametrize("pedestal_type", [Pedestal_d, Pedestal_f, Pedestal_i16]) +@pytest.mark.parametrize( + "args", [(-1, 1), (1, -1), (-1, 1, 10), (1, -1, 10), (1, 1, -1)] +) +def test_pedestal_constructor_rejects_negative_parameters(pedestal_type, args): + with pytest.raises(TypeError): + pedestal_type(*args) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +def test_pedestal_std_stays_finite_after_settling(dtype): + pedestal = Pedestal(1, 1, n_samples=10, dtype=dtype) + pedestal.push(np.array([[16382]], dtype=np.uint16)) + frame = np.array([[16383]], dtype=np.uint16) + for _ in range(201): + pedestal.push(frame) + + noise = pedestal.std().item() + assert np.isfinite(noise) + assert 0 <= noise < 1e-3 + + +@pytest.mark.parametrize( + ("pedestal_type", "expected_dtype"), + [(Pedestal_d, np.float64), (Pedestal_f, np.float32), (Pedestal_i16, np.int16)], +) +def test_double_precision_moments(pedestal_type, expected_dtype): + pedestal = pedestal_type(1, 1, 2) + for value in [30000, 30003, 30002]: + pedestal.push(np.array([[value]], dtype=np.uint16)) + + assert pedestal.mean().dtype == expected_dtype + assert pedestal.std().dtype == expected_dtype + np.testing.assert_array_equal( + pedestal.mean(), np.array([[30001.75]], dtype=expected_dtype) + ) + np.testing.assert_allclose( + pedestal.std(), + np.array([[np.sqrt(1.1875)]], dtype=expected_dtype), + rtol=1e-6, + ) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("method", ["push", "push_with_threshold"]) +@pytest.mark.parametrize("shape", [(2, 2), (3, 2)]) +def test_pedestal_rejects_mismatched_frame_shapes(dtype, method, shape): + pedestal = Pedestal(2, 3, dtype=dtype) + initial = np.full((2, 3), 7, dtype=np.uint16) + pedestal.push(initial) + frame = np.zeros(shape, dtype=np.uint16) + args = ( + (np.full((2, 3), 10, dtype=dtype),) + if method == "push_with_threshold" + else () + ) + + with pytest.raises( + RuntimeError, match="Frame shape does not match pedestal shape" + ): + getattr(pedestal, method)(frame, *args) + + np.testing.assert_array_equal(pedestal.mean(), initial) @pytest.mark.parametrize( @@ -38,3 +137,68 @@ def test_pedestal_exposes_mean_as_read_only_buffer(): np.testing.assert_array_equal(mean, pedestal.view()) assert np.shares_memory(mean, pedestal.view()) assert not mean.flags.writeable + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("method", ["push", "push_with_threshold"]) +def test_pedestal_push_accepts_contiguous_frames(dtype, method): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.arange(6, dtype=np.uint16).reshape(2, 3) + args = ( + (np.full((2, 3), 10, dtype=dtype),) + if method == "push_with_threshold" + else () + ) + + getattr(pedestal, method)(frame, *args) + + np.testing.assert_array_equal(pedestal.mean(), frame) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("input_name", ["push_frame", "threshold_frame", "threshold"]) +@pytest.mark.parametrize("layout", ["transpose", "slice", "reverse"]) +def test_pedestal_rejects_noncontiguous_inputs(dtype, input_name, layout): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.zeros((2, 3), dtype=np.uint16) + threshold = np.full((2, 3), 10, dtype=dtype) + input_dtype = dtype if input_name == "threshold" else np.uint16 + if layout == "transpose": + invalid = np.ones((3, 2), dtype=input_dtype).T + elif layout == "slice": + invalid = np.ones((2, 6), dtype=input_dtype)[:, ::2] + else: + invalid = np.ones((4, 6), dtype=input_dtype)[:2, :3][:, ::-1] + assert not invalid.flags.c_contiguous + + with pytest.raises(TypeError): + if input_name == "push_frame": + pedestal.push(invalid) + elif input_name == "threshold_frame": + pedestal.push_with_threshold(invalid, threshold) + else: + pedestal.push_with_threshold(frame, invalid) + + np.testing.assert_array_equal(pedestal.mean(), np.zeros((2, 3))) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("input_name", ["push_frame", "threshold_frame", "threshold"]) +@pytest.mark.parametrize("shape", [(), (6,), (2, 3, 2)]) +def test_pedestal_rejects_inputs_with_wrong_ndim(dtype, input_name, shape): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.ones((2, 3), dtype=np.uint16) + threshold = np.full((2, 3), 10, dtype=dtype) + input_dtype = dtype if input_name == "threshold" else np.uint16 + invalid = np.ones(shape, dtype=input_dtype) + name = "Threshold" if input_name == "threshold" else "Frame" + + with pytest.raises(ValueError, match=f"{name} must be 2-dimensional"): + if input_name == "push_frame": + pedestal.push(invalid) + elif input_name == "threshold_frame": + pedestal.push_with_threshold(invalid, threshold) + else: + pedestal.push_with_threshold(frame, invalid) + + np.testing.assert_array_equal(pedestal.mean(), np.zeros((2, 3))) diff --git a/python/tests/test_ROI.py b/python/tests/test_ROI.py new file mode 100644 index 00000000..f0b98de0 --- /dev/null +++ b/python/tests/test_ROI.py @@ -0,0 +1,21 @@ +import pytest + +import numpy as np + +from aare import ROI + + +def test_roi_slice(): + """test the ROI slice method""" + roi = ROI(0, 4, 2, 4) + + array = np.array([[1, 2, 3, 4], + [5, 6, 7, 8], + [9, 10, 11, 12], + [13, 14, 15, 16]]) + + sliced_array = array[roi.slice()] + + assert sliced_array.shape == (2, 4) + assert np.array_equal(sliced_array, np.array([[9, 10, 11, 12], + [13, 14, 15, 16]])) \ No newline at end of file diff --git a/python/tests/test_RawFile.py b/python/tests/test_RawFile.py index ca844a0e..1fb6c3f7 100644 --- a/python/tests/test_RawFile.py +++ b/python/tests/test_RawFile.py @@ -1,8 +1,299 @@ # SPDX-License-Identifier: MPL-2.0 import pytest -from aare import RawFile +import json +from aare import File, RawFile, RawSubFile, DetectorType, ROI, UDPPortPosition import numpy as np + +@pytest.fixture +def small_raw_file(tmp_path): + master_path = tmp_path / "run_master_0.json" + master_path.write_text(json.dumps({ + "Version": 7.2, + "Detector Type": "Jungfrau", + "Timing Mode": "auto", + "Geometry": {"x": 1, "y": 1}, + "Image Size in bytes": 12, + "Pixels": {"x": 3, "y": 2}, + "Max Frames Per File": 1, + "Total Frames": 2, + "Frames in File": 2, + "Frame Padding": 1, + "Frame Discard Policy": "nodiscard", + })) + for index in range(2): + data_path = tmp_path / f"run_d0_f{index}_0.raw" + # A detector header followed by six uint16 pixels. + data_path.write_bytes(bytes(112) + np.arange(6, dtype=np.uint16).tobytes()) + return master_path + + +@pytest.mark.parametrize("reader_type", [RawFile, File]) +@pytest.mark.parametrize("legacy_master", [False, True]) +@pytest.mark.parametrize("padding, policy, supported", [ + (0, "nodiscard", False), + (0, "discard", False), + (0, "discardpartial", True), + (1, "nodiscard", True), + (1, "discard", True), + (1, "discardpartial", True), +]) +def test_raw_frame_policy(small_raw_file, reader_type, legacy_master, + padding, policy, supported): + master_path = small_raw_file + if legacy_master: + master_path = master_path.with_suffix(".raw") + master_path.write_text( + "Version : 6.4\n" + "Detector Type : Jungfrau\n" + "Timing Mode : auto\n" + "Geometry : [1, 1]\n" + "Image Size : 12\n" + "Pixels : [3, 2]\n" + "Dynamic Range : 16\n" + "Max Frames Per File : 1\n" + "Total Frames : 2\n" + "Frames in File : 2\n" + f"Frame Padding : {padding}\n" + f"Frame Discard Policy : {policy}\n" + ) + else: + metadata = json.loads(master_path.read_text()) + metadata["Frame Padding"] = padding + metadata["Frame Discard Policy"] = policy + master_path.write_text(json.dumps(metadata)) + + if supported: + reader = reader_type(master_path) + assert reader.total_frames == 2 + frame = reader.read_frame() + if reader_type is RawFile: + _, frame = frame + np.testing.assert_array_equal(frame, np.arange(6).reshape(2, 3)) + else: + message = "requires frame padding or discardpartial" + with pytest.raises(RuntimeError, match=message) as error: + reader_type(master_path) + assert str(master_path) in str(error.value) + + for index in range(2): + (master_path.parent / f"run_d0_f{index}_0.raw").unlink() + with pytest.raises(RuntimeError, match=message) as error: + reader_type(master_path) + assert str(master_path) in str(error.value) + + +@pytest.mark.parametrize("method, args, kwargs", [ + ("read_frame", (), {}), + ("read_n", (1,), {}), + ("read_roi", (0,), {}), + ("read_rois", (), {}), + ("read_n_with_roi", (1,), {"roi_index": 0}), + ("frame_number", (2,), {}), +]) +def test_raw_read_error_context(small_raw_file, method, args, kwargs): + reader = RawFile(small_raw_file) + reader.seek(2) + with pytest.raises(RuntimeError) as error: + getattr(reader, method)(*args, **kwargs) + assert "frame index 2" in str(error.value) + assert str(small_raw_file) in str(error.value) + assert ".raw" not in str(error.value) + if method == "read_frame": + assert "file contains 2 frames (indices are zero-based)" in str(error.value) + + +@pytest.mark.parametrize("frame_index", [2, 17]) +def test_generic_raw_read_error_context(small_raw_file, frame_index): + reader = File(small_raw_file) + with pytest.raises(RuntimeError) as error: + reader.read_frame(frame_index) + assert f"frame index {frame_index}" in str(error.value) + assert str(small_raw_file) in str(error.value) + + +@pytest.mark.parametrize("raw_subfile", [False, True]) +def test_truncated_raw_read_error_context(small_raw_file, raw_subfile): + data_path = small_raw_file.parent / "run_d0_f1_0.raw" + if raw_subfile: + reader = RawSubFile( + small_raw_file.parent / "run_d0_f0_0.raw", + DetectorType.Jungfrau, 2, 3, 16, + ) + else: + reader = RawFile(small_raw_file) + data_path.write_bytes(bytes(112)) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + reader.read_frame() + assert "frame index 1" in str(error.value) + assert str(data_path) in str(error.value) + assert "End of file" in str(error.value) + assert "master" not in str(error.value) + assert "RawSubFile.cpp" not in str(error.value) + + +@pytest.mark.parametrize("method, args", [ + ("read_frame", ()), + ("read_n", (1,)), + ("read_roi", (0,)), + ("frame_number", (1,)), +]) +def test_short_raw_subfile_sets_read_bounds(small_raw_file, method, args): + (small_raw_file.parent / "run_d0_f1_0.raw").unlink() + reader = RawFile(small_raw_file) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + getattr(reader, method)(*args) + assert reader.total_frames == 1 + assert "frame index 1" in str(error.value) + assert str(small_raw_file) in str(error.value) + assert ".raw" not in str(error.value) + + +@pytest.mark.parametrize("counts, master_count, warn", [ + ([2], 2, False), + ([2, 2], 2, False), + ([2, 2], 0, True), + ([2, 2], 1, True), + ([2, 2], 5, True), + ([1, 2], 2, True), + ([2, 1], 2, True), + ([1, 2], 1, True), + ([0, 2], 2, True), + ([0, 0], 0, False), +]) +def test_raw_subfile_frame_counts(small_raw_file, capfd, counts, master_count, warn): + metadata = json.loads(small_raw_file.read_text()) + metadata["Geometry"]["y"] = len(counts) + metadata["Frames in File"] = master_count + metadata["Total Frames"] = 1000 + small_raw_file.write_text(json.dumps(metadata)) + data = (small_raw_file.parent / "run_d0_f0_0.raw").read_bytes() + for module, count in enumerate(counts): + for index in range(2): + path = small_raw_file.parent / f"run_d{module}_f{index}_0.raw" + if index < count: + path.write_bytes(data) + elif index == 0: + path.write_bytes(b"") + else: + path.unlink(missing_ok=True) + + reader = RawFile(small_raw_file) + warning = capfd.readouterr().err + if warn: + assert warning.count("WARNING") == 1 + assert str(small_raw_file) in warning + else: + assert "WARNING" not in warning + + assert reader.total_frames == min(counts) + assert len(reader) == min(counts) + assert reader.master.frames_in_file == master_count + if min(counts): + _, frames = reader.read() + assert frames.shape == (min(counts), 2 * len(counts), 3) + reader.seek(0) + _, frames = reader.read_n(10) + assert len(frames) == min(counts) + assert reader.frame_number(min(counts) - 1) == 0 + with pytest.raises(RuntimeError): + reader.frame_number(min(counts)) + reader.seek(min(counts)) + with pytest.raises(RuntimeError): + reader.read_frame() + assert "WARNING" not in capfd.readouterr().err + + +@pytest.mark.parametrize("short_module", [0, 1]) +def test_raw_frame_count_across_rois(small_raw_file, short_module): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({ + "Version": 8.0, + "Geometry": {"x": 1, "y": 2}, + "Image Size": 12, + "Receiver Rois": [ + {"xmin": 0, "xmax": 2, "ymin": 0, "ymax": 1}, + {"xmin": 0, "xmax": 2, "ymin": 2, "ymax": 3}, + ], + }) + small_raw_file.write_text(json.dumps(metadata)) + for index in range(2): + data = (small_raw_file.parent / f"run_d0_f{index}_0.raw").read_bytes() + (small_raw_file.parent / f"run_d1_f{index}_0.raw").write_bytes(data) + (small_raw_file.parent / f"run_d{short_module}_f1_0.raw").unlink() + + reader = RawFile(small_raw_file) + assert reader.total_frames == 1 + _, rois = reader.read_rois() + assert len(rois) == 2 + assert all(roi.shape == (2, 3) for roi in rois) + for roi_index in range(2): + reader.seek(0) + _, frames = reader.read_n_with_roi(10, roi_index=roi_index) + assert frames.shape == (1, 2, 3) + with pytest.raises(RuntimeError): + reader.read_roi(roi_index) + + +@pytest.mark.parametrize("disabled_port", [0, 1]) +def test_raw_frame_count_with_disabled_udp_port(small_raw_file, disabled_port): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({ + "Geometry": {"x": 1, "y": 2}, + "Frames in File": 0, + "Number of UDP Interfaces": 2, + "UDP Ports Type": ["bottom", "top"], + "UDP Ports Disabled": [disabled_port], + }) + small_raw_file.write_text(json.dumps(metadata)) + data = (small_raw_file.parent / "run_d0_f0_0.raw").read_bytes() + for index in range(2): + (small_raw_file.parent / f"run_d0_f{index}_0.raw").unlink() + path = small_raw_file.parent / f"run_d{1 - disabled_port}_f{index}_0.raw" + path.write_bytes(data) + + reader = RawFile(small_raw_file) + assert reader.total_frames == 2 + assert reader.master.frames_in_file == 0 + assert reader.frame_number(1) == 0 + _, frames = reader.read() + assert frames.shape == (2, 2, 3) + + +def test_raw_frame_count_without_selected_subfiles(small_raw_file): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({"Version": 8.0, "Image Size": 12, "Receiver Rois": []}) + small_raw_file.write_text(json.dumps(metadata)) + with pytest.raises(RuntimeError, match="No raw subfiles selected"): + RawFile(small_raw_file) + + +@pytest.mark.parametrize("lagging_module", [0, 1]) +def test_raw_synchronization_error_identifies_subfile( + small_raw_file, lagging_module +): + metadata = json.loads(small_raw_file.read_text()) + metadata["Geometry"]["y"] = 2 + small_raw_file.write_text(json.dumps(metadata)) + for index in range(2): + data = (small_raw_file.parent / f"run_d0_f{index}_0.raw").read_bytes() + (small_raw_file.parent / f"run_d1_f{index}_0.raw").write_bytes(data) + + ahead_path = small_raw_file.parent / f"run_d{1 - lagging_module}_f1_0.raw" + with ahead_path.open("r+b") as output: + output.write(np.array([1], dtype=np.uint64).tobytes()) + + reader = RawFile(small_raw_file) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + reader.read_frame() + assert "frame index 1" in str(error.value) + assert "synchroniz" in str(error.value) + lagging_path = small_raw_file.parent / f"run_d{lagging_module}_f1_0.raw" + assert str(lagging_path) in str(error.value) + @pytest.mark.withdata def test_read_rawfile_with_roi_spanning_over_one_module(test_data_path): @@ -110,3 +401,41 @@ def test_read_rawfile_eiger_and_compare_to_numpy(test_data_path): header, image1 = f.read_frame() assert (image == image1).all() + + +@pytest.mark.withdata +def test_read_eiger_udp_port_disabled(test_data_path): + with RawFile(test_data_path / "raw/eiger/one_udp_port_disabled_master_0.json") as f: + _, frame = f.read_rois() + + assert(len(frame) == 2) + assert frame[0].shape == (256, 512) + assert frame[1].shape == (256, 1024) + assert f.master.udp_port_types == [UDPPortPosition.LEFT, UDPPortPosition.RIGHT] + rois = f.master.rois + assert len(rois) == 2 + assert rois[0] == ROI(512, 1024, 0, 256) + assert rois[1] == ROI(0, 1024, 256, 512) + + with RawFile(test_data_path / "raw/eiger/quad_eiger_disabled_bottom_port_master_0.json") as f: + _, frame = f.read_frame() + + assert frame.shape == (256, 512) + assert(f.master.disabled_udp_ports == [1]) + assert f.master.udp_port_types == [UDPPortPosition.TOP, UDPPortPosition.BOTTOM] + rois = f.master.rois + assert len(rois) == 1 + assert rois[0] == ROI(0, 512, 256, 512) + + with RawFile(test_data_path / "raw/eiger/2_modules_eiger_disabled_udp_port_master_0.json") as f: + _, frame = f.read_rois() + + assert(len(frame) == 2) + assert frame[0].shape == (512, 512) + assert frame[1].shape == (512, 512) + assert (f.master.disabled_udp_ports == [1, 3, 5, 7]) + assert f.master.udp_port_types == [UDPPortPosition.LEFT, UDPPortPosition.RIGHT] + rois = f.master.rois + assert len(rois) == 2 + assert rois[0] == ROI(0, 512, 0, 512) + assert rois[1] == ROI(1024, 1536, 0, 512) diff --git a/python/tests/test_RawMasterFile.py b/python/tests/test_RawMasterFile.py index 6ea9ff74..5ad68b48 100644 --- a/python/tests/test_RawMasterFile.py +++ b/python/tests/test_RawMasterFile.py @@ -1,7 +1,9 @@ +import json + import pytest -from aare import RawMasterFile, ReadoutMode, DetectorType +from aare import RawMasterFile, ReadoutMode, DetectorType, FrameDiscardPolicy @pytest.mark.withdata @@ -11,4 +13,37 @@ def test_read_rawfile_quad_eiger_and_compare_to_numpy(test_data_path): f = RawMasterFile(file_name) assert(f.reading_mode == ReadoutMode.UNKNOWN) - assert(f.detector_type == DetectorType.Jungfrau) \ No newline at end of file + assert(f.detector_type == DetectorType.Jungfrau) + + +@pytest.mark.parametrize( + "policy, expected_policy", + [ + ("nodiscard", FrameDiscardPolicy.NoDiscard), + ("discard", FrameDiscardPolicy.Discard), + ("discardpartial", FrameDiscardPolicy.DiscardPartial), + ], +) +def test_raw_master_file_context_manager(tmp_path, policy, expected_policy): + file_name = tmp_path / "run_master_0.json" + file_name.write_text(json.dumps({ + "Version": 7.2, + "Detector Type": "Jungfrau", + "Timing Mode": "auto", + "Geometry": {"x": 1, "y": 1}, + "Image Size in bytes": 12, + "Pixels": {"x": 3, "y": 2}, + "Max Frames Per File": 1, + "Total Frames": 2, + "Frames in File": 2, + "Frame Padding": 1, + "Frame Discard Policy": policy, + })) + + + with RawMasterFile(file_name) as context_file: + assert context_file.reading_mode == ReadoutMode.UNKNOWN + assert context_file.detector_type == DetectorType.Jungfrau + frame_discard_policy = context_file.frame_discard_policy + assert isinstance(frame_discard_policy, FrameDiscardPolicy) + assert frame_discard_policy == expected_policy diff --git a/python/tests/test_make_view.py b/python/tests/test_make_view.py new file mode 100644 index 00000000..2df947ca --- /dev/null +++ b/python/tests/test_make_view.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: MPL-2.0 +"""Probe make_view in np_helper.hpp through the bindings in _aare.testing""" +import numpy as np +import pytest + +from aare._aare import testing + + + + +def test_make_view_roundtrip(): + """Send an array to C++ and check that we get the same array back""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + out = testing.view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + + #Sanity check to make sure we got a copy an not just the same thing back + arr[3,2] = 31655 + assert arr[3,2] != out[3,2] + +def test_make_view_const_roundtrip(): + """Const works if the array is not writeable""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr.flags.writeable = False + out = testing.const_view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + +def test_make_view_const_roundtrip_writable(): + """Const works also if the array is writeable""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + out = testing.const_view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + +def test_make_view_fill(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + testing.view_fill_2d(arr, 7) + assert (arr == 7).all() + +def test_make_view_fill_rejects_nonwriteable(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr.flags.writeable = False + with pytest.raises(ValueError, match="not writeable"): + testing.view_fill_2d(arr, 7) + +def test_make_view_2d_rejects_1d(): + arr = np.arange(30, dtype=np.uint16) + with pytest.raises(ValueError, match="Expected 2D array, got 1D"): + testing.view_roundtrip_2d(arr) + +def test_make_view_2d_rejects_3d(): + arr = np.arange(30, dtype=np.uint16).reshape(3, 5, 2) + with pytest.raises(ValueError, match="Expected 2D array, got 3D"): + testing.view_roundtrip_2d(arr) + +def test_make_view_rejects_noncontiguous(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr = arr[::2, ::3] # non-contiguous + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.view_roundtrip_2d(arr) + +def test_make_view_rejects_noncontiguous_const(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr = arr[::2, ::3] # non-contiguous + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.const_view_roundtrip_2d(arr) + +def test_make_view_rejects_fortran_order(): + arr = np.asfortranarray(np.arange(30, dtype=np.float64).reshape(5, 6)) + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.view_roundtrip_2d(arr) + diff --git a/src/Cluster.test.cpp b/src/Cluster.test.cpp index 4919b859..ddcd2c4b 100644 --- a/src/Cluster.test.cpp +++ b/src/Cluster.test.cpp @@ -24,8 +24,8 @@ TEST_CASE("Test sum of Cluster", "[cluster]") { using ClusterTypes = std::variant, Cluster, Cluster, Cluster>; -using ClusterTypesLargerThan2x2 = - std::variant, Cluster, Cluster>; +using ReducibleTo3x3ClusterTypes = + std::variant, Cluster, Cluster>; TEST_CASE("Test reduce to 2x2 Cluster", "[cluster]") { auto [cluster, expected_reduced_cluster] = GENERATE( @@ -68,14 +68,15 @@ TEST_CASE("Test reduce to 2x2 Cluster", "[cluster]") { TEST_CASE("Test reduce to 3x3 Cluster", "[cluster]") { auto [cluster, expected_reduced_cluster] = GENERATE( - std::make_tuple(ClusterTypesLargerThan2x2{Cluster{ - 5, 5, {1, 1, 1, 1, 3, 1, 1, 1, 1}}}, - Cluster{5, 5, {1, 1, 1, 1, 3, 1, 1, 1, 1}}), std::make_tuple( - ClusterTypesLargerThan2x2{Cluster{ - 5, 5, {2, 2, 1, 1, 2, 2, 1, 1, 1, 1, 3, 1, 1, 1, 1, 1}}}, - Cluster{5, 5, {2, 1, 1, 1, 3, 1, 1, 1, 1}}), - std::make_tuple(ClusterTypesLargerThan2x2{Cluster{ + ReducibleTo3x3ClusterTypes{Cluster{ + 5, 5, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}}}, + Cluster{5, 5, {3, 4, 5, 6, 7, 8, 9, 10, 11}}), + std::make_tuple( + ReducibleTo3x3ClusterTypes{Cluster{ + 5, 5, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}}}, + Cluster{5, 5, {1, 2, 3, 5, 6, 7, 9, 10, 11}}), + std::make_tuple(ReducibleTo3x3ClusterTypes{Cluster{ 5, 5, {1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 3, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1}}}, Cluster{5, 5, {2, 1, 1, 2, 3, 1, 2, 1, 1}})); @@ -89,4 +90,4 @@ TEST_CASE("Test reduce to 3x3 Cluster", "[cluster]") { CHECK(std::equal(reduced_cluster.data.begin(), reduced_cluster.data.begin() + 9, expected_reduced_cluster.data.begin())); -} \ No newline at end of file +} diff --git a/src/ClusterFile.test.cpp b/src/ClusterFile.test.cpp index 15caf6bb..6c009ba1 100644 --- a/src/ClusterFile.test.cpp +++ b/src/ClusterFile.test.cpp @@ -5,12 +5,74 @@ #include "aare/defs.hpp" #include #include +#include +#include +#include #include +#include +#include +#include using aare::Cluster; using aare::ClusterFile; using aare::ClusterVector; +namespace { + +class TemporaryClusterFile { + public: + TemporaryClusterFile() { + const auto unique = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_path = std::filesystem::temp_directory_path() / + ("aare-cluster-file-" + std::to_string(unique) + ".clust"); + } + + TemporaryClusterFile(const TemporaryClusterFile &) = delete; + TemporaryClusterFile &operator=(const TemporaryClusterFile &) = delete; + + ~TemporaryClusterFile() { + std::error_code error; + std::filesystem::remove(m_path, error); + } + + const std::filesystem::path &path() const { return m_path; } + + private: + std::filesystem::path m_path; +}; + +using TestCluster = Cluster; + +ClusterVector make_test_frame(int32_t frame_number, + double offset) { + ClusterVector clusters(2, frame_number); + clusters.push_back( + TestCluster{5, + 6, + {offset, offset + 1, offset + 2, offset + 3, offset + 4, + offset + 5, offset + 6, offset + 7, offset + 8}}); + clusters.push_back(TestCluster{7, + 8, + {offset + 9, offset + 10, offset + 11, + offset + 12, offset + 13, offset + 14, + offset + 15, offset + 16, offset + 17}}); + return clusters; +} + +void check_frame(const ClusterVector &actual, + const ClusterVector &expected) { + REQUIRE(actual.frame_number() == expected.frame_number()); + REQUIRE(actual.size() == expected.size()); + for (size_t i = 0; i < expected.size(); ++i) { + CHECK(actual[i].x == expected[i].x); + CHECK(actual[i].y == expected[i].y); + CHECK(actual[i].data == expected[i].data); + } +} + +} // namespace + TEST_CASE("Read one frame from a cluster file", "[.with-data]") { // We know that the frame has 97 clusters auto fpath = test_data_path() / "clust" / "single_frame_97_clustrers.clust"; @@ -20,12 +82,14 @@ TEST_CASE("Read one frame from a cluster file", "[.with-data]") { CHECK(f.estimate_n_clusters() == 97); CHECK(f.tell() == 0); auto clusters = f.read_frame(); - CHECK(clusters.size() == 97); - CHECK(clusters.frame_number() == 135); - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + REQUIRE(clusters); + CHECK(clusters->size() == 97); + CHECK(clusters->frame_number() == 135); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } @@ -42,22 +106,24 @@ TEST_CASE("Read one frame using ROI", "[.with-data]") { roi.ymax = 249; f.set_roi(roi); auto clusters = f.read_frame(); - REQUIRE(clusters.size() == 49); - REQUIRE(clusters.frame_number() == 135); + REQUIRE(clusters); + REQUIRE(clusters->size() == 49); + REQUIRE(clusters->frame_number() == 135); // Check that all clusters are within the ROI - for (size_t i = 0; i < clusters.size(); i++) { - auto c = clusters[i]; + for (size_t i = 0; i < clusters->size(); i++) { + auto c = (*clusters)[i]; REQUIRE(c.x >= roi.xmin); REQUIRE(c.x <= roi.xmax); REQUIRE(c.y >= roi.ymin); REQUIRE(c.y <= roi.ymax); } - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } @@ -284,70 +350,551 @@ TEST_CASE("Read cluster from multiple frame file", "[.with-data]") { } } -TEST_CASE("Write cluster with potential padding", - "[.with-data][.ClusterFile]") { +TEST_CASE("ClusterFile flushes a frame when the writer is destroyed", + "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); - using ClusterType = Cluster; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } - REQUIRE(std::filesystem::exists(test_data_path() / "clust")); + const auto expected_file_size = sizeof(int32_t) + sizeof(uint32_t) + + expected.size() * sizeof(TestCluster); + CHECK(std::filesystem::file_size(file.path()) == expected_file_size); - auto fpath = test_data_path() / "clust" / "single_frame_2_clusters.clust"; - - ClusterFile file(fpath, 1000, "w"); - - ClusterVector clustervec(2); - uint16_t coordinate = 5; - clustervec.push_back(ClusterType{ - coordinate, coordinate, {0., 0., 0., 0., 0., 0., 0., 0., 0.}}); - clustervec.push_back(ClusterType{ - coordinate, coordinate, {0., 0., 0., 0., 0., 0., 0., 0., 0.}}); - - file.write_frame(clustervec); - - file.close(); - - file.open("r"); - - auto read_cluster_vector = file.read_frame(); - - CHECK(read_cluster_vector.size() == 2); - CHECK(read_cluster_vector.frame_number() == 0); - - CHECK(read_cluster_vector[0].x == clustervec[0].x); - CHECK(read_cluster_vector[0].y == clustervec[0].y); - CHECK(std::equal( - clustervec[0].data.begin(), clustervec[0].data.end(), - read_cluster_vector[0].data.begin(), [](double a, double b) { - return std::abs(a - b) < std::numeric_limits::epsilon(); - })); - - CHECK(read_cluster_vector[1].x == clustervec[1].x); - CHECK(read_cluster_vector[1].y == clustervec[1].y); - CHECK(std::equal( - clustervec[1].data.begin(), clustervec[1].data.end(), - read_cluster_vector[1].data.begin(), [](double a, double b) { - return std::abs(a - b) < std::numeric_limits::epsilon(); - })); + ClusterFile reader(file.path()); + auto actual = reader.read_frame(); + REQUIRE(actual); + check_frame(*actual, expected); } -TEST_CASE("Read frame and modify cluster data", "[.with-data][.ClusterFile]") { +TEST_CASE("ClusterFile appends frames", "[ClusterFile]") { + TemporaryClusterFile file; + auto first_expected = make_test_frame(42, 0.0); + auto second_expected = make_test_frame(43, 100.0); + + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first_expected); + } + { + ClusterFile writer(file.path(), 1000, "a"); + writer.write_frame(second_expected); + } + + ClusterFile reader(file.path()); + auto first_actual = reader.read_frame(); + auto second_actual = reader.read_frame(); + REQUIRE(first_actual); + REQUIRE(second_actual); + check_frame(*first_actual, first_expected); + check_frame(*second_actual, second_expected); +} + +TEST_CASE("ClusterFile reads frames into an existing cluster vector", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first_expected = make_test_frame(42, 0.0); + auto second_expected = make_test_frame(43, 100.0); + + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first_expected); + writer.write_frame(second_expected); + } + + ClusterFile reader(file.path()); + ClusterVector actual(2); + const auto initial_data = actual.data(); + + REQUIRE(reader.read_frame(actual)); + CHECK(actual.data() == initial_data); + check_frame(actual, first_expected); + + REQUIRE(reader.read_frame(actual)); + CHECK(actual.data() == initial_data); + check_frame(actual, second_expected); + + CHECK_FALSE(reader.read_frame(actual)); + check_frame(actual, second_expected); +} + +TEST_CASE("ClusterFile reports a clean end of file", "[ClusterFile]") { + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + } + + ClusterFile reader(file.path()); + ClusterVector clusters; + CHECK_FALSE(reader.read_frame(clusters)); + CHECK(clusters.empty()); + CHECK_FALSE(reader.read_frame().has_value()); +} + +TEST_CASE("ClusterFile reports complete frames with no output clusters", + "[ClusterFile]") { + TemporaryClusterFile file; + ClusterVector empty_frame(0, 41); + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(empty_frame); + writer.write_frame(expected); + } + + ClusterFile reader(file.path()); + ClusterVector clusters; + auto empty_actual = reader.read_frame(); + REQUIRE(empty_actual); + CHECK(empty_actual->empty()); + CHECK(empty_actual->frame_number() == 41); + + aare::ROI roi; + roi.xmin = 100; + roi.xmax = 200; + roi.ymin = 100; + roi.ymax = 200; + reader.set_roi(roi); + + REQUIRE(reader.read_frame(clusters)); + CHECK(clusters.empty()); + CHECK(clusters.frame_number() == expected.frame_number()); + CHECK_FALSE(reader.read_frame(clusters)); +} + +TEST_CASE("ClusterFile rejects incomplete frames", "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + + SECTION("incomplete frame number") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) - 1); + } + SECTION("incomplete cluster count") { + std::filesystem::resize_file(file.path(), + sizeof(int32_t) + sizeof(uint32_t) - 1); + } + SECTION("incomplete cluster data") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + } + + { + ClusterFile reader(file.path()); + ClusterVector clusters; + CHECK_THROWS_AS(reader.read_frame(clusters), std::runtime_error); + } + { + ClusterFile reader(file.path()); + CHECK_THROWS_AS(reader.read_frame(), std::runtime_error); + } +} + +TEST_CASE("ClusterFile close is idempotent", "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + writer.close(); + + CHECK_NOTHROW(writer.close()); + CHECK_THROWS_AS(writer.tell(), std::runtime_error); + CHECK_THROWS_AS(writer.write_frame(expected), std::runtime_error); + + ClusterFile reader(file.path()); + auto actual = reader.read_frame(); + REQUIRE(actual); + check_frame(*actual, expected); +} + +TEST_CASE("ClusterFile chunk reads reject incomplete frames", "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + CAPTURE(use_roi); + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + + SECTION("incomplete frame number") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) - 1); + } + SECTION("missing cluster count") { + std::filesystem::resize_file(file.path(), sizeof(int32_t)); + } + SECTION("incomplete cluster count") { + std::filesystem::resize_file(file.path(), + sizeof(int32_t) + sizeof(uint32_t) - 1); + } + SECTION("missing cluster record") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) + + sizeof(uint32_t) + + sizeof(TestCluster)); + } + SECTION("incomplete cluster record") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + } + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + CHECK_THROWS_AS(reader.read_clusters(10), std::runtime_error); +} + +TEST_CASE("ClusterFile chunk reads reject truncated leftover clusters", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + const auto requested = GENERATE(1, 10); + CAPTURE(use_roi, requested); + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + std::filesystem::resize_file(file.path(), + std::filesystem::file_size(file.path()) - 1); + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + auto first = reader.read_clusters(1); + REQUIRE(first.size() == 1); + CHECK(first[0].data == expected[0].data); + CHECK_THROWS_AS(reader.read_clusters(requested), std::runtime_error); +} + +TEST_CASE("ClusterFile chunk reads preserve valid frame traversal", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + CAPTURE(use_roi); + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + ClusterVector empty(0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(empty); + writer.write_frame(first); + writer.write_frame(empty); + writer.write_frame(second); + writer.write_frame(empty); + } + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.tell() == 0); + + auto initial = reader.read_clusters(1); + REQUIRE(initial.size() == 1); + CHECK(initial[0].data == first[0].data); + const auto position = reader.tell(); + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.tell() == position); + CHECK_THROWS_AS(reader.read_frame(), std::runtime_error); + + auto crossing = reader.read_clusters(2); + REQUIRE(crossing.size() == 2); + CHECK(crossing[0].data == first[1].data); + CHECK(crossing[1].data == second[0].data); + + auto final = reader.read_clusters(10); + REQUIRE(final.size() == 1); + CHECK(final[0].data == second[1].data); + CHECK(static_cast(reader.tell()) == + std::filesystem::file_size(file.path())); + CHECK(reader.read_clusters(10).empty()); + CHECK_FALSE(reader.read_frame()); +} + +TEST_CASE("ClusterFile frames preserve empty frames and reuse storage", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(-42, 0.0); + auto second = make_test_frame(105, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(ClusterVector(0, -43)); + writer.write_frame(first); + writer.write_frame(ClusterVector(0, 0)); + writer.write_frame(second); + } + + ClusterFile reader(file.path()); + auto frames = reader.frames(); + CHECK(reader.tell() == 0); + const TestCluster *storage = nullptr; + std::vector frame_numbers; + for (auto &frame : frames) { + frame_numbers.push_back(frame.frame_number()); + if (frame.frame_number() == -42) { + check_frame(frame, first); + storage = frame.data(); + } else if (frame.frame_number() == 105) { + check_frame(frame, second); + CHECK(frame.data() == storage); + } else { + CHECK(frame.empty()); + } + } + CHECK(frame_numbers == std::vector{-43, -42, 0, 105}); + CHECK(frames.begin() == frames.end()); +} + +TEST_CASE("ClusterFile chunks preserve cluster order across frame boundaries", + "[ClusterFile]") { + const auto chunk_size = GENERATE(1, 2, 3, 10); + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(ClusterVector(0)); + writer.write_frame(first); + writer.write_frame(ClusterVector(0)); + writer.write_frame(second); + writer.write_frame(ClusterVector(0)); + } + + ClusterFile reader(file.path(), chunk_size); + auto chunks = reader.chunks(); + auto requested = static_cast(chunk_size); + SECTION("constructor size") {} + SECTION("explicit size overrides the constructor") { + chunks = reader.chunks(3); + requested = 3; + CHECK(reader.chunk_size() == static_cast(chunk_size)); + } + CHECK(reader.tell() == 0); + size_t count = 0; + for (auto &chunk : chunks) { + REQUIRE(chunk.size() == std::min(requested, 4 - count)); + for (const auto &cluster : chunk) { + REQUIRE(count < 4); + const auto &expected = count < 2 ? first[count] : second[count - 2]; + CHECK(cluster.x == expected.x); + CHECK(cluster.y == expected.y); + CHECK(cluster.data == expected.data); + ++count; + } + } + CHECK(count == 4); + CHECK(chunks.begin() == chunks.end()); +} + +TEST_CASE("ClusterFile iterator results can be moved out without copying", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first); + writer.write_frame(second); + } + ClusterFile reader(file.path(), 2); + auto check_moves = [&](auto range) { + auto it = range.begin(); + REQUIRE(it != range.end()); + CHECK(range.end() != it); + const auto storage = it->data(); + auto retained = std::move(*it); + CHECK(retained.data() == storage); + ++it; + REQUIRE(it != range.end()); + check_frame(*it, second); + check_frame(retained, first); + it++; + CHECK(it == range.end()); + CHECK(range.end() == it); + ++it; + CHECK(it == range.end()); + reader.close(); + check_frame(retained, first); + }; + SECTION("frames") { check_moves(reader.frames()); } + SECTION("chunks") { check_moves(reader.chunks()); } +} + +TEST_CASE("ClusterFile iteration preserves filtering and gain correction", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + const auto use_noise = GENERATE(false, true); + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(make_test_frame(42, 0.0)); + writer.write_frame(make_test_frame(43, 100.0)); + } + ClusterFile reader(file.path(), 3); + ClusterFile reference(file.path()); + aare::NDArray gain({12, 12}, 2.0); + aare::NDArray noise({12, 12}, 5); + for (auto *file_reader : {&reader, &reference}) { + file_reader->set_gain_map(gain.view()); + if (use_roi) { + file_reader->set_roi(aare::ROI{0, 6, 0, 12}); + } + if (use_noise) { + file_reader->set_noise_map(noise.view()); + } + } + SECTION("frames include fully rejected frames") { + size_t count = 0; + for (auto &frame : reader.frames()) { + auto expected = reference.read_frame(); + REQUIRE(expected); + check_frame(frame, *expected); + ++count; + } + CHECK(count == 2); + CHECK_FALSE(reference.read_frame()); + } + SECTION("chunks count selected clusters") { + for (auto &chunk : reader.chunks()) { + auto expected = reference.read_clusters(3); + check_frame(chunk, expected); + } + CHECK(reference.read_clusters(3).empty()); + } +} + +TEST_CASE("ClusterFile iteration resumes after an early exit", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first); + writer.write_frame(second); + } + ClusterFile reader(file.path()); + SECTION("breaking a frame loop does not consume the following frame") { + for (auto &frame : reader.frames()) { + check_frame(frame, first); + break; + } + auto next = reader.read_frame(); + REQUIRE(next); + check_frame(*next, second); + } + SECTION("partial chunks must be completed before frame iteration") { + for (auto &chunk : reader.chunks(1)) { + REQUIRE(chunk.size() == 1); + CHECK(chunk[0].data == first[0].data); + break; + } + CHECK_THROWS_AS(reader.frames().begin(), std::runtime_error); + auto remainder = reader.read_clusters(1); + REQUIRE(remainder.size() == 1); + CHECK(remainder[0].data == first[1].data); + auto it = reader.frames().begin(); + REQUIRE(it != reader.frames().end()); + check_frame(*it, second); + } +} + +TEST_CASE("ClusterFile iterators distinguish EOF from read errors", + "[ClusterFile]") { + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(make_test_frame(42, 0.0)); + } + auto check_error = [&] { + ClusterFile frame_reader(file.path()); + ClusterFile chunk_reader(file.path()); + CHECK_THROWS_AS(frame_reader.frames().begin(), std::runtime_error); + CHECK_THROWS_AS(chunk_reader.chunks().begin(), std::runtime_error); + }; + SECTION("empty file") { + std::filesystem::resize_file(file.path(), 0); + ClusterFile reader(file.path()); + CHECK(reader.frames().begin() == reader.frames().end()); + CHECK(reader.chunks().begin() == reader.chunks().end()); + } + SECTION("partial header") { + std::filesystem::resize_file(file.path(), 7); + check_error(); + } + SECTION("partial record") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + check_error(); + } + SECTION("partial record encountered on increment") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + ClusterFile reader(file.path()); + auto it = reader.chunks(1).begin(); + REQUIRE(it != reader.chunks(1).end()); + CHECK_THROWS_AS(++it, std::runtime_error); + } + SECTION("closed before first read") { + ClusterFile reader(file.path()); + auto frames = reader.frames(); + auto chunks = reader.chunks(); + reader.close(); + CHECK_THROWS_AS(frames.begin(), std::runtime_error); + CHECK_THROWS_AS(chunks.begin(), std::runtime_error); + } + SECTION("closed during traversal") { + ClusterFile reader(file.path()); + auto it = reader.frames().begin(); + reader.close(); + CHECK_THROWS_AS(++it, std::runtime_error); + } + SECTION("write and append modes") { + const auto mode = GENERATE("w", "a"); + ClusterFile writer(file.path(), 1000, mode); + CHECK_THROWS_AS(writer.frames().begin(), std::runtime_error); + CHECK_THROWS_AS(writer.chunks().begin(), std::runtime_error); + } + SECTION("zero chunk sizes are rejected without reading") { + ClusterFile reader(file.path(), 0); + CHECK_THROWS_AS(reader.chunks(), std::invalid_argument); + CHECK_THROWS_AS(reader.chunks(0), std::invalid_argument); + CHECK(reader.tell() == 0); + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.frames().begin() != reader.frames().end()); + } +} + +TEST_CASE("Read frame and modify cluster data", "[.with-data]") { auto fpath = test_data_path() / "clust" / "single_frame_97_clustrers.clust"; REQUIRE(std::filesystem::exists(fpath)); ClusterFile> f(fpath); auto clusters = f.read_frame(); - CHECK(clusters.size() == 97); - CHECK(clusters.frame_number() == 135); + REQUIRE(clusters); + CHECK(clusters->size() == 97); + CHECK(clusters->frame_number() == 135); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - clusters.push_back( + clusters->push_back( Cluster{0, 0, {0, 1, 2, 3, 4, 5, 6, 7, 8}}); - CHECK(clusters.size() == 98); - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + CHECK(clusters->size() == 98); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } diff --git a/src/ClusterFinderMT.test.cpp b/src/ClusterFinderMT.test.cpp index fae8b247..ca630de4 100644 --- a/src/ClusterFinderMT.test.cpp +++ b/src/ClusterFinderMT.test.cpp @@ -22,9 +22,11 @@ class ClusterFinderMTWrapper public: ClusterFinderMTWrapper(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, size_t capacity = 2000, size_t n_threads = 3, - size_t queue_depth = 16) + size_t queue_depth = 16, + size_t minimum_pedestal_samples = 1000) : ClusterFinderMT( - image_size, nSigma, capacity, n_threads, queue_depth) {} + image_size, nSigma, capacity, n_threads, queue_depth, + minimum_pedestal_samples) {} size_t get_m_input_queues_size() const { return this->m_input_queues.size(); @@ -73,13 +75,14 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { File file(fpath); size_t n_threads = 2; - size_t n_frames_pd = 10; + size_t minimum_pedestal_samples = 1; + size_t n_frames = 10; using ClusterType = Cluster; ClusterFinderMTWrapper cf( {static_cast(file.rows()), static_cast(file.cols())}, - 5, 2000, n_threads); // no idea what frame type is!!! default uint16_t + 5, 2000, n_threads, 16, minimum_pedestal_samples); CHECK(cf.get_m_input_queues_size() == n_threads); CHECK(cf.get_m_output_queues_size() == n_threads); @@ -87,8 +90,11 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_output_queues_are_empty() == true); CHECK(cf.m_input_queues_are_empty() == true); - for (size_t i = 0; i < n_frames_pd; ++i) { - auto frame = file.read_frame(); + auto frame = file.read_frame(); + cf.push_pedestal_frame(frame.view()); + + for (size_t i = 0; i < n_frames; ++i) { + frame = file.read_frame(); cf.find_clusters(frame.view()); } @@ -97,7 +103,7 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_output_queues_are_empty() == true); CHECK(cf.m_input_queues_are_empty() == true); - CHECK(cf.m_sink_size() == n_frames_pd); + CHECK(cf.m_sink_size() == n_frames); ClusterCollector clustercollector(&cf); clustercollector.stop(); @@ -105,7 +111,6 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_sink_size() == 0); auto clustervec = clustercollector.steal_clusters(); - // CHECK(clustervec.size() == ) //dont know how many clusters to expect } TEST_CASE("frame buffers are recycled when pushing more frames than the pool " diff --git a/src/ClusterVector.test.cpp b/src/ClusterVector.test.cpp index 2c098e14..135719c1 100644 --- a/src/ClusterVector.test.cpp +++ b/src/ClusterVector.test.cpp @@ -1,5 +1,8 @@ // SPDX-License-Identifier: MPL-2.0 #include "aare/ClusterVector.hpp" +#include "aare/GainMap.hpp" +#include "aare/NDArray.hpp" +#include #include #include @@ -22,27 +25,79 @@ TEST_CASE("After pushing back one element the ClusterVector is not empty") { REQUIRE(!cv.empty()); } -TEST_CASE("ClusterVector move constructor preserves contents") { - ClusterVector source(4, 42); +TEST_CASE("Filtering a ClusterVector selects the masked elements") { + ClusterVector source(8, 123); source.push_back(C1{1, 2, {3, 4, 5, 6}}); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); + source.push_back(C1{13, 14, {15, 16, 17, 18}}); + source.push_back(C1{19, 20, {21, 22, 23, 24}}); - ClusterVector moved(std::move(source)); + SECTION("Some elements are selected") { + std::array mask{true, false, false, true}; - REQUIRE(moved.size() == 1); - CHECK(moved.frame_number() == 42); - CHECK(moved[0].data[3] == 6); + auto filtered = source(aare::NDView{mask.data(), {4}}); + + CHECK(filtered.size() == 2); + CHECK(filtered.frame_number() == 123); + CHECK(filtered[0].x == 1); + CHECK(filtered[1].x == 19); + } + + SECTION("No elements are selected") { + std::array mask{false, false, false, false}; + + auto filtered = source(aare::NDView{mask.data(), {4}}); + + CHECK(filtered.empty()); + CHECK(filtered.frame_number() == 123); + } + + SECTION("An empty vector accepts a default empty mask") { + ClusterVector empty_source(0, -123); + + auto filtered = empty_source(aare::NDView{}); + + CHECK(filtered.empty()); + CHECK(filtered.frame_number() == -123); + } } -TEST_CASE("ClusterVector move assignment preserves contents") { - ClusterVector source(4, 42); +TEST_CASE("Move constructing a ClusterVector transfers its storage") { + ClusterVector source(4, 123); source.push_back(C1{1, 2, {3, 4, 5, 6}}); - ClusterVector moved(1); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); - moved = std::move(source); + const auto *source_data = source.data(); + const auto source_capacity = source.capacity(); - REQUIRE(moved.size() == 1); - CHECK(moved.frame_number() == 42); - CHECK(moved[0].data[3] == 6); + ClusterVector destination(std::move(source)); + + CHECK(destination.data() == source_data); + CHECK(destination.capacity() == source_capacity); + CHECK(destination.size() == 2); + CHECK(destination.frame_number() == 123); + CHECK(destination[0].x == 1); + CHECK(destination[1].x == 7); +} + +TEST_CASE("Move assigning a ClusterVector transfers its storage") { + ClusterVector source(4, 123); + source.push_back(C1{1, 2, {3, 4, 5, 6}}); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); + + const auto *source_data = source.data(); + const auto source_capacity = source.capacity(); + + ClusterVector destination(2, 456); + destination.push_back(C1{13, 14, {15, 16, 17, 18}}); + destination = std::move(source); + + CHECK(destination.data() == source_data); + CHECK(destination.capacity() == source_capacity); + CHECK(destination.size() == 2); + CHECK(destination.frame_number() == 123); + CHECK(destination[0].x == 1); + CHECK(destination[1].x == 7); } TEST_CASE("item_size return the size of the cluster stored") { @@ -256,6 +311,28 @@ TEST_CASE("Concatenate two cluster vectors where we need to allocate") { REQUIRE(ptr[3].y == 17); } +TEST_CASE("Reducing a ClusterVector preserves its frame number") { + SECTION("Reduce to 2x2") { + ClusterVector> source(1, -135); + source.push_back(Cluster{}); + + auto reduced = aare::reduce_to_2x2(source); + + CHECK(reduced.size() == source.size()); + CHECK(reduced.frame_number() == source.frame_number()); + } + + SECTION("Reduce to 3x3") { + ClusterVector> source(1, -246); + source.push_back(Cluster{}); + + auto reduced = aare::reduce_to_3x3(source); + + CHECK(reduced.size() == source.size()); + CHECK(reduced.frame_number() == source.frame_number()); + } +} + struct ClusterTestData { uint8_t ClusterSizeX; uint8_t ClusterSizeY; @@ -300,3 +377,73 @@ TEST_CASE("Gain Map Calculation Index Map") { CHECK(index_map_x == clustertestdata.index_map_x); CHECK(index_map_y == clustertestdata.index_map_y); } + +namespace { + +template +void check_gain_map_cluster_bounds() { + using ClusterType = Cluster; + + constexpr ssize_t rows = 16; + constexpr ssize_t cols = 16; + constexpr uint16_t left = ClusterSizeX / 2; + constexpr uint16_t right = ClusterSizeX - left - 1; + constexpr uint16_t top = ClusterSizeY / 2; + constexpr uint16_t bottom = ClusterSizeY - top - 1; + + aare::NDArray gain_map({rows, cols}, 2.0); + aare::InvertedGainMap inverted_gain_map(gain_map); + ClusterVector clusters(6); + + const auto add_cluster = [&clusters](uint16_t x, uint16_t y) { + ClusterType cluster{}; + cluster.x = x; + cluster.y = y; + cluster.data.fill(2.0); + clusters.push_back(cluster); + }; + + add_cluster(left, top); + add_cluster(cols - right - 1, rows - bottom - 1); + add_cluster(left - 1, top); + add_cluster(left, top - 1); + add_cluster(cols - right, top); + add_cluster(left, rows - bottom); + + inverted_gain_map.apply_gain_map(clusters); + + for (size_t i = 0; i < 2; ++i) { + CHECK(std::all_of(clusters[i].data.begin(), clusters[i].data.end(), + [](double value) { return value == 1.0; })); + } + for (size_t i = 2; i < clusters.size(); ++i) { + CHECK(std::all_of(clusters[i].data.begin(), clusters[i].data.end(), + [](double value) { return value == 0.0; })); + } + + aare::NDArray small_gain_map( + {ClusterSizeY - 1, ClusterSizeX - 1}, 2.0); + aare::InvertedGainMap small_inverted_gain_map(small_gain_map); + ClusterVector oversized_cluster(1); + ClusterType cluster{}; + cluster.x = left; + cluster.y = top; + cluster.data.fill(2.0); + oversized_cluster.push_back(cluster); + + small_inverted_gain_map.apply_gain_map(oversized_cluster); + + CHECK(std::all_of(oversized_cluster[0].data.begin(), + oversized_cluster[0].data.end(), + [](double value) { return value == 0.0; })); +} + +} // namespace + +TEST_CASE("Gain map bounds cover the full cluster footprint", "[GainMap]") { + SECTION("3x3") { check_gain_map_cluster_bounds<3, 3>(); } + SECTION("5x5") { check_gain_map_cluster_bounds<5, 5>(); } + SECTION("7x7") { check_gain_map_cluster_bounds<7, 7>(); } + SECTION("9x9") { check_gain_map_cluster_bounds<9, 9>(); } + SECTION("5x7") { check_gain_map_cluster_bounds<5, 7>(); } +} diff --git a/src/CtbRawFile.cpp b/src/CtbRawFile.cpp index 260ab668..e4d0b958 100644 --- a/src/CtbRawFile.cpp +++ b/src/CtbRawFile.cpp @@ -24,9 +24,9 @@ void CtbRawFile::read_into(std::byte *image_buf, DetectorHeader *header) { throw std::runtime_error(LOCATION + " End of file reached"); } - if (m_current_frame != 0 && - m_current_frame % m_master.max_frames_per_file() == 0) { - open_data_file(m_current_subfile + 1); + const auto index = sub_file_index(m_current_frame); + if (index != m_current_subfile) { + open_data_file(index); } if (header) { @@ -79,4 +79,4 @@ void CtbRawFile::open_data_file(size_t subfile_index) { } } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/src/DetectorGeometry.cpp b/src/DetectorGeometry.cpp index ee3525f2..c0ceef9a 100644 --- a/src/DetectorGeometry.cpp +++ b/src/DetectorGeometry.cpp @@ -11,7 +11,8 @@ DetectorGeometry::DetectorGeometry(const xy &geometry, const ssize_t module_pixels_x, const ssize_t module_pixels_y, const xy udp_interfaces_per_module, - const bool quad) { + const bool quad) + : m_udp_interfaces_per_module(udp_interfaces_per_module) { size_t num_modules = geometry.col * geometry.row; module_geometries.reserve(num_modules); @@ -43,6 +44,10 @@ DetectorGeometry::DetectorGeometry(const xy &geometry, m_pixels_y += static_cast((geometry.row - 1) * cfg.module_gap_row); } +xy DetectorGeometry::udp_interfaces_per_module() const { + return m_udp_interfaces_per_module; +} + size_t DetectorGeometry::n_modules() const { return m_modules_x * m_modules_y; } size_t DetectorGeometry::pixels_x() const { return m_pixels_x; } diff --git a/src/FastPedestal.test.cpp b/src/FastPedestal.test.cpp new file mode 100644 index 00000000..7c003849 --- /dev/null +++ b/src/FastPedestal.test.cpp @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/FastPedestal.hpp" + +#include +#include + +using namespace aare; + +TEMPLATE_TEST_CASE("fast pedestal keeps variance in double precision", + "[fast-pedestal]", double, float, int16_t) { + FastPedestal pedestal(1, 1, 2); + NDArray frame({1, 1}, 0); + pedestal.add_init_frame(frame.view()); + frame[0] = 1000; + pedestal.add_init_frame(frame.view()); + + const auto &ready = pedestal; + REQUIRE(ready.std(0, 0) == static_cast(500)); + REQUIRE(ready.std_unchecked(0) == ready.std(0, 0)); + REQUIRE(ready.std()[0] == ready.std(0, 0)); + + pedestal.push_ema(frame.view()); + REQUIRE(ready.std(0, 0) == static_cast(std::sqrt(187500.0))); + REQUIRE(ready.std_unchecked(0) == ready.std(0, 0)); + REQUIRE(ready.std()[0] == ready.std(0, 0)); +} + +TEST_CASE("fast pedestal validates standard deviation access", + "[fast-pedestal]") { + FastPedestal pedestal(1, 1, 1); + REQUIRE_THROWS(pedestal.std()); + REQUIRE_THROWS(pedestal.std(0, 0)); + + NDArray frame({1, 1}, 42); + pedestal.add_init_frame(frame.view()); + REQUIRE(pedestal.std(0, 0) == 0.0); + REQUIRE_THROWS(pedestal.std(1, 0)); + REQUIRE_THROWS(pedestal.std(0, 1)); +} + +TEMPLATE_TEST_CASE("fast pedestal std stays finite after settling", + "[fast-pedestal]", double, float, int16_t) { + FastPedestal pedestal(1, 1, 10); + NDArray frame({1, 1}, 16382); + pedestal.add_init_frame(frame.view()); + frame[0] = 16383; + for (int i = 0; i < 9; ++i) { + pedestal.add_init_frame(frame.view()); + } + for (int i = 0; i < 300; ++i) { + pedestal.push_ema(frame.view()); + } + + const auto noise = pedestal.std(0, 0); + REQUIRE(std::isfinite(noise)); + REQUIRE(noise >= 0); + REQUIRE(static_cast(noise) < 1e-3); + REQUIRE(pedestal.std_unchecked(0) == noise); + REQUIRE(pedestal.std()[0] == noise); +} diff --git a/src/FilePtr.cpp b/src/FilePtr.cpp index dd907bda..45e65d46 100644 --- a/src/FilePtr.cpp +++ b/src/FilePtr.cpp @@ -22,6 +22,8 @@ FilePtr &FilePtr::operator=(FilePtr &&other) { return *this; } +FilePtr::operator bool() const noexcept { return fp_ != nullptr; } + FILE *FilePtr::get() { return fp_; } ssize_t FilePtr::tell() { diff --git a/src/FilePtr.test.cpp b/src/FilePtr.test.cpp new file mode 100644 index 00000000..6b23e62a --- /dev/null +++ b/src/FilePtr.test.cpp @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/FilePtr.hpp" + +#include +#include + +TEST_CASE("FilePtr converts to its open state", "[FilePtr]") { + aare::FilePtr empty; + CHECK_FALSE(empty); + + aare::FilePtr open(__FILE__, "rb"); + CHECK(open); + + aare::FilePtr moved(std::move(open)); + CHECK(moved); + CHECK_FALSE(open); +} diff --git a/src/Pedestal.test.cpp b/src/Pedestal.test.cpp index 099c47cf..abab15d2 100644 --- a/src/Pedestal.test.cpp +++ b/src/Pedestal.test.cpp @@ -1,12 +1,35 @@ // SPDX-License-Identifier: MPL-2.0 #include "aare/Pedestal.hpp" +#include #include #include #include #include using namespace aare; + +TEMPLATE_TEST_CASE("pedestal uses double precision moments", "[pedestal]", + double, float, int16_t) { + Pedestal pedestal(1, 1, 2); + pedestal.push(0, 0, uint16_t{30000}); + pedestal.push(0, 0, uint16_t{30003}); + + REQUIRE(pedestal.mean(0, 0) == static_cast(30001.5)); + REQUIRE(pedestal.std(0, 0) == static_cast(1.5)); + + pedestal.push(0, 0, uint16_t{30002}); + + REQUIRE(pedestal.mean(0, 0) == static_cast(30001.75)); + // Static cast to double is needed to avoid a Catch2 warning about + // float to double conversion on mac. + REQUIRE_THAT( + static_cast(pedestal.std(0, 0)), + Catch::Matchers::WithinAbs(static_cast(static_cast( + std::sqrt(TestType(1.1875)))), + static_cast(TestType(1e-6)))); +} + TEST_CASE("test pedestal constructor") { aare::Pedestal pedestal(10, 10, 5); REQUIRE(pedestal.rows() == 10); @@ -14,13 +37,47 @@ TEST_CASE("test pedestal constructor") { REQUIRE(pedestal.n_samples() == 5); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == 0); - REQUIRE(pedestal.get_sum2()(i, j) == 0); + REQUIRE(pedestal.mean(i, j) == 0); REQUIRE(pedestal.cur_samples()(i, j) == 0); } } } +TEST_CASE("pedestal rejects mismatched frame shapes", "[pedestal]") { + Pedestal<> pedestal(2, 3); + pedestal.push(0, 0, uint16_t{7}); + NDArray threshold({2, 3}, 10.0); + + for (const auto shape : + {std::array{2, 2}, std::array{3, 2}}) { + Frame frame(shape[0], shape[1], Dtype::UINT16); + REQUIRE_THROWS_WITH(pedestal.push(frame.view()), + "Frame shape does not match pedestal shape"); + REQUIRE_THROWS_WITH(pedestal.push_with_threshold(frame.view(), + threshold.view()), + "Frame shape does not match pedestal shape"); + REQUIRE_THROWS_WITH(pedestal.push(frame), + "Frame shape does not match pedestal shape"); + REQUIRE(pedestal.mean(0, 0) == 7); + REQUIRE(pedestal.cur_samples()(0, 0) == 1); + } +} + +TEMPLATE_TEST_CASE("pedestal std stays finite after settling", "[pedestal]", + double, float, int16_t) { + Pedestal pedestal(1, 1, 10); + pedestal.push(0, 0, uint16_t{16382}); + for (int i = 0; i < 201; ++i) { + pedestal.push(0, 0, uint16_t{16383}); + } + + const auto noise = pedestal.std(0, 0); + REQUIRE(std::isfinite(noise)); + REQUIRE(noise >= 0); + REQUIRE(static_cast(noise) < 1e-3); + REQUIRE(pedestal.std()(0, 0) == noise); +} + TEST_CASE("test pedestal push") { aare::Pedestal pedestal(10, 10, 5); aare::Frame frame(10, 10, Dtype::UINT16); @@ -34,8 +91,7 @@ TEST_CASE("test pedestal push") { pedestal.push(frame); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == i + j); - REQUIRE(pedestal.get_sum2()(i, j) == (i + j) * (i + j)); + REQUIRE(pedestal.mean(i, j) == i + j); REQUIRE(pedestal.cur_samples()(i, j) == 1); } } @@ -44,8 +100,7 @@ TEST_CASE("test pedestal push") { pedestal.clear(); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == 0); - REQUIRE(pedestal.get_sum2()(i, j) == 0); + REQUIRE(pedestal.mean(i, j) == 0); REQUIRE(pedestal.cur_samples()(i, j) == 0); } } @@ -57,16 +112,10 @@ TEST_CASE("test pedestal push") { for (uint32_t j = 0; j < 10; j++) { if (k < 5) { REQUIRE(pedestal.cur_samples()(i, j) == k + 1); - REQUIRE(pedestal.get_sum()(i, j) == (k + 1) * (i + j)); - REQUIRE(pedestal.get_sum2()(i, j) == - (k + 1) * (i + j) * (i + j)); } else { REQUIRE(pedestal.cur_samples()(i, j) == 5); - REQUIRE(pedestal.get_sum()(i, j) == 5 * (i + j)); - REQUIRE(pedestal.get_sum2()(i, j) == 5 * (i + j) * (i + j)); } REQUIRE(pedestal.mean(i, j) == (i + j)); - REQUIRE(pedestal.variance(i, j) == 0); REQUIRE(pedestal.std(i, j) == 0); } } @@ -74,7 +123,7 @@ TEST_CASE("test pedestal push") { } TEST_CASE("test pedestal with normal distribution") { - const double MEAN = 5.0, STD = 2.0, VAR = STD * STD, TOLERANCE = 0.1; + const double MEAN = 5.0, STD = 2.0, TOLERANCE = 0.1; unsigned seed = std::chrono::system_clock::now().time_since_epoch().count(); std::default_random_engine generator(seed); @@ -91,17 +140,14 @@ TEST_CASE("test pedestal with normal distribution") { pedestal.push(frame); } auto mean = pedestal.mean(); - auto variance = pedestal.variance(); auto standard_deviation = pedestal.std(); for (int i = 0; i < 3; i++) { for (int j = 0; j < 5; j++) { REQUIRE_THAT(mean(i, j), Catch::Matchers::WithinAbs(MEAN, MEAN * TOLERANCE)); - REQUIRE_THAT(variance(i, j), - Catch::Matchers::WithinAbs(VAR, VAR * TOLERANCE)); REQUIRE_THAT(standard_deviation(i, j), Catch::Matchers::WithinAbs(STD, STD * TOLERANCE)); } } -} \ No newline at end of file +} diff --git a/src/ROI.cpp b/src/ROI.cpp new file mode 100644 index 00000000..9cb37961 --- /dev/null +++ b/src/ROI.cpp @@ -0,0 +1,43 @@ +#include "aare/ROI.hpp" +#include "aare/DetectorGeometry.hpp" +#include "aare/ROIGeometry.hpp" + +namespace aare { + +/** + * @brief Check if the ROI covers the entire detector geometry + * @param roi Region of interest + * @param geometry Detector geometry + * @return true if the ROI covers the entire detector geometry, false otherwise + */ +bool complete_ROI(const ROI &roi, const DetectorGeometry &geometry) { + return roi.xmin == 0 && + roi.xmax == static_cast(geometry.pixels_x()) && + roi.ymin == 0 && + roi.ymax == static_cast(geometry.pixels_y()); +} + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry) { + if (rois.empty() or rois.size() > 1) { + return false; + } else { + return complete_ROI(rois[0], geometry); + } +} + +bool complete_ROI(const ROIGeometry &roi, const DetectorGeometry &geometry) { + return roi.pixels_x() == geometry.pixels_x() && + roi.pixels_y() == geometry.pixels_y(); +} + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry) { + if (rois.empty() or rois.size() > 1) { + return false; + } else { + return complete_ROI(rois[0], geometry); + } +} + +} // namespace aare \ No newline at end of file diff --git a/src/ROI.test.cpp b/src/ROI.test.cpp new file mode 100644 index 00000000..6cc7bb26 --- /dev/null +++ b/src/ROI.test.cpp @@ -0,0 +1,57 @@ +#include "aare/ROI.hpp" +#include + +namespace aare { + +TEST_CASE("merge ROIs", "[utility_functions]") { + + SECTION("not fully contiguous") { + + std::vector rois = {ROI{20, 30, 50, 60}, ROI{20, 30, 40, 50}, + ROI{10, 20, 40, 50}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 2); + + REQUIRE(merged_rois[0] == ROI{10, 30, 40, 50}); + REQUIRE(merged_rois[1] == ROI{20, 30, 50, 60}); + } + SECTION("complex merge") { + + std::vector rois = {ROI{40, 50, 20, 30}, ROI{10, 20, 30, 40}, + ROI{10, 20, 50, 60}, ROI{20, 30, 30, 40}, + ROI{60, 70, 30, 40}, ROI{60, 70, 20, 30}, + ROI{20, 30, 20, 30}, ROI{10, 20, 20, 30}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 4); + + REQUIRE(merged_rois[0] == ROI{10, 30, 20, 40}); + REQUIRE(merged_rois[1] == ROI{10, 20, 50, 60}); + REQUIRE(merged_rois[2] == ROI{40, 50, 20, 30}); + REQUIRE(merged_rois[3] == ROI{60, 70, 20, 40}); + } + SECTION("horizontally aligned") { + std::vector rois = {ROI{10, 20, 30, 40}, ROI{10, 20, 50, 60}, + ROI{10, 20, 20, 30}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 2); + REQUIRE(merged_rois[0] == ROI{10, 20, 20, 40}); + REQUIRE(merged_rois[1] == ROI{10, 20, 50, 60}); + } + SECTION("vertically aligned") { + std::vector rois = {ROI{10, 20, 30, 40}, ROI{30, 40, 30, 40}, + ROI{20, 30, 30, 40}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 1); + REQUIRE(merged_rois[0] == ROI{10, 40, 30, 40}); + } +} + +} // namespace aare diff --git a/src/ROIGeometry.cpp b/src/ROIGeometry.cpp index 04cf9187..f61f9079 100644 --- a/src/ROIGeometry.cpp +++ b/src/ROIGeometry.cpp @@ -5,25 +5,24 @@ namespace aare { ROIGeometry::ROIGeometry(const ROI &roi, DetectorGeometry &geometry) : m_pixels_x(roi.width()), m_pixels_y(roi.height()), m_geometry(geometry) { - m_module_indices_in_roi.reserve(m_geometry.n_modules()); - // determine which modules are in the roi - for (size_t i = 0; i < m_geometry.n_modules(); ++i) { - auto &module_geometry = m_geometry.get_module_geometries(i); - if (module_geometry.module_in_roi(roi)) { - module_geometry.update_geometry_with_roi(roi); - m_module_indices_in_roi.push_back(i); + + if (complete_ROI({roi}, geometry)) { + m_module_indices_in_roi.resize(m_geometry.n_modules()); + std::iota(m_module_indices_in_roi.begin(), + m_module_indices_in_roi.end(), 0); + } else { + m_module_indices_in_roi.reserve(m_geometry.n_modules()); + // determine which modules are in the roi + for (size_t i = 0; i < m_geometry.n_modules(); ++i) { + auto &module_geometry = m_geometry.get_module_geometries(i); + if (module_geometry.module_in_roi(roi)) { + module_geometry.update_geometry_with_roi(roi); + m_module_indices_in_roi.push_back(i); + } } } } -ROIGeometry::ROIGeometry(DetectorGeometry &geometry) - : m_pixels_x(geometry.pixels_x()), m_pixels_y(geometry.pixels_y()), - m_geometry(geometry) { - m_module_indices_in_roi.resize(m_geometry.n_modules()); - std::iota(m_module_indices_in_roi.begin(), m_module_indices_in_roi.end(), - 0); -} - size_t ROIGeometry::num_modules_in_roi() const { return m_module_indices_in_roi.size(); } diff --git a/src/RawFile.cpp b/src/RawFile.cpp index cecf3e82..1e33bf83 100644 --- a/src/RawFile.cpp +++ b/src/RawFile.cpp @@ -2,11 +2,13 @@ #include "aare/RawFile.hpp" #include "aare/DetectorGeometry.hpp" #include "aare/PixelMap.hpp" +#include "aare/ROI.hpp" #include "aare/ROIGeometry.hpp" #include "aare/algorithm.hpp" #include "aare/defs.hpp" #include "aare/logger.hpp" +#include #include #include @@ -14,36 +16,62 @@ using json = nlohmann::json; namespace aare { +std::runtime_error RawFile::frame_error(size_t frame_index, + const std::string &message) const { + return std::runtime_error( + fmt::format("Error reading frame index {} from file '{}': {}", + frame_index, m_master.master_fname().string(), message)); +} + RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode) - : m_master(fname), - m_geometry(m_master.geometry(), m_master.pixels_x(), m_master.pixels_y(), - m_master.udp_interfaces_per_module(), m_master.quad()) { + : m_master(fname) { m_mode = mode; - m_subfiles.resize(m_master.rois().has_value() ? m_master.rois()->size() - : 1); - if (mode == "r") { - if (m_master.rois().has_value()) { - m_ROI_geometries.reserve(m_master.rois()->size()); - - // iterate over all ROIS - size_t roi_index = 0; - const auto rois = m_master.rois().value(); - for (const auto &roi : rois) { - m_ROI_geometries.push_back(ROIGeometry(roi, m_geometry)); - // open subfiles - open_subfiles(roi_index); - ++roi_index; - } - - } else { - // no ROI use full detector - m_ROI_geometries.reserve(1); - m_ROI_geometries.push_back(ROIGeometry(m_geometry)); - open_subfiles(0); + if (m_master.frame_padding() == 0 && + m_master.frame_discard_policy() != + FrameDiscardPolicy::DiscardPartial) { + throw std::runtime_error(fmt::format( + "Cannot open '{}': RawFile requires frame padding or " + "discardpartial.", + m_master.master_fname().string())); } + + m_subfiles.resize(m_master.roi_geometries().size()); + // iterate over all ROIS + const size_t num_rois = m_master.roi_geometries().size(); + + for (size_t roi_index = 0; roi_index < num_rois; ++roi_index) { + // open subfiles + open_subfiles(roi_index); + } + + std::optional min_frames; + size_t max_frames = 0; + for (const auto &subfiles : m_subfiles) { + for (const auto &subfile : subfiles) { + const auto count = subfile->frames_in_file(); + min_frames = min_frames ? std::min(*min_frames, count) : count; + max_frames = std::max(max_frames, count); + } + } + if (!min_frames) { + throw std::runtime_error(fmt::format( + "No raw subfiles selected by '{}'", fname.string())); + } + m_frames_in_file = *min_frames; + + if (m_frames_in_file != max_frames || + m_frames_in_file != m_master.frames_in_file()) { + LOG(logWARNING) << fmt::format( + "'{}': Different number of frames across subfiles. Expected {} " + "frames but found min/max {}/{}, " + "using {} frames.", + fname.string(), m_master.frames_in_file(), m_frames_in_file, + max_frames, m_frames_in_file); + } + LOG(logDEBUG) << "Frames in file: " << m_frames_in_file; } else { throw std::runtime_error(LOCATION + " Unsupported mode. Can only read RawFiles."); @@ -52,24 +80,15 @@ RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode) Frame RawFile::read_roi(const size_t roi_index) { - if (!m_master.rois()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } - if (roi_index >= m_ROI_geometries.size()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + if (roi_index >= m_master.roi_geometries().size()) { + throw frame_error(m_current_frame, + LOCATION + "ROI index out of range."); } return get_frame(m_current_frame++, roi_index); } std::vector RawFile::read_rois() { - - if (!m_master.rois()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } - - const size_t num_rois = m_ROI_geometries.size(); + const size_t num_rois = m_master.roi_geometries().size(); std::vector frames; frames.reserve(num_rois); @@ -83,18 +102,19 @@ std::vector RawFile::read_rois() { } Frame RawFile::read_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Multiple ROIs defined in the master file. " - "Use read_ROIs() instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, LOCATION + + "Multiple ROIs present in file. " + "Use read_ROIs() instead."); } return get_frame(m_current_frame++); } Frame RawFile::read_frame(size_t frame_number) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error( - LOCATION + "Multiple ROIs defined in the master file. " + if (m_master.roi_geometries().size() > 1) { + throw frame_error( + frame_number, + LOCATION + "Multiple ROIs present in file. " "Use read_ROIs(const size_t frame_number) instead."); } seek(frame_number); @@ -103,9 +123,9 @@ Frame RawFile::read_frame(size_t frame_number) { void RawFile::read_into(std::byte *image_buf, size_t n_frames) { // TODO: implement this in a more efficient way - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + "Cannot use read_into for multiple ROIs."); } for (size_t i = 0; i < n_frames; i++) { @@ -115,31 +135,29 @@ void RawFile::read_into(std::byte *image_buf, size_t n_frames) { } void RawFile::read_into(std::byte *image_buf) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs. Use " - "read_roi_into() for a single ROI instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + + "Cannot use read_into for multiple ROIs. Use " + "read_roi_into() for a single ROI instead."); } return get_frame_into(m_current_frame++, image_buf); } void RawFile::read_roi_into(std::byte *image_buf, const size_t roi_index, const size_t frame_number, DetectorHeader *header) { - if (!m_master.rois().has_value()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } if (roi_index >= num_rois()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + throw frame_error(frame_number, LOCATION + "ROI index out of range."); } return get_frame_into(frame_number, image_buf, roi_index, header); } void RawFile::read_into(std::byte *image_buf, DetectorHeader *header) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs. Use " - "read_roi_into() for a single ROI instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + + "Cannot use read_into for multiple ROIs. Use " + "read_roi_into() for a single ROI instead."); } return get_frame_into(m_current_frame++, image_buf, 0, header); } @@ -148,25 +166,29 @@ void RawFile::read_into(std::byte *image_buf, size_t n_frames, DetectorHeader *header) { // return get_frame_into(m_current_frame++, image_buf, header); - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error( + if (m_master.roi_geometries().size() > 1) { + throw frame_error( + m_current_frame, LOCATION + - "Cannot use read_into for multiple ROIs."); // TODO: maybe pass - // roi_index so one can - // use read_into for a - // specific ROI + "Cannot use read_into for multiple ROIs."); // TODO: maybe + // pass + // roi_index so + // one can use + // read_into for + // a specific + // ROI } for (size_t i = 0; i < n_frames; i++) { this->get_frame_into(m_current_frame++, image_buf, 0, header); image_buf += bytes_per_frame(); if (header) - header += m_ROI_geometries[0].num_modules_in_roi(); + header += m_master.roi_geometries()[0].num_modules_in_roi(); } } size_t RawFile::bytes_per_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error( LOCATION + "Pass the desired roi_index to bytes_per_frame to get " "bytes_per_frame for the specific ROI. "); @@ -175,13 +197,13 @@ size_t RawFile::bytes_per_frame() { } size_t RawFile::bytes_per_frame(const size_t roi_index) { - return m_ROI_geometries.at(roi_index).pixels_x() * - m_ROI_geometries.at(roi_index).pixels_y() * m_master.bitdepth() / - bits_per_byte; + return m_master.roi_geometries().at(roi_index).pixels_x() * + m_master.roi_geometries().at(roi_index).pixels_y() * + m_master.bitdepth() / bits_per_byte; } size_t RawFile::pixels_per_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error( LOCATION + "Pass the desired roi_index to pixels_per_frame to get " "pixels_per_frame for the specific ROI. "); @@ -190,8 +212,8 @@ size_t RawFile::pixels_per_frame() { } size_t RawFile::pixels_per_frame(const size_t roi_index) { - return m_ROI_geometries.at(roi_index).pixels_x() * - m_ROI_geometries.at(roi_index).pixels_y(); + return m_master.roi_geometries().at(roi_index).pixels_x() * + m_master.roi_geometries().at(roi_index).pixels_y(); } DetectorType RawFile::detector_type() const { return m_master.detector_type(); } @@ -201,7 +223,8 @@ void RawFile::seek(size_t frame_index) { // if frame_number == total_frames, then the next read will throw an // error if (frame_index > total_frames()) { - throw std::runtime_error( + throw frame_error( + frame_index, fmt::format("frame number {} is greater than total frames {}", frame_index, total_frames())); } @@ -210,10 +233,10 @@ void RawFile::seek(size_t frame_index) { size_t RawFile::tell() { return m_current_frame; } -size_t RawFile::total_frames() const { return m_master.frames_in_file(); } +size_t RawFile::total_frames() const { return m_frames_in_file; } size_t RawFile::rows() const { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error(LOCATION + "Pass the desired roi_index to rows to get " "rows for the specific ROI. "); @@ -221,10 +244,10 @@ size_t RawFile::rows() const { return rows(0); } size_t RawFile::rows(const size_t roi_index) const { - return m_ROI_geometries.at(roi_index).pixels_y(); + return m_master.roi_geometries().at(roi_index).pixels_y(); } size_t RawFile::cols() const { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error(LOCATION + "Pass the desired roi_index to cols to get " "cols for the specific ROI. "); @@ -232,50 +255,41 @@ size_t RawFile::cols() const { return cols(0); } size_t RawFile::cols(const size_t roi_index) const { - return m_ROI_geometries.at(roi_index).pixels_x(); + return m_master.roi_geometries().at(roi_index).pixels_x(); } size_t RawFile::bitdepth() const { return m_master.bitdepth(); } -xy RawFile::geometry() const { - return xy{static_cast(m_geometry.modules_y()), - static_cast(m_geometry.modules_x())}; -} -size_t RawFile::n_modules() const { return m_geometry.n_modules(); }; +xy RawFile::geometry() const { return m_master.detector_layout(); } -size_t RawFile::num_rois() const { - if (m_master.rois().has_value()) { - return m_master.rois()->size(); - } else { - return 0; - } -} +size_t RawFile::n_modules() const { return m_master.n_modules(); }; + +size_t RawFile::num_rois() const { return m_master.roi_geometries().size(); } const ROIGeometry &RawFile::roi_geometries(size_t roi_index) const { - return m_ROI_geometries[roi_index]; + return m_master.roi_geometries().at(roi_index); } std::vector RawFile::n_modules_in_roi() const { - std::vector results(m_ROI_geometries.size()); + std::vector results(m_master.roi_geometries().size()); std::transform( - m_ROI_geometries.begin(), m_ROI_geometries.end(), results.begin(), + m_master.roi_geometries().begin(), m_master.roi_geometries().end(), + results.begin(), [](const ROIGeometry &roi) { return roi.num_modules_in_roi(); }); return results; } void RawFile::open_subfiles(const size_t roi_index) { - if (m_mode == "r") { m_subfiles[roi_index].reserve( - m_ROI_geometries[roi_index].num_modules_in_roi()); + m_master.roi_geometries().at(roi_index).num_modules_in_roi()); auto module_indices = - m_ROI_geometries[roi_index].module_indices_in_roi(); + m_master.roi_geometries().at(roi_index).module_indices_in_roi(); - for (const size_t i : - m_ROI_geometries[roi_index].module_indices_in_roi()) { - const auto pos = m_geometry.get_module_geometries(i); + for (const size_t i : module_indices) { + const auto pos = m_master.geometry().get_module_geometries(i); m_subfiles[roi_index].emplace_back(std::make_unique( m_master.data_fname(i, 0), m_master.detector_type(), pos.height, pos.width, m_master.bitdepth(), pos.row_index, pos.col_index)); @@ -290,14 +304,22 @@ DetectorHeader RawFile::read_header(const std::filesystem::path &fname) { DetectorHeader h{}; FILE *fp = fopen(fname.string().c_str(), "r"); if (!fp) - throw std::runtime_error( - fmt::format("Could not open: {} for reading", fname.string())); + throw std::runtime_error(fmt::format( + "Could not open file '{}' for frame index 0", fname.string())); size_t const rc = fread(reinterpret_cast(&h), sizeof(h), 1, fp); - if (rc != 1) - throw std::runtime_error(LOCATION + "Could not read header from file"); + if (rc != 1) { + fclose(fp); + throw std::runtime_error( + LOCATION + fmt::format("Could not read header for frame index 0 " + "from file '{}'", + fname.string())); + } if (fclose(fp)) { - throw std::runtime_error(LOCATION + "Could not close file"); + throw std::runtime_error( + LOCATION + fmt::format("Could not close file '{}' after reading " + "frame index 0", + fname.string())); } return h; @@ -306,8 +328,8 @@ DetectorHeader RawFile::read_header(const std::filesystem::path &fname) { RawMasterFile RawFile::master() const { return m_master; } Frame RawFile::get_frame(size_t frame_index, const size_t roi_index) { - auto f = Frame(m_ROI_geometries[roi_index].pixels_y(), - m_ROI_geometries[roi_index].pixels_x(), + auto f = Frame(m_master.roi_geometries().at(roi_index).pixels_y(), + m_master.roi_geometries().at(roi_index).pixels_x(), Dtype::from_bitdepth(m_master.bitdepth())); std::byte *frame_buffer = f.data(); get_frame_into(frame_index, frame_buffer, roi_index); @@ -320,19 +342,26 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, const size_t roi_index, DetectorHeader *header) { LOG(logDEBUG) << "RawFile::get_frame_into(" << frame_index << ")"; if (frame_index >= total_frames()) { - throw std::runtime_error(LOCATION + "Frame number out of range"); + throw frame_error( + frame_index, + LOCATION + + fmt::format("Frame index {} is out of range: file contains {} " + "frames (indices are zero-based)", + frame_index, total_frames())); } std::vector frame_numbers( - m_ROI_geometries[roi_index].num_modules_in_roi()); + m_master.roi_geometries().at(roi_index).num_modules_in_roi()); std::vector frame_indices( - m_ROI_geometries[roi_index].num_modules_in_roi(), frame_index); + m_master.roi_geometries().at(roi_index).num_modules_in_roi(), + frame_index); // sync the frame numbers - if (m_ROI_geometries[roi_index].num_modules_in_roi() != + if (m_master.roi_geometries().at(roi_index).num_modules_in_roi() != 1) { // if we have more than one module for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { frame_numbers[part_idx] = m_subfiles[roi_index][part_idx]->frame_number(frame_index); @@ -352,8 +381,17 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, // 4. if we can't increase its index => throw error if (frame_indices[min_frame_idx] >= total_frames()) { - throw std::runtime_error(LOCATION + - "Frame number out of range"); + throw frame_error( + frame_index, + LOCATION + + fmt::format( + "Frame index {} out of range while synchronizing " + "module {} for ROI {}; last data file '{}'", + frame_indices[min_frame_idx], min_frame_idx, + roi_index, + m_subfiles[roi_index][min_frame_idx] + ->current_path() + .string())); } frame_numbers[min_frame_idx] = @@ -362,41 +400,45 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, } } - if (m_master.geometry().col == 1) { + if (m_master.detector_layout().col == 1) { // get the part from each subfile and copy it to the frame for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { auto corrected_idx = frame_indices[part_idx]; // This is where we start writing auto offset = - (m_geometry + (m_master.geometry() .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_y * - m_ROI_geometries[roi_index].pixels_x() + - m_geometry + m_master.roi_geometries().at(roi_index).pixels_x() + + m_master.geometry() .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_x) * m_master.bitdepth() / 8; - if (m_geometry - .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + if (m_master.geometry() + .get_module_geometries(m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_x != 0) - throw std::runtime_error( + throw frame_error( + frame_index, LOCATION + - " Implementation error. x pos not 0."); // TODO: - // origin - // can still - // change if - // roi - // changes + " Implementation error. x pos not 0."); // TODO: + // origin + // can still + // change if + // roi + // changes // TODO! What if the files don't match? m_subfiles[roi_index][part_idx]->seek(corrected_idx); m_subfiles[roi_index][part_idx]->read_into(frame_buffer + offset, @@ -420,10 +462,12 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, // the module level for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { - auto pos = m_geometry.get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi(part_idx)); + auto pos = m_master.geometry().get_module_geometries( + m_master.roi_geometries().at(roi_index).module_indices_in_roi( + part_idx)); auto corrected_idx = frame_indices[part_idx]; m_subfiles[roi_index][part_idx]->seek(corrected_idx); @@ -437,7 +481,8 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, auto irow = (pos.origin_y + cur_row); auto icol = pos.origin_x; auto dest = - (irow * m_ROI_geometries[roi_index].pixels_x() + icol); + (irow * m_master.roi_geometries().at(roi_index).pixels_x() + + icol); dest = dest * m_master.bitdepth() / 8; memcpy(frame_buffer + dest, part_buffer + @@ -452,12 +497,12 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, std::vector RawFile::read_n(size_t n_frames) { // TODO: implement this in a more efficient way if (num_rois() > 1) { - throw std::runtime_error(LOCATION + - "Multiple ROIs defined in the master " - "file. Use " - "read_num_rois for a specific ROI or use " - "read_ROIs to read one frame after " - "the other."); + throw frame_error(m_current_frame, + LOCATION + "Multiple ROIs defined in the master " + "file. Use " + "read_num_rois for a specific ROI or use " + "read_ROIs to read one frame after " + "the other."); } std::vector frames; @@ -472,7 +517,8 @@ std::vector RawFile::read_n(size_t n_frames) { std::vector RawFile::read_n_with_roi(const size_t n_frames, const size_t roi_index) { if (roi_index >= num_rois()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + throw frame_error(m_current_frame, + LOCATION + "ROI index out of range."); } std::vector frames; @@ -485,8 +531,8 @@ std::vector RawFile::read_n_with_roi(const size_t n_frames, } size_t RawFile::frame_number(size_t frame_index) { - if (frame_index >= m_master.frames_in_file()) { - throw std::runtime_error(LOCATION + " Frame number out of range"); + if (frame_index >= total_frames()) { + throw frame_error(frame_index, "Frame number out of range"); } return m_subfiles[0][0]->frame_number(frame_index); } diff --git a/src/RawFile.test.cpp b/src/RawFile.test.cpp index 88efc027..ee0b9e24 100644 --- a/src/RawFile.test.cpp +++ b/src/RawFile.test.cpp @@ -5,8 +5,10 @@ #include #include +#include #include +#include "raw_file_helpers.hpp" #include "test_config.hpp" #include "test_macros.hpp" @@ -14,6 +16,212 @@ using aare::File; using aare::RawFile; using namespace aare; +TEST_CASE("RawFile requires padding or discarding partial frames", + "[RawFile][frame-policy]") { + const auto [padding, policy, supported] = + GENERATE(table({ + {0, "nodiscard", false}, + {0, "discard", false}, + {0, "discardpartial", true}, + {1, "nodiscard", true}, + {1, "discard", true}, + {1, "discardpartial", true}, + })); + CAPTURE(padding, policy); + TemporaryRawFiles files; + nlohmann::json metadata; + std::ifstream(files.master_path()) >> metadata; + metadata["Frame Padding"] = padding; + metadata["Frame Discard Policy"] = policy; + std::ofstream(files.master_path()) << metadata; + + REQUIRE(RawMasterFile(files.master_path()).frame_padding() == padding); + if (supported) { + RawFile reader(files.master_path()); + REQUIRE(reader.total_frames() == 2); + auto frame = reader.read_frame(); + REQUIRE(frame.view()(0, 0) == 1); + File generic_reader(files.master_path()); + auto generic_frame = generic_reader.read_frame(); + REQUIRE(generic_frame.view()(0, 0) == 1); + } else { + SECTION("with data subfiles") {} + SECTION("without data subfiles") { + std::filesystem::remove(files.data_path(0, 0)); + std::filesystem::remove(files.data_path(0, 1)); + } + const auto path_matcher = + Catch::Matchers::ContainsSubstring(files.master_path().string()); + const auto policy_matcher = Catch::Matchers::ContainsSubstring( + "requires frame padding or discardpartial"); + const auto message = path_matcher && policy_matcher; + REQUIRE_THROWS_AS(RawFile(files.master_path()), std::runtime_error); + REQUIRE_THROWS_WITH(RawFile(files.master_path()), message); + REQUIRE_THROWS_WITH(File(files.master_path()), message); + } +} + +TEST_CASE("RawFile read errors identify the frame index and master path", + "[RawFile][read-errors]") { + TemporaryRawFiles files; + RawFile file(files.master_path()); + REQUIRE(file.frame_number(1) == 101); + file.seek(2); + const auto index_matcher = + Catch::Matchers::ContainsSubstring("frame index 2"); + const auto path_matcher = + Catch::Matchers::ContainsSubstring(files.master_path().string()); + const auto context = index_matcher && path_matcher; + std::vector buffer(file.bytes_per_frame() * 2); + DetectorHeader headers[2]{}; + + SECTION("sequential frame") { + REQUIRE_THROWS_WITH( + file.read_frame(), + index_matcher && path_matcher && + Catch::Matchers::ContainsSubstring( + "file contains 2 frames (indices are zero-based)") && + !Catch::Matchers::ContainsSubstring(".raw")); + } + SECTION("indexed frame at EOF") { + REQUIRE_THROWS_WITH(file.read_frame(2), context); + } + SECTION("indexed frame beyond EOF") { + REQUIRE_THROWS_WITH( + file.read_frame(17), + Catch::Matchers::ContainsSubstring("frame index 17") && + Catch::Matchers::ContainsSubstring( + files.master_path().string())); + } + SECTION("batch crosses EOF") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_n(2), context); + } + SECTION("read into buffer") { + FileInterface &reader = file; + REQUIRE_THROWS_WITH(reader.read_into(buffer.data()), context); + } + SECTION("batch into buffer") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), size_t{2}), context); + } + SECTION("read with header") { + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), headers), context); + } + SECTION("batch with headers") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), 2, headers), context); + } + SECTION("single ROI") { REQUIRE_THROWS_WITH(file.read_roi(0), context); } + SECTION("all ROIs") { REQUIRE_THROWS_WITH(file.read_rois(), context); } + SECTION("ROI into buffer") { + REQUIRE_THROWS_WITH(file.read_roi_into(buffer.data(), 0, 2), context); + } + SECTION("ROI batch") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_n_with_roi(2, 0), context); + } + SECTION("frame number") { + REQUIRE_THROWS_WITH(file.frame_number(2), context); + } + SECTION("generic File") { + File reader(files.master_path()); + REQUIRE_THROWS_WITH(reader.read_frame(2), context); + } +} + +TEST_CASE("RawFile synchronization errors retain the requested index", + "[RawFile][read-errors]") { + TemporaryRawFiles files(2); + const size_t lagging_module = GENERATE(0, 1); + { + std::fstream output(files.data_path(1 - lagging_module, 1), + std::ios::binary | std::ios::in | std::ios::out); + DetectorHeader header{}; + header.frameNumber = 102; + output.write(reinterpret_cast(&header), sizeof(header)); + } + RawFile file(files.master_path()); + REQUIRE_THROWS_WITH( + file.read_frame(1), + Catch::Matchers::ContainsSubstring("frame index 1") && + Catch::Matchers::ContainsSubstring(files.master_path().string()) && + Catch::Matchers::ContainsSubstring( + files.data_path(lagging_module, 1).string()) && + Catch::Matchers::ContainsSubstring("synchroniz")); +} + +TEST_CASE("RawFile reports the failing data file and requested frame", + "[RawFile][read-errors]") { + TemporaryRawFiles files; + RawFile file(files.master_path()); + std::filesystem::resize_file(files.data_path(0, 1), sizeof(DetectorHeader)); + REQUIRE_THROWS_WITH( + file.read_frame(1), + Catch::Matchers::ContainsSubstring("frame index 1") && + !Catch::Matchers::ContainsSubstring(files.master_path().string()) && + Catch::Matchers::ContainsSubstring( + files.data_path(0, 1).string()) && + Catch::Matchers::ContainsSubstring("End of file")); +} + +TEST_CASE("RawFile bounds use the shortest subfile", + "[RawFile][read-errors][frame-count]") { + TemporaryRawFiles files(2); + const size_t short_module = GENERATE(0, 1); + std::filesystem::remove(files.data_path(short_module, 1)); + RawFile file(files.master_path()); + REQUIRE(file.total_frames() == 1); + REQUIRE(file.master().frames_in_file() == 2); + REQUIRE_NOTHROW(file.read_frame(0)); + const auto expected = Catch::Matchers::ContainsSubstring( + "Error reading frame index 1 from file '" + + files.master_path().string() + "':"); + + SECTION("indexed frame") { + REQUIRE_THROWS_WITH(file.read_frame(1), expected); + } + SECTION("batch") { + file.seek(0); + REQUIRE_THROWS_WITH(file.read_n(2), expected); + } + SECTION("ROI") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_roi(0), expected); + } + SECTION("generic File") { + File reader(files.master_path()); + REQUIRE_THROWS_WITH(reader.read_frame(1), expected); + } + SECTION("frame number") { + REQUIRE_THROWS_WITH(file.frame_number(1), expected); + } +} + +TEST_CASE("RawFile frame count comes from the complete subfile series", + "[RawFile][frame-count]") { + TemporaryRawFiles files; + const size_t master_frames = GENERATE(0, 1, 2, 5); + auto metadata = nlohmann::json::parse(std::ifstream(files.master_path())); + metadata["Frames in File"] = master_frames; + metadata["Total Frames"] = 1000; + std::ofstream(files.master_path()) << metadata; + + RawFile file(files.master_path()); + REQUIRE(file.total_frames() == 2); + REQUIRE(file.master().frames_in_file() == master_frames); + REQUIRE(file.master().total_frames_expected() == 1000); + REQUIRE(file.frame_number(1) == 101); + REQUIRE_NOTHROW(file.read_frame(1)); + REQUIRE_THROWS(file.frame_number(2)); + REQUIRE_THROWS(file.read_frame(2)); + + File reader(files.master_path()); + REQUIRE(reader.total_frames() == 2); + REQUIRE(reader.frame_number(1) == 101); + REQUIRE_NOTHROW(reader.read_frame(1)); +} + TEST_CASE("Read number of frames from a jungfrau raw file", "[.with-data][RawFile]") { @@ -263,8 +471,9 @@ TEST_CASE("check find_geometry", "[.with-data][RawFile]") { RawMasterFile master_file(fpath); auto geometry = DetectorGeometry( - master_file.geometry(), master_file.pixels_x(), master_file.pixels_y(), - master_file.udp_interfaces_per_module(), master_file.quad()); + master_file.detector_layout(), master_file.pixels_x(), + master_file.pixels_y(), master_file.udp_interfaces_per_module(), + master_file.quad()); CHECK(geometry.modules_x() == test_parameters.modules_x); CHECK(geometry.modules_y() == test_parameters.modules_y); @@ -308,8 +517,8 @@ TEST_CASE("Open multi module file with ROI", RawFile f(fpath, "r"); SECTION("read 2 frames") { - REQUIRE(f.master().roi().value().width() == 256); - REQUIRE(f.master().roi().value().height() == 256); + REQUIRE(f.master().roi().width() == 256); + REQUIRE(f.master().roi().height() == 256); CHECK(f.n_modules() == 2); @@ -400,8 +609,299 @@ TEST_CASE("Read Mythenframe", "[.with-data][RawFile]") { auto fpath = test_data_path() / "raw/newmythen03/run_2_master_1.json"; REQUIRE(std::filesystem::exists(fpath)); RawFile f(fpath); - REQUIRE(f.master().roi().value().width() == 2560); - REQUIRE(f.master().roi().value().height() == 1); + REQUIRE(f.master().roi().width() == 2560); + REQUIRE(f.master().roi().height() == 1); auto frame = f.read_frame(); REQUIRE(frame.cols() == 2560); -} \ No newline at end of file +} + +TEST_CASE("Read Jungfrau frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled top port") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_interfaces_top_disabled_master_6.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 256); + + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + } + + SECTION("disabled bottom port") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_interfaces_bottom_port_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 256, 512}); + } + SECTION("2 modules - top ports disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_modules_2_interfaces_top_ports_disabled_master_2.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); // cannot read frame + // because multiple rois + + auto frame = f.read_rois(); + + REQUIRE(frame.size() == 2); + REQUIRE(frame[0].cols() == 1024); + REQUIRE(frame[0].rows() == 256); + REQUIRE(frame[1].cols() == 1024); + REQUIRE(frame[1].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 512, 768}); + } + SECTION("2 modules - top ports disabled - bottom port disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_modules_2_interfaces_disabled_ports_master_1.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 256, 768}); + } + SECTION("4 modules- mixed ports disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "4_modules_udp_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == + std::vector{1, 3, 4, 6}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); + auto frames = f.read_rois(); + REQUIRE(frames.size() == 4); + REQUIRE(frames[0].cols() == 1024); + REQUIRE(frames[0].rows() == 256); + REQUIRE(frames[1].cols() == 1024); + REQUIRE(frames[1].rows() == 256); + REQUIRE(frames[2].cols() == 1024); + REQUIRE(frames[2].rows() == 256); + REQUIRE(frames[3].cols() == 1024); + REQUIRE(frames[3].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 4); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 512, 768}); + REQUIRE(rois[2] == ROI{1024, 2048, 256, 512}); + REQUIRE(rois[3] == ROI{1024, 2048, 768, 1024}); + } +} + +TEST_CASE("Read Moench frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled top port") { + auto fpath = test_data_path() / "raw/moench" / + "2_interfaces_top_port_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 400); + REQUIRE(frame.rows() == 200); + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 400, 0, 200}); + } + + SECTION("disabled bottom port") { + auto fpath = test_data_path() / "raw/moench" / + "2_interfaces_bottom_port_disabled_master_6.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 400); + REQUIRE(frame.rows() == 200); + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 400, 200, 400}); + } +} + +TEST_CASE("Read Eiger frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled left port (bottom and top half module)") { + auto fpath = test_data_path() / "raw/eiger" / + "left_udp_port_disabled_master_2.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0, 2}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + auto frame = f.read_frame(); + + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 512); + + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{512, 1024, 0, 512}); + } + SECTION("disabled right port") { + auto fpath = test_data_path() / "raw/eiger" / + "right_udp_port_disabled_master_3.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 512, 0, 512}); + } + SECTION("2 full modules stacked vertically - right ports disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "2_modules_eiger_disabled_udp_port_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == + std::vector{1, 3, 5, 7}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); + auto frames = f.read_rois(); + REQUIRE(frames.size() == 2); + REQUIRE(frames[0].cols() == 512); + REQUIRE(frames[0].rows() == 512); + REQUIRE(frames[1].cols() == 512); + REQUIRE(frames[1].rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + REQUIRE(rois[0] == ROI{0, 512, 0, 512}); + REQUIRE(rois[1] == ROI{1024, 1536, 0, 512}); + } + SECTION("quad module - bottom port disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "quad_eiger_disabled_bottom_port_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 512, 256, 512}); + } + SECTION("only bottom left port disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "one_udp_port_disabled_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() instead")); + + auto frame = f.read_rois(); + REQUIRE(frame.size() == 2); + REQUIRE(frame[0].cols() == 512); + REQUIRE(frame[0].rows() == 256); + REQUIRE(frame[1].cols() == 1024); + REQUIRE(frame[1].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + + REQUIRE(rois[0] == ROI{512, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 256, 512}); + } + SECTION("No udp ports disabled") { + auto fpath = test_data_path() / + "raw/eiger_virtual_500k_disabled_ports" / + "all_active_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + + auto disabled_udp_ports = f.master().disabled_udp_ports(); + REQUIRE(disabled_udp_ports.empty()); + + auto udp_port_types = f.master().udp_port_types(); + REQUIRE(udp_port_types.has_value()); + REQUIRE(udp_port_types.value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + REQUIRE(f.total_frames() == 5); + + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 512); + + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 512}); + } +} diff --git a/src/RawMasterFile.cpp b/src/RawMasterFile.cpp index 9eff0d26..e0880ba5 100644 --- a/src/RawMasterFile.cpp +++ b/src/RawMasterFile.cpp @@ -4,7 +4,7 @@ #include "aare/logger.hpp" #include -#include "to_string.hpp" +#include "aare/to_string.hpp" namespace aare { @@ -115,6 +115,32 @@ RawMasterFile::RawMasterFile(const std::filesystem::path &fpath) } else { throw std::runtime_error(LOCATION + "Unsupported file type"); } + + m_geometry = DetectorGeometry(m_detector_layout, m_pixels_x, m_pixels_y, + m_udp_interfaces_per_module, m_quad); + + if (m_quad == 1 && m_udp_port_types.has_value()) { + m_udp_port_types.value() = {UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}; + } + + if (!m_disabled_udp_ports.empty()) { + // ROI takes precedence over disabled UDP ports, if both are defined in + // the master file + if (!complete_ROI(m_rois, m_geometry)) { + LOG(logWARNING) + << "ROI and disabled UDP ports defined in master file. ROI " + "will be used and disabled UDP ports will be ignored."; + } else { + update_rois_from_disabled_udp_ports(); + } + } + + m_ROI_geometries.reserve(m_rois.size()); + + for (const auto &roi : m_rois) { + m_ROI_geometries.push_back(ROIGeometry(roi, m_geometry)); + } } RawMasterFile::RawMasterFile(std::istream &is, const std::string &fname) @@ -164,10 +190,16 @@ std::optional RawMasterFile::counter_mask() const { return m_counter_mask; } -xy RawMasterFile::geometry() const { return m_geometry; } +xy RawMasterFile::detector_layout() const { return m_detector_layout; } + +const DetectorGeometry &RawMasterFile::geometry() const { return m_geometry; } + +const std::vector &RawMasterFile::roi_geometries() const { + return m_ROI_geometries; +} size_t RawMasterFile::n_modules() const { - return m_geometry.row * m_geometry.col; + return m_detector_layout.row * m_detector_layout.col; } xy RawMasterFile::udp_interfaces_per_module() const { @@ -193,26 +225,30 @@ ScanParameters RawMasterFile::scan_parameters() const { return m_scan_parameters; } -std::optional RawMasterFile::roi() const { - if (!m_rois) { - return std::nullopt; - } +std::optional> +RawMasterFile::udp_port_types() const { + return m_udp_port_types; +} - if (m_rois->empty()) { +std::vector RawMasterFile::disabled_udp_ports() const { + return m_disabled_udp_ports; +} + +ROI RawMasterFile::roi() const { + + if (m_rois.empty()) { throw std::runtime_error(LOCATION + "Zero ROIs in metadata."); } - if (m_rois.value().size() > 1) { + if (m_rois.size() > 1) { throw std::runtime_error(LOCATION + "Multiple ROIs present, use rois() method."); } else { - return m_rois.has_value() - ? std::optional(m_rois.value().at(0)) - : std::nullopt; // TODO: maybe throw if no roi exists + return m_rois.at(0); } } -std::optional> RawMasterFile::rois() const { return m_rois; } +std::vector RawMasterFile::rois() const { return m_rois; } ReadoutMode RawMasterFile::get_reading_mode() const { @@ -248,10 +284,11 @@ void RawMasterFile::parse_json(std::istream &is) { m_type = string_to(j["Detector Type"].get()); m_timing_mode = string_to(j["Timing Mode"].get()); - m_geometry = {j["Geometry"]["y"], - j["Geometry"]["x"]}; // TODO: isnt it only available for - // version > 7.1? - // - try block default should be 1x1 + m_detector_layout = { + j["Geometry"]["y"], + j["Geometry"]["x"]}; // TODO: isnt it only available for + // version > 7.1? + // - try block default should be 1x1 m_image_size_in_bytes = v < 8.0 ? j["Image Size in bytes"] : j["Image Size"]; @@ -301,6 +338,9 @@ void RawMasterFile::parse_json(std::istream &is) { if (j.contains("Number of rows") && j["Number of rows"].is_number()) { m_number_of_rows = j["Number of rows"]; } + if (j.contains("Number of Rows") && j["Number of Rows"].is_number()) { + m_number_of_rows = j["Number of Rows"]; // changed at some point + } // ---------------------------------------------------------------- // Special treatment of analog flag because of Moench03. @@ -377,7 +417,26 @@ void RawMasterFile::parse_json(std::istream &is) { } catch (const json::out_of_range &e) { // not a scan } - + try { + auto json_list_obj = j.at("UDP Ports Type"); + m_udp_port_types.emplace(); + for (auto &elem : json_list_obj) { + m_udp_port_types.value().push_back( + string_to(elem)); + } + } catch (const json::out_of_range &e) { + // leave the optional empty + } + try { + auto json_list_obj = j.at("UDP Ports Disabled"); + m_disabled_udp_ports.clear(); + for (auto &elem : json_list_obj) { + m_disabled_udp_ports.push_back(static_cast(elem)); + } + } catch (const json::out_of_range &e) { + m_disabled_udp_ports + .clear(); // empty list if not present in master file + } try { m_udp_interfaces_per_module = {j.at("Number of UDP Interfaces"), 1}; } catch (const json::out_of_range &e) { @@ -397,14 +456,21 @@ void RawMasterFile::parse_json(std::istream &is) { obj.at("ymin") = 0; obj.at("ymax") = 0; } - m_rois.emplace(); - m_rois.value().push_back(ROI{ - obj.at("xmin"), static_cast(obj.at("xmax")) + 1, - obj.at("ymin"), static_cast(obj.at("ymax")) + 1}); + m_rois.push_back({static_cast(obj.at("xmin")), + static_cast(obj.at("xmax")) + 1, + static_cast(obj.at("ymin")), + static_cast(obj.at("ymax")) + 1}); + } else { + // fill ROI with full detector size if not present in master + // file + m_rois.push_back( + {0, + m_detector_layout.col * static_cast(m_pixels_x), + 0, + m_detector_layout.row * static_cast(m_pixels_y)}); } } else { auto obj = j.at("Receiver Rois"); - m_rois.emplace(); for (auto &elem : obj) { // handle Mythen if (elem.at("ymin") == -1 && elem.at("ymax") == -1) { @@ -412,15 +478,18 @@ void RawMasterFile::parse_json(std::istream &is) { elem.at("ymax") = 0; } - m_rois.value().push_back(ROI{ - elem.at("xmin"), static_cast(elem.at("xmax")) + 1, - elem.at("ymin"), - static_cast(elem.at("ymax")) + 1}); + m_rois.push_back({static_cast(elem.at("xmin")), + static_cast(elem.at("xmax")) + 1, + static_cast(elem.at("ymin")), + static_cast(elem.at("ymax")) + 1}); } } } catch (const json::out_of_range &e) { - // leave the optional empty + // fill ROI with full detector size if not present in master file + m_rois.push_back( + {0, m_detector_layout.col * static_cast(m_pixels_x), 0, + m_detector_layout.row * static_cast(m_pixels_y)}); } if (j.contains("Counter Mask")) { @@ -477,9 +546,8 @@ void RawMasterFile::parse_raw(std::istream &is) { m_image_size_in_bytes = std::stoi(value); } else if (key == "Frame Padding") { m_frame_padding = std::stoi(value); - // } else if (key == "Frame Discard Policy"){ - // m_frame_discard_policy = - // StringTo(value); + } else if (key == "Frame Discard Policy") { + m_frame_discard_policy = string_to(value); // } else if (key == "Number of rows"){ // m_number_of_rows = std::stoi(value); } else if (key == "Analog Flag") { @@ -528,7 +596,7 @@ void RawMasterFile::parse_raw(std::istream &is) { m_max_frames_per_file = std::stoi(value); } else if (key == "Geometry") { pos = value.find(','); - m_geometry = { + m_detector_layout = { static_cast(std::stoi(value.substr(1, pos))), static_cast(std::stoi(value.substr(pos + 1)))}; } else if (key == "Number of UDP Interfaces") { @@ -548,13 +616,17 @@ void RawMasterFile::parse_raw(std::istream &is) { m_type = DetectorType::Moench03_old; } - if (m_geometry.col == 0 && m_geometry.row == 0) { + if (m_detector_layout.col == 0 && m_detector_layout.row == 0) { retrieve_geometry(); LOG(TLogLevel::logWARNING) << "No geometry found in master file. Retrieved geometry of " - << m_geometry.row << " x " << m_geometry.col << "\n "; + << m_detector_layout.row << " x " << m_detector_layout.col << "\n "; } + m_rois.push_back( + {0, m_detector_layout.col * static_cast(m_pixels_x), 0, + m_detector_layout.row * static_cast(m_pixels_y)}); + // TODO! Read files and find actual frames if (m_frames_in_file == 0) m_frames_in_file = m_total_frames_expected; @@ -578,7 +650,153 @@ void RawMasterFile::retrieve_geometry() { ++rows; ++cols; - m_geometry = {rows, cols}; + m_detector_layout = {rows, cols}; +} + +/** + * @brief Update ROIs from disabled UDP ports + */ +void RawMasterFile::update_rois_from_disabled_udp_ports() { + + const size_t num_udp_port_types = m_udp_port_types.value().size(); + + size_t first_port = m_disabled_udp_ports[0] % num_udp_port_types; + + bool all_ports_equal = + std::all_of(m_disabled_udp_ports.begin(), m_disabled_udp_ports.end(), + [&num_udp_port_types, first_port](size_t &port) { + return port % num_udp_port_types == first_port; + }); + + m_rois.clear(); // remove global roi + + const ssize_t udp_ports_per_module = + m_geometry.udp_interfaces_per_module().col * + m_geometry.udp_interfaces_per_module().row; + + bool port_disabled_for_all_modules = + m_disabled_udp_ports.size() == + m_geometry.modules_x() * m_geometry.modules_y() / udp_ports_per_module; + + if (all_ports_equal && port_disabled_for_all_modules) { + + if (m_udp_port_types.value()[first_port] == UDPPortPosition::LEFT) { + const size_t num_rois = + m_geometry.modules_x() / udp_ports_per_module; + m_rois.resize(num_rois); + + const ssize_t pixels_per_module_x = + m_geometry.pixels_x() / m_geometry.modules_x(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_x, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{ + idx * udp_ports_per_module * pixels_per_module_x + + pixels_per_module_x, + idx * udp_ports_per_module * pixels_per_module_x + + 2 * pixels_per_module_x, + 0, static_cast(m_geometry.pixels_y())}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::RIGHT) { + const size_t num_rois = + m_geometry.modules_x() / udp_ports_per_module; + m_rois.resize(num_rois); + + const ssize_t pixels_per_module_x = + m_geometry.pixels_x() / m_geometry.modules_x(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_x, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{idx * udp_ports_per_module * pixels_per_module_x, + idx * udp_ports_per_module * + pixels_per_module_x + + pixels_per_module_x, + 0, static_cast(m_geometry.pixels_y())}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::TOP) { + size_t num_rois = m_geometry.modules_y() / udp_ports_per_module; + m_rois.resize(num_rois); + // assumes euclidean coordinate system with origin at bottom + // left corner of the detector + const ssize_t pixels_per_module_y = + m_geometry.pixels_y() / m_geometry.modules_y(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_y, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{0, static_cast(m_geometry.pixels_x()), + idx * udp_ports_per_module * pixels_per_module_y, + idx * udp_ports_per_module * + pixels_per_module_y + + pixels_per_module_y}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::BOTTOM) { + size_t num_rois = m_geometry.modules_y() / udp_ports_per_module; + m_rois.resize(num_rois); + const ssize_t pixels_per_module_y = + m_geometry.pixels_y() / m_geometry.modules_y(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_y, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{ + 0, static_cast(m_geometry.pixels_x()), + idx * udp_ports_per_module * pixels_per_module_y + + pixels_per_module_y, + idx * udp_ports_per_module * pixels_per_module_y + + 2 * pixels_per_module_y}; + }); + } + } else { + // iterate over all ports and create ROIs for each disabled port + LOG(logDEBUG) << "Creating ROIs from disabled UDP ports"; + // get the enabled ones: + std::vector enabled_ports(m_geometry.n_modules()); + std::iota(enabled_ports.begin(), enabled_ports.end(), 0); + std::for_each(m_disabled_udp_ports.begin(), m_disabled_udp_ports.end(), + [&enabled_ports](size_t &port) { + enabled_ports.erase(std::remove(enabled_ports.begin(), + enabled_ports.end(), + port), + enabled_ports.end()); + }); + + m_rois.reserve(enabled_ports.size()); + for (const auto enabled_port : enabled_ports) { + + auto module_geometry = + m_geometry.get_module_geometries(enabled_port); + m_rois.push_back( + ROI{module_geometry.origin_x, + module_geometry.origin_x + module_geometry.width, + module_geometry.origin_y, + module_geometry.origin_y + module_geometry.height}); + } + + if (m_udp_port_types == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}) { + m_rois = merge_consecutive_rois(m_rois); + } else if (m_udp_port_types == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP} || + m_udp_port_types == + std::vector{UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}) { + m_rois = merge_consecutive_rois(m_rois); + } else { + throw std::runtime_error(LOCATION + "Unsupported UDP port types"); + } + } } } // namespace aare diff --git a/src/RawMasterFile.test.cpp b/src/RawMasterFile.test.cpp index 59efed71..3838bfc2 100644 --- a/src/RawMasterFile.test.cpp +++ b/src/RawMasterFile.test.cpp @@ -3,11 +3,38 @@ #include "test_config.hpp" #include +#include #include #include using namespace aare; +TEST_CASE("Parse legacy master frame discard policies", + "[rawmasterfile][frame-policy]") { + const auto [policy, expected] = + GENERATE(table({ + {"nodiscard", FrameDiscardPolicy::NoDiscard}, + {"discard", FrameDiscardPolicy::Discard}, + {"discardpartial", FrameDiscardPolicy::DiscardPartial}, + {"", FrameDiscardPolicy::NoDiscard}, + })); + CAPTURE(policy); + std::ostringstream content; + content << "Version : 6.4\n" + "Detector Type : Jungfrau\n" + "Timing Mode : auto\n" + "Geometry : [1, 1]\n" + "Pixels : [3, 2]\n" + "Frame Padding : 0\n"; + if (!policy.empty()) { + content << "Frame Discard Policy : " << policy << '\n'; + } + std::istringstream input(content.str()); + RawMasterFile master(input, "run_master_0.raw"); + REQUIRE(master.frame_padding() == 0); + REQUIRE(master.frame_discard_policy() == expected); +} + TEST_CASE("Parse a master file fname") { RawFileNameComponents m("test_master_1.json"); REQUIRE(m.base_name() == "test"); @@ -68,7 +95,7 @@ TEST_CASE("A disabled scan") { REQUIRE(s.step() == 0); } -TEST_CASE("Parse a master file in .json format", "[.integration]") { +TEST_CASE("Parse a master file in .json format", "[.with-data]") { auto fpath = test_data_path() / "raw" / "jungfrau" / "jungfrau_single_master_0.json"; REQUIRE(std::filesystem::exists(fpath)); @@ -80,14 +107,14 @@ TEST_CASE("Parse a master file in .json format", "[.integration]") { // "Detector Type": "Jungfrau", REQUIRE(f.detector_type() == DetectorType::Jungfrau); // "Timing Mode": "auto", - REQUIRE(f.timing_mode() == TimingMode::Auto); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); // "Geometry": { // "x": 1, // "y": 1 // }, - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); // "Image Size in bytes": 1048576, REQUIRE(f.image_size_in_bytes() == 1048576); @@ -151,7 +178,7 @@ TEST_CASE("Parse a master file in .json format", "[.integration]") { } TEST_CASE("Parse a master file in old .raw format", - "[.integration][.with-data][.rawmasterfile]") { + "[.with-data][rawmasterfile]") { auto fpath = test_data_path() / "raw/jungfrau_2modules_version6.1.2/run_master_0.raw"; REQUIRE(std::filesystem::exists(fpath)); @@ -159,11 +186,11 @@ TEST_CASE("Parse a master file in old .raw format", CHECK(f.udp_interfaces_per_module() == xy{1, 1}); CHECK(f.n_modules() == 2); - CHECK(f.geometry().row == 2); - CHECK(f.geometry().col == 1); + CHECK(f.detector_layout().row == 2); + CHECK(f.detector_layout().col == 1); } -TEST_CASE("Parse a master file in .raw format", "[.integration]") { +TEST_CASE("Parse a master file in .raw format", "[.with-data]") { auto fpath = test_data_path() / @@ -179,10 +206,10 @@ TEST_CASE("Parse a master file in .raw format", "[.integration]") { // Detector Type : ChipTestBoard REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); // Timing Mode : auto - REQUIRE(f.timing_mode() == TimingMode::Auto); - // Geometry : [1, 1] - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + // Detector Layout : [1, 1] + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); // Image Size : 360000 bytes REQUIRE(f.image_size_in_bytes() == 360000); // Pixels : [96, 1] @@ -232,8 +259,7 @@ TEST_CASE("Parse a master file in .raw format", "[.integration]") { // Packets Caught Mask : 64 bytes } -TEST_CASE("Parse a master file in new .json format", - "[.integration][.with-data]") { +TEST_CASE("Parse a master file in new .json format", "[.with-data]") { auto file_path = test_data_path() / "raw" / "newmythen03" / "run_87_master_0.json"; @@ -246,10 +272,10 @@ TEST_CASE("Parse a master file in new .json format", REQUIRE(f.detector_type() == DetectorType::Mythen3); // Timing Mode : auto - REQUIRE(f.timing_mode() == TimingMode::Auto); - // Geometry : [2, 1] - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + // Detector Layout : [2, 1] + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 1); // Image Size : 5120 bytes REQUIRE(f.image_size_in_bytes() == 5120); @@ -260,14 +286,14 @@ TEST_CASE("Parse a master file in new .json format", REQUIRE(f.scan_parameters().step() == 0); REQUIRE(f.scan_parameters().settleTime() == 0); - auto roi = f.roi().value(); + auto roi = f.roi(); REQUIRE(roi.xmin == 0); REQUIRE(roi.xmax == 2560); REQUIRE(roi.ymin == 0); REQUIRE(roi.ymax == 1); } -TEST_CASE("Read eiger master file", "[.integration]") { +TEST_CASE("Read eiger master file", "[.with-data]") { auto fpath = test_data_path() / "raw/eiger/eiger_500k_32bit_master_0.json"; REQUIRE(std::filesystem::exists(fpath)); RawMasterFile f(fpath); @@ -279,7 +305,7 @@ TEST_CASE("Read eiger master file", "[.integration]") { // "Detector Type": "Eiger", REQUIRE(f.detector_type() == DetectorType::Eiger); // "Timing Mode": "auto", - REQUIRE(f.timing_mode() == TimingMode::Auto); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); // "Geometry": { // "x": 2, // "y": 2 @@ -399,9 +425,9 @@ TEST_CASE("Parse EIGER 7.2 master from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Eiger); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 2); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 2); REQUIRE(f.image_size_in_bytes() == 524288); REQUIRE(f.pixels_x() == 512); @@ -476,9 +502,9 @@ TEST_CASE("Parse JUNGFRAU 7.2 master from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Jungfrau); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 2); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 2); REQUIRE(f.n_modules() == 2); REQUIRE(f.image_size_in_bytes() == 524288); REQUIRE(f.pixels_x() == 1024); @@ -559,8 +585,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 192000); REQUIRE(f.pixels_x() == 32); REQUIRE(f.pixels_y() == 1); @@ -637,8 +663,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 16000); REQUIRE(f.pixels_x() == 64); REQUIRE(f.pixels_y() == 1); @@ -709,8 +735,8 @@ TEST_CASE("Parse Moench 7.2 master (SW 7.0.3) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Moench03_old); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -780,8 +806,8 @@ TEST_CASE("Parse Moench 7.2 master (SW 8.0.0) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Moench03); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -862,8 +888,8 @@ TEST_CASE("Parse CTB 7.2 master (SW 8.0.0) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 192000); REQUIRE(f.pixels_x() == 32); REQUIRE(f.pixels_y() == 1); @@ -943,8 +969,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 16000); REQUIRE(f.pixels_x() == 64); REQUIRE(f.pixels_y() == 1); @@ -1010,9 +1036,9 @@ TEST_CASE("Parse a CTB file from stream") { REQUIRE(f.version() == "8.0"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 18432); REQUIRE(f.pixels_x() == 2); REQUIRE(f.pixels_y() == 1); @@ -1100,9 +1126,9 @@ TEST_CASE("Parse v8.0 MYTHEN3 from stream") { REQUIRE(f.version() == "8.0"); REQUIRE(f.detector_type() == DetectorType::Mythen3); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 5120); REQUIRE(f.pixels_x() == 1280); REQUIRE(f.pixels_y() == 1); @@ -1185,9 +1211,9 @@ TEST_CASE("Parse a v7.1 Mythen3 from stream") { REQUIRE(f.version() == "7.1"); REQUIRE(f.detector_type() == DetectorType::Mythen3); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 15360); REQUIRE(f.pixels_x() == 3840); REQUIRE(f.pixels_y() == 1); @@ -1267,9 +1293,9 @@ TEST_CASE("Parse old Moench03 from stream") { REQUIRE(f.version() == "7.1"); REQUIRE(f.detector_type() == DetectorType::Moench03_old); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -1286,3 +1312,111 @@ TEST_CASE("Parse old Moench03 from stream") { REQUIRE(f.period() == std::chrono::microseconds(600)); REQUIRE(f.analog_samples() == 5000); } + +TEST_CASE("Parse Eiger json v8.1 all ports active") { + std::string master_content = R"( + { + "Version": 8.1, + "Timestamp": "Mon Aug 10 17:44:46 2026", + "Detector Type": "Eiger", + "Timing Mode": "auto", + "Geometry": { + "x": 2, + "y": 2 + }, + "Image Size": 262144, + "Pixels": { + "x": 512, + "y": 256 + }, + "Max Frames Per File": 10000, + "Frame Discard Policy": "nodiscard", + "Frame Padding": 1, + "Scan Parameters": { + "enable": 0, + "dacInd": 0, + "start offset": 0, + "stop offset": 0, + "step size": 0, + "dac settle time ns": 0 + }, + "Total Frames": 5, + "Receiver Rois": [ + { + "xmin": 0, + "xmax": 1023, + "ymin": 0, + "ymax": 511 + } + ], + "Dynamic Range": 16, + "Ten Giga": 0, + "Exposure Time": "1s", + "Acquisition Period": "1s", + "Threshold Energy": -1, + "Sub Exposure Time": "2.62144ms", + "Sub Acquisition Period": "2.62144ms", + "Quad": 0, + "UDP Ports Type": [ + "left", + "right" + ], + "UDP Ports Disabled": [], + "Number of Rows": 256, + "Rate Corrections": [ + 0, + 0 + ], + "Readout Speed": "full_speed", + "Frames in File": 5, + "Additional JSON Header": {} +} + +)"; + + std::istringstream iss(master_content); + RawMasterFile f(iss, "test_master_0.json"); + + REQUIRE(f.version() == "8.1"); + REQUIRE(f.detector_type() == DetectorType::Eiger); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{2, 2}); + REQUIRE(f.n_modules() == 4); + REQUIRE(f.image_size_in_bytes() == 262144); + REQUIRE(f.pixels_x() == 512); + REQUIRE(f.pixels_y() == 256); + REQUIRE(f.max_frames_per_file() == 10000); + REQUIRE(f.frame_discard_policy() == FrameDiscardPolicy::NoDiscard); + REQUIRE(f.frame_padding() == 1); + REQUIRE(f.total_frames_expected() == 5); + REQUIRE(f.frames_in_file() == 5); + REQUIRE(f.bitdepth() == 16); + REQUIRE(f.exptime() == std::chrono::seconds(1)); + REQUIRE(f.period() == std::chrono::seconds(1)); + REQUIRE(f.quad() == 0); + // rows return a std::optional, so we need to check if it has a value before + // using it + auto rows = f.number_of_rows(); + REQUIRE(rows.has_value()); + REQUIRE(rows.value() == 256); + REQUIRE(f.udp_interfaces_per_module() == xy{1, 2}); + + auto scan_parameters = f.scan_parameters(); + REQUIRE_FALSE(scan_parameters.enabled()); + REQUIRE(scan_parameters.dac() == DACIndex::DAC_0); + REQUIRE(scan_parameters.start() == 0); + REQUIRE(scan_parameters.stop() == 0); + REQUIRE(scan_parameters.step() == 0); + REQUIRE(scan_parameters.settleTime() == 0); + + REQUIRE(f.udp_port_types().has_value()); + REQUIRE(f.udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + REQUIRE(f.disabled_udp_ports().empty()); + + auto rois = f.rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 512}); +} diff --git a/src/RawSubFile.cpp b/src/RawSubFile.cpp index ed60a4a5..7a42bbba 100644 --- a/src/RawSubFile.cpp +++ b/src/RawSubFile.cpp @@ -12,6 +12,17 @@ namespace aare { +std::runtime_error RawSubFile::frame_error(size_t frame_index, + const std::string &message) const { + return std::runtime_error( + fmt::format("Error reading frame index {} from file '{}': {}", + frame_index, current_path().string(), message)); +} + +std::filesystem::path RawSubFile::current_path() const { + return fpath(m_current_file_index + m_offset); +} + RawSubFile::RawSubFile(const std::filesystem::path &fname, DetectorType detector, size_t rows, size_t cols, size_t bitdepth, uint32_t pos_row, uint32_t pos_col) @@ -34,8 +45,9 @@ RawSubFile::RawSubFile(const std::filesystem::path &fname, void RawSubFile::seek(size_t frame_index) { LOG(logDEBUG) << "RawSubFile::seek(" << frame_index << ")"; if (frame_index >= m_total_frames) { - throw std::runtime_error(LOCATION + " Frame index out of range: " + - std::to_string(frame_index)); + throw frame_error( + frame_index, + fmt::format("out of range (available frames: {})", m_total_frames)); } m_current_frame_index = frame_index; auto file_index = first_larger(m_last_frame_in_file, frame_index); @@ -49,6 +61,9 @@ void RawSubFile::seek(size_t frame_index) { auto byte_offset = frame_offset * (m_bytes_per_frame + sizeof(DetectorHeader)); m_file.seekg(byte_offset); + if (m_file.fail()) { + throw frame_error(frame_index, ifstream_error_msg(m_file)); + } } size_t RawSubFile::tell() { @@ -66,7 +81,7 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } if (m_file.fail()) { - throw std::runtime_error(LOCATION + ifstream_error_msg(m_file)); + throw frame_error(m_current_frame_index, ifstream_error_msg(m_file)); } // TODO! expand support for different bitdepths @@ -81,8 +96,8 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } else if (m_bitdepth == 32) { read_with_map(image_buf); } else { - throw std::runtime_error( - "Unsupported bitdepth for read with pixel map"); + throw frame_error(m_current_frame_index, + "Unsupported bitdepth for read with pixel map"); } } else { @@ -91,7 +106,7 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } if (m_file.fail()) { - throw std::runtime_error(LOCATION + ifstream_error_msg(m_file)); + throw frame_error(m_current_frame_index, ifstream_error_msg(m_file)); } ++m_current_frame_index; @@ -135,6 +150,9 @@ size_t RawSubFile::frame_number(size_t frame_index) { seek(frame_index); DetectorHeader h{}; m_file.read(reinterpret_cast(&h), sizeof(DetectorHeader)); + if (m_file.fail()) { + throw frame_error(frame_index, ifstream_error_msg(m_file)); + } return h.frameNumber; } @@ -178,8 +196,8 @@ void RawSubFile::open_file(size_t file_index) { m_file.open(fname, std::ios::binary); if (!m_file.is_open()) { throw std::runtime_error( - LOCATION + - fmt::format("Could not open file {}", fpath(file_index).string())); + fmt::format("Could not open file '{}' for frame index {}", + fname.string(), m_current_frame_index)); } m_current_file_index = file_index; } @@ -208,4 +226,4 @@ void RawSubFile::scan_files() { } } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/src/RawSubFile.test.cpp b/src/RawSubFile.test.cpp index 9550ed17..e09dff7e 100644 --- a/src/RawSubFile.test.cpp +++ b/src/RawSubFile.test.cpp @@ -2,11 +2,70 @@ #include "aare/RawSubFile.hpp" #include "aare/File.hpp" #include "aare/NDArray.hpp" +#include "raw_file_helpers.hpp" #include "test_config.hpp" #include +#include +#include using namespace aare; +TEST_CASE("RawSubFile errors identify the frame and actual data path", + "[RawSubFile][read-errors]") { + TemporaryRawFiles files; + const size_t offset = GENERATE(0, 5); + if (offset != 0) { + for (size_t index = 0; index < 2; ++index) { + std::filesystem::rename(files.data_path(0, index), + files.data_path(0, offset + index)); + } + } + RawSubFile file(files.data_path(0, offset), DetectorType::Jungfrau, 2, 3, + 16); + CHECK(file.current_path() == files.data_path(0, offset)); + std::vector buffer(file.bytes_per_frame() * 3); + DetectorHeader header{}; + const auto second_path = files.data_path(0, offset + 1); + const auto index_matcher = + Catch::Matchers::ContainsSubstring("frame index 1"); + const auto path_matcher = + Catch::Matchers::ContainsSubstring(second_path.string()); + const auto context = index_matcher && path_matcher; + + SECTION("truncated header") { + std::filesystem::resize_file(second_path, 0); + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), &header), context); + } + SECTION("truncated pixels") { + std::filesystem::resize_file(second_path, sizeof(DetectorHeader)); + file.seek(1); + CHECK(file.current_path() == second_path); + REQUIRE_THROWS_WITH(file.read_into(buffer.data()), context); + } + SECTION("frame number from truncated header") { + std::filesystem::resize_file(second_path, 0); + REQUIRE_THROWS_WITH(file.frame_number(1), context); + } + SECTION("missing next file") { + std::filesystem::remove(second_path); + REQUIRE_THROWS_WITH(file.seek(1), context); + } + SECTION("batch beyond EOF") { + REQUIRE_THROWS_WITH( + file.read_into(buffer.data(), size_t{3}), + Catch::Matchers::ContainsSubstring("frame index 2") && + Catch::Matchers::ContainsSubstring(second_path.string())); + } + SECTION("seek beyond EOF") { + REQUIRE_THROWS_WITH( + file.seek(17), + Catch::Matchers::ContainsSubstring("frame index 17") && + Catch::Matchers::ContainsSubstring( + files.data_path(0, offset).string())); + } +} + TEST_CASE("Read frames directly from a RawSubFile", "[.with-data]") { auto fpath_raw = test_data_path() / "raw/jungfrau" / "jungfrau_single_d0_f0_0.raw"; @@ -77,4 +136,4 @@ TEST_CASE("Read frames directly from a RawSubFile starting at the second file", auto npy_frame = npy.read_frame(); CHECK((image.view() == npy_frame.view())); } -} \ No newline at end of file +} diff --git a/src/algorithm.test.cpp b/src/algorithm.test.cpp index daa0616d..e498c3b2 100644 --- a/src/algorithm.test.cpp +++ b/src/algorithm.test.cpp @@ -231,4 +231,4 @@ TEST_CASE("Bilinear interpolation", "[algorithm]") { 0.75); REQUIRE(interpolated_value == 5.25); } -} +} \ No newline at end of file diff --git a/src/to_string.cpp b/src/to_string.cpp index d950b13c..8776c501 100644 --- a/src/to_string.cpp +++ b/src/to_string.cpp @@ -1,4 +1,4 @@ -#include "to_string.hpp" +#include "aare/to_string.hpp" namespace aare { @@ -35,9 +35,15 @@ template <> DetectorType string_to(const std::string &arg) { template <> TimingMode string_to(const std::string &arg) { if (arg == "auto") - return TimingMode::Auto; + return TimingMode::AUTO_TIMING; if (arg == "trigger") - return TimingMode::Trigger; + return TimingMode::TRIGGER_EXPOSURE; + if (arg == "gating") + return TimingMode::GATED; + if (arg == "burst_trigger") + return TimingMode::BURST_TRIGGER; + if (arg == "trigger_gating") + return TimingMode::TRIGGER_GATED; throw std::runtime_error("Could not decode timing mode from: \"" + arg + "\""); } @@ -235,6 +241,34 @@ template <> DACIndex string_to(const std::string &arg) { "\""); } +template <> UDPPortPosition string_to(const std::string &arg) { + if (arg == "left") + return UDPPortPosition::LEFT; + if (arg == "right") + return UDPPortPosition::RIGHT; + if (arg == "top") + return UDPPortPosition::TOP; + if (arg == "bottom") + return UDPPortPosition::BOTTOM; + throw std::runtime_error("Could not decode UDPPortPosition from: \"" + arg + + "\""); +} + +std::string to_string(const corner &arg) { + switch (arg) { + case corner::cTopLeft: + return "TopLeft"; + case corner::cTopRight: + return "TopRight"; + case corner::cBottomLeft: + return "BottomLeft"; + case corner::cBottomRight: + return "BottomRight"; + default: + throw std::runtime_error("Invalid corner value"); + } +} + std::string remove_unit(std::string &str) { auto it = str.begin(); while (it != str.end()) { diff --git a/src/to_string.test.cpp b/src/to_string.test.cpp index 2385700a..1489632f 100644 --- a/src/to_string.test.cpp +++ b/src/to_string.test.cpp @@ -1,10 +1,10 @@ // SPDX-License-Identifier: MPL-2.0 +#include "aare/to_string.hpp" #include "aare/defs.hpp" #include #include -#include "to_string.hpp" using aare::string_to; TEST_CASE("DetectorType string to enum") { @@ -36,9 +36,10 @@ TEST_CASE("DetectorType string to enum") { } TEST_CASE("TimingMode string to enum") { - REQUIRE(string_to("auto") == aare::TimingMode::Auto); + REQUIRE(string_to("auto") == + aare::TimingMode::AUTO_TIMING); REQUIRE(string_to("trigger") == - aare::TimingMode::Trigger); + aare::TimingMode::TRIGGER_EXPOSURE); REQUIRE_THROWS(string_to("invalid_mode")); } diff --git a/tests/raw_file_helpers.hpp b/tests/raw_file_helpers.hpp new file mode 100644 index 00000000..b1a9b17c --- /dev/null +++ b/tests/raw_file_helpers.hpp @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "aare/defs.hpp" + +#include +#include +#include +#include +#include + +class TemporaryRawFiles { + std::filesystem::path m_directory; + + public: + explicit TemporaryRawFiles(size_t modules = 1) { + const auto unique = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_directory = std::filesystem::temp_directory_path() / + ("aare-raw-" + std::to_string(unique)); + std::filesystem::create_directory(m_directory); + const nlohmann::json metadata = { + {"Version", 7.2}, + {"Detector Type", "Jungfrau"}, + {"Timing Mode", "auto"}, + {"Geometry", {{"x", 1}, {"y", modules}}}, + {"Image Size in bytes", 12}, + {"Pixels", {{"x", 3}, {"y", 2}}}, + {"Max Frames Per File", 1}, + {"Total Frames", 2}, + {"Frames in File", 2}, + {"Frame Padding", 1}, + {"Frame Discard Policy", "nodiscard"}}; + std::ofstream(master_path()) << metadata; + for (size_t module = 0; module < modules; ++module) { + for (size_t file = 0; file < 2; ++file) { + std::ofstream output(data_path(module, file), std::ios::binary); + aare::DetectorHeader header{}; + header.frameNumber = 100 + file; + const uint16_t pixels[6] = {1, 2, 3, 4, 5, 6}; + output.write(reinterpret_cast(&header), + sizeof(header)); + output.write(reinterpret_cast(pixels), + sizeof(pixels)); + } + } + } + + TemporaryRawFiles(const TemporaryRawFiles &) = delete; + TemporaryRawFiles &operator=(const TemporaryRawFiles &) = delete; + ~TemporaryRawFiles() { std::filesystem::remove_all(m_directory); } + + std::filesystem::path master_path() const { + return m_directory / "run_master_0.json"; + } + + std::filesystem::path data_path(size_t module = 0, size_t file = 0) const { + return m_directory / ("run_d" + std::to_string(module) + "_f" + + std::to_string(file) + "_0.raw"); + } +};