diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..5646aaae --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,332 @@ +# Repository guidelines + +## Scope and project goals + +These instructions apply to the entire repository unless a more specific +`AGENTS.md` exists in a subdirectory. + +Aare is a data-analysis library for PSI hybrid detectors. The C++17 core is +the canonical implementation, while Python is the main user-facing interface +through pybind11. Changes should preserve the project's priorities: fast and +efficient processing, simple interfaces, API stability, and a small dependency +footprint. + +## Repository layout + +- `include/aare/` contains the public C++ API and header-defined templates. +- `src/` contains C++ implementations and colocated Catch2 tests named + `*.test.cpp`, grouped into `src/fit/`, `src/hist/` and `src/utils/` where + a component has several private files. +- `tests/` contains the C++ test executable, configuration, and shared helpers. +- `python/src/` contains pybind11 bindings and module registration. +- `python/aare/` contains Python facades, convenience APIs, and public exports. +- `python/tests/` contains the pytest suite. +- `docs/src/` contains Sphinx/reStructuredText documentation. Doxygen and + Breathe integrate the C++ API into the Sphinx site. +- `benchmarks/` contains Google Benchmark programs. +- `cmake/`, `conda-recipe/`, and `pyproject.toml` support builds and packaging. +- `RELEASE.md` contains pending and published release notes. + +## Build environment + +The project requires CMake 3.15 or newer, C++17 with compiler extensions +disabled, and Python 3.11 or newer. `etc/dev-env.yml` defines the Conda +development environment. + +CMake fetches several dependencies by default. Use +`-DAARE_SYSTEM_LIBRARIES=ON` only when all required system or Conda packages +are available. + +Use this configuration for normal development: + +```bash +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Debug \ + -DAARE_TESTS=ON \ + -DAARE_PYTHON_BINDINGS=ON +cmake --build build -j4 +``` + +Useful optional settings include `AARE_DOCS`, `AARE_BENCHMARKS`, `AARE_ASAN`, +`AARE_WARNINGS_AS_ERRORS`, `AARE_VERBOSE` (raises the compile-time log +level), and `AARE_TUNE_LOCAL` (`-march=native`, not portable). Reconfigure an +existing build directory instead of creating alternate in-tree build layouts +unless isolation is needed; `build/CMakeCache.txt` records the options and +interpreter it was configured with. + +`-Werror=return-type` is always enabled. New code should also compile cleanly +under the project's `-Wall -Wextra -pedantic -Wshadow -Wold-style-cast +-Wdouble-promotion` flags, which `AARE_WARNINGS_AS_ERRORS` turns into errors. + +The build copies `python/aare/*.py` and the compiled `_aare` extension into +`build/aare/`, so `PYTHONPATH=build` makes the package importable. The +extension only imports into the interpreter CMake was configured with, which +is normally the Conda or venv Python active at configure time. Activate that +environment, or resolve the interpreter from the cache: + +```bash +PY=$(grep -E '^_?Python_EXECUTABLE' build/CMakeCache.txt | head -1 | cut -d= -f2) +PYTHONPATH="$PWD/build" "$PY" -m pytest python/tests +``` + +Do not fall back to a different system Python. + +With `AARE_BENCHMARKS=ON`, Google Benchmark programs build as +`build/run_benchmarks` and `build/fit_benchmark`. + +## Architecture + +### Core containers + +- `NDArray` owns contiguous C-order memory; `NDView` is a + non-owning view with the same shape and stride API. Both derive from + `ArrayExpr` (expression templates for element-wise math). Shapes are + `std::array`. `NDView` converts to `NDView`. +- `Frame` is a byte buffer plus rows, cols, and `Dtype`, the lowest common + denominator returned by file readers. `Dtype` is a runtime type tag built + from a bitdepth, a NumPy descriptor, or `typeid`. +- `defs.hpp` holds detector enums (`DetectorType`, `TimingMode`, + `FrameDiscardPolicy`, `ReadoutMode`), the on-disk `DetectorHeader`, chip + constants (`Moench04`, `Matterhorn02`, ...), the `LOCATION` macro used in + error messages, and `AARE_ASSERT` (active only with `AARE_CUSTOM_ASSERT`). +- `logger.hpp` provides `LOG(logLEVEL) << ...`; levels above `AARE_LOG_LEVEL` + (set by CMake) are compiled out. + +### File reading + +`File` is the generic RAII facade. It holds a `unique_ptr` and +dispatches on file extension to one of the concrete readers: + +- `RawFile` reads slsDetector `.raw` data with a `_master_N.json` (or legacy + `.raw`) master file. It owns a `RawMasterFile` (master JSON, older raw-text + masters, scan parameters, ROIs, disabled UDP ports) and a two-dimensional + table of `RawSubFile` indexed `[roi][module]`. Each `RawSubFile` handles one + module's sequence of `_dN_fM_` data files and per-frame `DetectorHeader`s. + Module placement into the assembled image goes through `DetectorGeometry`, + `ModuleGeometry`, and `ROIGeometry` (module gaps, ROIs, UDP-port halves). + Frame numbers from the detector are distinct from frame indices in the + file; the API uses both and names them accordingly. +- `NumpyFile` (`.npy`), `JungfrauDataFile` (legacy `.dat`), and `CtbRawFile` + (chip test board; raw ADC samples decoded through `decode.hpp` and the + `PixelMap` generators, exposed in Python as `aare.transform` callables). +- `experimental::MultiThreadedFileReader` opens one `File` per worker and + fills a caller-owned buffer in file order. + +Some chip-specific decoding (Moench03/05 pixel maps, ADC SAR transforms) +lives in `python/aare/transform.py` on top of C++ generators. + +### Pedestal and cluster finding + +- `Pedestal` and `FastPedestal` maintain per-pixel running mean and + standard deviation, always accumulating in `double`. `FastPedestal` must + see `min_pedestal_samples` frames before `ready()`. +- `ClusterFinder` is single-threaded: + push pedestal frames, then `find_clusters(view, frame_number)` appends to an + internal `ClusterVector` that callers take with `steal_clusters()`. + Thresholds are `n_sigma * std` and are recomputed by `update_threshold()`. +- `ClusterFinderMT` wraps N `ClusterFinder`s, one per thread, each with a + `CircularFifo` input queue backed by a preallocated `FramePool` + and a `ProducerConsumerQueue` output. A collector thread funnels all + outputs into a single sink queue. Consumers of that sink run their own + thread: `ClusterCollector` (accumulate in memory, `steal_clusters()`) or + `ClusterFileSink` (write the binary cluster file). The `stop()` and + `sync()` ordering matters; read the existing tests before changing + lifecycle code. +- `Cluster` is a POD (x, y, data[]) and + `ClusterVector>` is a frame-tagged `std::vector` of them. + `ClusterFile` reads and writes the binary format (`int32 frame_number`, + `uint32 n_clusters`, raw cluster bytes). The format carries no shape or + dtype metadata, so the caller must supply matching template arguments. + `CalculateEta.hpp` and `Interpolator` operate on `ClusterVector`s; + `algorithm.hpp` holds small 1-D search helpers. +- `hist/PixelHistogram` and `PedestalTrackingPixelHistogram` build per-pixel + histograms across a frame series, multithreaded by row bands. + +### Fitting + +`Fit.hpp`, `FitModel`, and `Models.hpp` (`Gaussian`, `Pol1`, `Pol2`, +`RisingScurve`, `FallingScurve`, `GaussianErfcPlateau`, +`GaussianChargeSharing`, ...) form the public fitting API. Each model +provides `eval`, `eval_and_grad`, `is_valid`, `estimate_par`, and +`param_info`; the separable models also declare their linear parameters +(`linear_par`, `basis_and_grad`, `basis_columns`). `FitModel` is plain data +(limits, fixed flags, start values, minimizer settings) and selects one of +four minimizers through `aare::Minimizer`: Minuit2's Migrad (default) and +Fumili, or the built-in Levenberg-Marquardt and variable projection +solvers. The private implementation lives in `src/fit/`: `Fit.cpp` holds +the `FitModel` method bodies, the per-thread `detail::PixelFitter` that +dispatches on the minimizer and falls back from VarPro to +Levenberg-Marquardt, `fit_pixel`/`fit_3d`, and the explicit +instantiations; `FitHelpers.hpp` holds the start-value precedence and +shape checks; `DampedGaussNewton.hpp` is the damped Gauss-Newton driver +shared by `LevenbergMarquardt.hpp` and `VariableProjection.hpp`, with the +small fixed-size linear algebra on `std::array` (`Vec`, `Mat`). +`FitMinuit2.cpp` (with `Chi2.hpp` and `MinuitSteps.hpp`) is the only +translation unit that includes Minuit2 headers; Minuit2 is a private, +`BUILD_INTERFACE`-only dependency of `aare_core`. The `[fit]` tests run +every configuration through all four minimizers and check that they +agree; run them when changing any solver. + +### Python layer + +- `python/src/module.cpp` registers everything. Templated classes are + instantiated per value type and cluster size through the + `DEFINE_CLUSTER_BINDINGS` and `DEFINE_BINDINGS_CLUSTERFINDER` macros, + producing names such as `ClusterFinder_Cluster3x3i`, `Pedestal_d`, and + `FastPedestal_i16`. Supported cluster sizes are 2x2, 3x3, 5x5, 7x7, and + 9x9; value types are `i`, `f`, `d`, and `i16` (3x3 only). + `module_config.hpp` fixes the module-wide pedestal type (`pd_type`). +- `python/aare/factory.py` maps NumPy dtypes to those suffixes. The facades + in `python/aare/*.py` are thin factory functions or subclasses that select + the right binding and add Pythonic conveniences (iteration, context + managers, `ScanParameters`). +- `np_helper.hpp` provides `return_image_data` (moves an `NDArray` into a + NumPy array with a capsule deleter) and `make_view_1d/2d/3d` (views into a + NumPy buffer). +- `aare.experimental` and `_aare.experimental` hold unstable APIs. + +## Implementation conventions + +- Put public declarations in `include/aare/` and private implementation details + in `src/`. +- Add new compiled headers, sources, and C++ tests to the explicit lists in the + root `CMakeLists.txt`. +- Keep template implementations in headers unless the supported types are + explicitly instantiated. +- Follow the naming in adjacent code. Broadly, use the `aare` namespace, + CamelCase types, and snake_case functions. +- Treat ownership, lifetime, const-correctness, array shapes, and buffer + contiguity as part of the API when working with `NDArray`, `NDView`, or NumPy + bindings. +- Avoid unnecessary allocations and copies in detector-data and per-pixel + processing paths. Add or update a benchmark when performance is central to a + change. +- Place APIs that may change without notice under the existing experimental + namespace/module. +- Start new source files with `SPDX-License-Identifier: MPL-2.0`, using the + appropriate comment syntax. +- Code copied from other projects keeps its original license. Retain its + copyright and license notice, set the SPDX identifier to that license, and + list the license in `LICENSES/`, `pyproject.toml`, `conda-recipe/meta.yaml`, + and `README.md`. `etc/add_license.py` skips files with a license notice. +- When adding or bumping a fetched dependency that is compiled into the + Python extension, update its version and notice in + `THIRD-PARTY-NOTICES.txt` and the license expressions in `pyproject.toml` + and `conda-recipe/meta.yaml`. +- Prefer descriptive names to comments that only restate the code. +- Do not add a dependency unless the benefit justifies the packaging and + deployment cost. + +## Python-facing changes + +A Python-facing feature can require coordinated changes in several layers: + +1. Update the C++ public API and implementation. +2. Add or update its binding in `python/src/`. +3. Register new bindings in `python/src/module.cpp`. +4. Update the facade or public exports in `python/aare/`. +5. If adding a Python module, add it to `PYTHON_FILES` in + `python/CMakeLists.txt` so it is copied and installed. +6. Add Python tests and update user documentation. + +Bindings must validate NumPy dimensions and data types before constructing +views. Do not return a view whose backing C++ or Python storage can expire while +the view remains reachable. + +## Tests + +Run tests that do not require external detector data with: + +```bash +ctest --test-dir build --output-on-failure -j4 +PYTHONPATH="$PWD/build" python -m pytest python/tests +``` + +For focused runs, use a Catch2 tag or test name, or an individual pytest +file or test: + +```bash +build/run_tests "[tag]" +build/run_tests "Exact test case name" +PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py +PYTHONPATH="$PWD/build" python -m pytest python/tests/test_example.py -k name +``` + +Large detector test files live outside this repository in +`https://gitea.psi.ch/detectors/aare-test-data.git` (Git LFS). To include +data-backed tests, set `AARE_TEST_DATA` and opt in explicitly: + +```bash +export AARE_TEST_DATA=/path/to/aare-test-data +build/run_tests "[.with-data]" +PYTHONPATH="$PWD/build" python -m pytest python/tests --with-data +``` + +- Start bug fixes with a failing regression test when practical. +- Put C++ tests beside the relevant implementation as `src/Thing.test.cpp` + and add them to `TestSources` in the root `CMakeLists.txt`. +- Mark C++ tests requiring external files with `[.with-data]`. +- Mark Python tests requiring external files with `@pytest.mark.withdata` and + use the `test_data_path` fixture from `python/tests/conftest.py`. +- Prefer synthesizing small input files over depending on external data. + `tests/raw_file_helpers.hpp` builds throwaway raw files in C++, and many + Python tests write small raw files into `tmp_path`. +- `tests/test_config.hpp.in` provides `test_data_path()`; + `tests/test_macros.hpp` and `tests/friend_test.hpp` (`TEST_CASE_PRIVATE` + with `FRIEND_TEST`) let a test reach private members. +- Run both suites for changes that cross the C++/Python boundary. +- Do not silently skip required data-backed coverage. Report when the external + test data is unavailable. + +## Formatting and static analysis + +The project uses pre-commit. Install the git hook once per clone with +`pre-commit install` so it runs on every commit. The hooks in +`.pre-commit-config.yaml` are clang-format on C++, cmake-format on CMake +files, `typos` on `docs/` (project words are whitelisted in `typos.toml`), and +basic YAML, TOML, JSON, and large-file checks. CI runs the same hooks on pull +requests. + +Before handing off a broad change, run the relevant checks: + +```bash +pre-commit install # once per clone +pre-commit run --all-files # what CI runs +cmake --build build --target check-format # diff-based clang-format check +cmake --build build --target format-files # apply clang-format to all .cpp/.hpp +cmake --build build --target clang-tidy # uses build/compile_commands.json +``` + +C++ formatting follows `.clang-format` (four-space indentation and an 80-column +limit). CMake files are checked by `cmake-format`. Avoid formatting unrelated +code as part of a focused change. + +## Documentation and release notes + +- Update the relevant `.rst` pages for public behavior or API changes. C++ + pages live at `docs/src/.rst`; Python pages live under + `docs/src/python//`. +- Add new pages to `docs/src/index.rst` or the relevant nested toctree. +- Update `RELEASE.md` under `## Next` for user-visible features, bug fixes, and + API changes. +- Do not change `VERSION` unless performing an explicitly requested release. +- Preserve compatibility with existing detector formats and older recorded + files where practical. Call out intentional API or format incompatibilities. + +Build the documentation with: + +```bash +cmake -S . -B build \ + -DAARE_DOCS=ON \ + -DAARE_PYTHON_BINDINGS=ON +cmake --build build --target docs +``` + +## Working practices and handoff + +- Inspect the adjacent implementation, tests, and documentation before editing. +- Keep changes focused and preserve unrelated modifications in the worktree. +- Do not edit generated files or fetched dependency sources under `build/`. +- For large features, prefer independently testable increments. +- At handoff, summarize the behavior and important files changed, checks run, + checks not run and why, and any compatibility or performance considerations. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..040b659a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,8 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +All project guidance lives in `AGENTS.md`: layout, build and test commands, +architecture, conventions, formatting, documentation, and release notes. +Read it and follow it. Keep new guidance there rather than here so that every +agent and contributor sees the same instructions. diff --git a/CMakeLists.txt b/CMakeLists.txt index c464e2ef..84cdfcc0 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -55,6 +55,10 @@ option( "Install the python extension in the install tree under CMAKE_INSTALL_PREFIX/aare/" OFF) option(AARE_ASAN "Enable AddressSanitizer" OFF) +option( + AARE_TUNE_LOCAL + "Optimize for the building machine's CPU (-march=native -mtune=native). Not portable, the resulting binaries might not run on other machines." + OFF) # Configure which of the dependencies to use FetchContent for option(AARE_FETCH_FMT "Use FetchContent to download fmt" ON) @@ -102,7 +106,7 @@ if(AARE_FETCH_MINUIT2) FetchContent_Declare( Minuit2 GIT_REPOSITORY https://github.com/GooFit/Minuit2.git - GIT_TAG master + GIT_TAG v6-40-02 UPDATE_DISCONNECTED 1) # Disable Minuit2 extras we don't need set(minuit2_mpi @@ -125,7 +129,9 @@ if(AARE_FETCH_MINUIT2) add_library(aare::Minuit2 ALIAS Minuit2) else() find_package(Minuit2 CONFIG QUIET) - if(NOT Minuit2_FOUND) + if(Minuit2_FOUND) + add_library(aare::Minuit2 ALIAS Minuit2::Minuit2) + else() message( STATUS "Stand alone Minuit2 not found. Trying to find it as a part of ROOT") @@ -149,7 +155,7 @@ if(AARE_FETCH_ZMQ) FetchContent_Declare( libzmq GIT_REPOSITORY https://github.com/zeromq/libzmq.git - GIT_TAG v4.3.4 + GIT_TAG v4.3.5 PATCH_COMMAND ${ZMQ_PATCH_COMMAND} UPDATE_DISCONNECTED 1) # Disable unwanted options from libzmq @@ -296,6 +302,22 @@ else() target_compile_options(aare_compiler_flags INTERFACE -Werror) endif() + if(AARE_TUNE_LOCAL) + include(CheckCXXCompilerFlag) + check_cxx_compiler_flag("-march=native" AARE_HAS_MARCH_NATIVE) + check_cxx_compiler_flag("-mtune=native" AARE_HAS_MTUNE_NATIVE) + if(AARE_HAS_MARCH_NATIVE AND AARE_HAS_MTUNE_NATIVE) + message(STATUS "Tuning for the local CPU: -march=native -mtune=native") + target_compile_options(aare_compiler_flags INTERFACE -march=native + -mtune=native) + else() + message( + WARNING + "AARE_TUNE_LOCAL requested but the compiler does not support -march=native/-mtune=native. Ignoring." + ) + endif() + endif() + endif() # GCC/Clang specific if(AARE_PYTHON_BINDINGS) @@ -339,12 +361,14 @@ set(PUBLICHEADERS include/aare/Models.hpp include/aare/FileInterface.hpp include/aare/FilePtr.hpp + include/aare/FastPedestal.hpp include/aare/Frame.hpp include/aare/MultiThreadedFileReader.hpp include/aare/hist/PixelHistogram.hpp include/aare/hist/PixelHistogramImpl.hpp include/aare/hist/PedestalTrackingPixelHistogram.hpp include/aare/GainMap.hpp + include/aare/ROI.hpp include/aare/ROIGeometry.hpp include/aare/DetectorGeometry.hpp include/aare/DataSetAggregation.hpp @@ -360,6 +384,7 @@ set(PUBLICHEADERS include/aare/RawMasterFile.hpp include/aare/RawSubFile.hpp include/aare/VarClusterFinder.hpp + include/aare/to_string.hpp include/aare/utils/task.hpp include/aare/utils/ifstream_helpers.hpp include/aare/utils/math_helpers.hpp) @@ -369,13 +394,15 @@ set(SourceFiles ${CMAKE_CURRENT_SOURCE_DIR}/src/CtbRawFile.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/decode.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/defs.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/ROI.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/ROIGeometry.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/DetectorGeometry.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PedestalTrackingPixelHistogram.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Dtype.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/File.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/FilePtr.cpp - ${CMAKE_CURRENT_SOURCE_DIR}/src/Fit.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/fit/Fit.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/fit/FitMinuit2.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Frame.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Interpolator.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/JungfrauDataFile.cpp @@ -417,7 +444,7 @@ if(AARE_VERBOSE) target_compile_definitions(aare_core PUBLIC AARE_VERBOSE) target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logDEBUG5) else() - target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logERROR) + target_compile_definitions(aare_core PUBLIC AARE_LOG_LEVEL=aare::logWARNING) endif() if(AARE_CUSTOM_ASSERT) @@ -429,12 +456,14 @@ set_target_properties(aare_core PROPERTIES ARCHIVE_OUTPUT_DIRECTORY if(AARE_TESTS) set(TestSources + ${CMAKE_CURRENT_SOURCE_DIR}/tests/raw_file_helpers.hpp ${CMAKE_CURRENT_SOURCE_DIR}/src/algorithm.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/calibration.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/defs.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/decode.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Dtype.test.cpp - ${CMAKE_CURRENT_SOURCE_DIR}/src/Fit.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/FilePtr.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/fit/Fit.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Frame.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/DetectorGeometry.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Interpolation.test.cpp @@ -447,15 +476,19 @@ if(AARE_TESTS) ${CMAKE_CURRENT_SOURCE_DIR}/src/CalculateEta.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/ClusterFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/ClusterFinderMT.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/ProducerConsumerQueue.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/Pedestal.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/FastPedestal.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PixelHistogramImpl.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PixelHistogram.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/hist/PedestalTrackingPixelHistogram.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/JungfrauDataFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/MultiThreadedFileReader.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/NumpyFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/NumpyHelpers.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/RawFile.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/RawSubFile.test.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/src/ROI.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/utils/task.test.cpp ${CMAKE_CURRENT_SOURCE_DIR}/src/to_string.test.cpp) target_sources(tests PRIVATE ${TestSources}) diff --git a/LICENSES/Apache-2.0.txt b/LICENSES/Apache-2.0.txt new file mode 100644 index 00000000..d6456956 --- /dev/null +++ b/LICENSES/Apache-2.0.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/LICENSES/LGPL-2.1-or-later.txt b/LICENSES/LGPL-2.1-or-later.txt new file mode 100644 index 00000000..7f01a06c --- /dev/null +++ b/LICENSES/LGPL-2.1-or-later.txt @@ -0,0 +1,459 @@ + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it becomes +a de-facto standard. To achieve this, non-free programs must be +allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control compilation +and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at + least three years, to give the same user the materials + specified in Subsection 6a, above, for a charge no more + than the cost of performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under any +particular circumstance, the balance of the section is intended to apply, +and the section as a whole is intended to apply in other circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License may add +an explicit geographical distribution limitation excluding those countries, +so that distribution is permitted only in or among countries not thus +excluded. In such case, this License incorporates the limitation as if +written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + + END OF TERMS AND CONDITIONS + diff --git a/LICENSES/MIT.txt b/LICENSES/MIT.txt new file mode 100644 index 00000000..7dfed9f2 --- /dev/null +++ b/LICENSES/MIT.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright 2017-2023 Leon Merten Lohse + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/LICENSES/MPL-2.0.txt b/LICENSES/MPL-2.0.txt new file mode 100644 index 00000000..f4bbcd20 --- /dev/null +++ b/LICENSES/MPL-2.0.txt @@ -0,0 +1,373 @@ +Mozilla Public License Version 2.0 +================================== + +1. Definitions +-------------- + +1.1. "Contributor" + means each individual or legal entity that creates, contributes to + the creation of, or owns Covered Software. + +1.2. "Contributor Version" + means the combination of the Contributions of others (if any) used + by a Contributor and that particular Contributor's Contribution. + +1.3. "Contribution" + means Covered Software of a particular Contributor. + +1.4. "Covered Software" + means Source Code Form to which the initial Contributor has attached + the notice in Exhibit A, the Executable Form of such Source Code + Form, and Modifications of such Source Code Form, in each case + including portions thereof. + +1.5. "Incompatible With Secondary Licenses" + means + + (a) that the initial Contributor has attached the notice described + in Exhibit B to the Covered Software; or + + (b) that the Covered Software was made available under the terms of + version 1.1 or earlier of the License, but not also under the + terms of a Secondary License. + +1.6. "Executable Form" + means any form of the work other than Source Code Form. + +1.7. "Larger Work" + means a work that combines Covered Software with other material, in + a separate file or files, that is not Covered Software. + +1.8. "License" + means this document. + +1.9. "Licensable" + means having the right to grant, to the maximum extent possible, + whether at the time of the initial grant or subsequently, any and + all of the rights conveyed by this License. + +1.10. "Modifications" + means any of the following: + + (a) any file in Source Code Form that results from an addition to, + deletion from, or modification of the contents of Covered + Software; or + + (b) any new file in Source Code Form that contains any Covered + Software. + +1.11. "Patent Claims" of a Contributor + means any patent claim(s), including without limitation, method, + process, and apparatus claims, in any patent Licensable by such + Contributor that would be infringed, but for the grant of the + License, by the making, using, selling, offering for sale, having + made, import, or transfer of either its Contributions or its + Contributor Version. + +1.12. "Secondary License" + means either the GNU General Public License, Version 2.0, the GNU + Lesser General Public License, Version 2.1, the GNU Affero General + Public License, Version 3.0, or any later versions of those + licenses. + +1.13. "Source Code Form" + means the form of the work preferred for making modifications. + +1.14. "You" (or "Your") + means an individual or a legal entity exercising rights under this + License. For legal entities, "You" includes any entity that + controls, is controlled by, or is under common control with You. For + purposes of this definition, "control" means (a) the power, direct + or indirect, to cause the direction or management of such entity, + whether by contract or otherwise, or (b) ownership of more than + fifty percent (50%) of the outstanding shares or beneficial + ownership of such entity. + +2. License Grants and Conditions +-------------------------------- + +2.1. Grants + +Each Contributor hereby grants You a world-wide, royalty-free, +non-exclusive license: + +(a) under intellectual property rights (other than patent or trademark) + Licensable by such Contributor to use, reproduce, make available, + modify, display, perform, distribute, and otherwise exploit its + Contributions, either on an unmodified basis, with Modifications, or + as part of a Larger Work; and + +(b) under Patent Claims of such Contributor to make, use, sell, offer + for sale, have made, import, and otherwise transfer either its + Contributions or its Contributor Version. + +2.2. Effective Date + +The licenses granted in Section 2.1 with respect to any Contribution +become effective for each Contribution on the date the Contributor first +distributes such Contribution. + +2.3. Limitations on Grant Scope + +The licenses granted in this Section 2 are the only rights granted under +this License. No additional rights or licenses will be implied from the +distribution or licensing of Covered Software under this License. +Notwithstanding Section 2.1(b) above, no patent license is granted by a +Contributor: + +(a) for any code that a Contributor has removed from Covered Software; + or + +(b) for infringements caused by: (i) Your and any other third party's + modifications of Covered Software, or (ii) the combination of its + Contributions with other software (except as part of its Contributor + Version); or + +(c) under Patent Claims infringed by Covered Software in the absence of + its Contributions. + +This License does not grant any rights in the trademarks, service marks, +or logos of any Contributor (except as may be necessary to comply with +the notice requirements in Section 3.4). + +2.4. Subsequent Licenses + +No Contributor makes additional grants as a result of Your choice to +distribute the Covered Software under a subsequent version of this +License (see Section 10.2) or under the terms of a Secondary License (if +permitted under the terms of Section 3.3). + +2.5. Representation + +Each Contributor represents that the Contributor believes its +Contributions are its original creation(s) or it has sufficient rights +to grant the rights to its Contributions conveyed by this License. + +2.6. Fair Use + +This License is not intended to limit any rights You have under +applicable copyright doctrines of fair use, fair dealing, or other +equivalents. + +2.7. Conditions + +Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted +in Section 2.1. + +3. Responsibilities +------------------- + +3.1. Distribution of Source Form + +All distribution of Covered Software in Source Code Form, including any +Modifications that You create or to which You contribute, must be under +the terms of this License. You must inform recipients that the Source +Code Form of the Covered Software is governed by the terms of this +License, and how they can obtain a copy of this License. You may not +attempt to alter or restrict the recipients' rights in the Source Code +Form. + +3.2. Distribution of Executable Form + +If You distribute Covered Software in Executable Form then: + +(a) such Covered Software must also be made available in Source Code + Form, as described in Section 3.1, and You must inform recipients of + the Executable Form how they can obtain a copy of such Source Code + Form by reasonable means in a timely manner, at a charge no more + than the cost of distribution to the recipient; and + +(b) You may distribute such Executable Form under the terms of this + License, or sublicense it under different terms, provided that the + license for the Executable Form does not attempt to limit or alter + the recipients' rights in the Source Code Form under this License. + +3.3. Distribution of a Larger Work + +You may create and distribute a Larger Work under terms of Your choice, +provided that You also comply with the requirements of this License for +the Covered Software. If the Larger Work is a combination of Covered +Software with a work governed by one or more Secondary Licenses, and the +Covered Software is not Incompatible With Secondary Licenses, this +License permits You to additionally distribute such Covered Software +under the terms of such Secondary License(s), so that the recipient of +the Larger Work may, at their option, further distribute the Covered +Software under the terms of either this License or such Secondary +License(s). + +3.4. Notices + +You may not remove or alter the substance of any license notices +(including copyright notices, patent notices, disclaimers of warranty, +or limitations of liability) contained within the Source Code Form of +the Covered Software, except that You may alter any license notices to +the extent required to remedy known factual inaccuracies. + +3.5. Application of Additional Terms + +You may choose to offer, and to charge a fee for, warranty, support, +indemnity or liability obligations to one or more recipients of Covered +Software. However, You may do so only on Your own behalf, and not on +behalf of any Contributor. You must make it absolutely clear that any +such warranty, support, indemnity, or liability obligation is offered by +You alone, and You hereby agree to indemnify every Contributor for any +liability incurred by such Contributor as a result of warranty, support, +indemnity or liability terms You offer. You may include additional +disclaimers of warranty and limitations of liability specific to any +jurisdiction. + +4. Inability to Comply Due to Statute or Regulation +--------------------------------------------------- + +If it is impossible for You to comply with any of the terms of this +License with respect to some or all of the Covered Software due to +statute, judicial order, or regulation then You must: (a) comply with +the terms of this License to the maximum extent possible; and (b) +describe the limitations and the code they affect. Such description must +be placed in a text file included with all distributions of the Covered +Software under this License. Except to the extent prohibited by statute +or regulation, such description must be sufficiently detailed for a +recipient of ordinary skill to be able to understand it. + +5. Termination +-------------- + +5.1. The rights granted under this License will terminate automatically +if You fail to comply with any of its terms. However, if You become +compliant, then the rights granted under this License from a particular +Contributor are reinstated (a) provisionally, unless and until such +Contributor explicitly and finally terminates Your grants, and (b) on an +ongoing basis, if such Contributor fails to notify You of the +non-compliance by some reasonable means prior to 60 days after You have +come back into compliance. Moreover, Your grants from a particular +Contributor are reinstated on an ongoing basis if such Contributor +notifies You of the non-compliance by some reasonable means, this is the +first time You have received notice of non-compliance with this License +from such Contributor, and You become compliant prior to 30 days after +Your receipt of the notice. + +5.2. If You initiate litigation against any entity by asserting a patent +infringement claim (excluding declaratory judgment actions, +counter-claims, and cross-claims) alleging that a Contributor Version +directly or indirectly infringes any patent, then the rights granted to +You by any and all Contributors for the Covered Software under Section +2.1 of this License shall terminate. + +5.3. In the event of termination under Sections 5.1 or 5.2 above, all +end user license agreements (excluding distributors and resellers) which +have been validly granted by You or Your distributors under this License +prior to termination shall survive termination. + +************************************************************************ +* * +* 6. Disclaimer of Warranty * +* ------------------------- * +* * +* Covered Software is provided under this License on an "as is" * +* basis, without warranty of any kind, either expressed, implied, or * +* statutory, including, without limitation, warranties that the * +* Covered Software is free of defects, merchantable, fit for a * +* particular purpose or non-infringing. The entire risk as to the * +* quality and performance of the Covered Software is with You. * +* Should any Covered Software prove defective in any respect, You * +* (not any Contributor) assume the cost of any necessary servicing, * +* repair, or correction. This disclaimer of warranty constitutes an * +* essential part of this License. No use of any Covered Software is * +* authorized under this License except under this disclaimer. * +* * +************************************************************************ + +************************************************************************ +* * +* 7. Limitation of Liability * +* -------------------------- * +* * +* Under no circumstances and under no legal theory, whether tort * +* (including negligence), contract, or otherwise, shall any * +* Contributor, or anyone who distributes Covered Software as * +* permitted above, be liable to You for any direct, indirect, * +* special, incidental, or consequential damages of any character * +* including, without limitation, damages for lost profits, loss of * +* goodwill, work stoppage, computer failure or malfunction, or any * +* and all other commercial damages or losses, even if such party * +* shall have been informed of the possibility of such damages. This * +* limitation of liability shall not apply to liability for death or * +* personal injury resulting from such party's negligence to the * +* extent applicable law prohibits such limitation. Some * +* jurisdictions do not allow the exclusion or limitation of * +* incidental or consequential damages, so this exclusion and * +* limitation may not apply to You. * +* * +************************************************************************ + +8. Litigation +------------- + +Any litigation relating to this License may be brought only in the +courts of a jurisdiction where the defendant maintains its principal +place of business and such litigation shall be governed by laws of that +jurisdiction, without reference to its conflict-of-law provisions. +Nothing in this Section shall prevent a party's ability to bring +cross-claims or counter-claims. + +9. Miscellaneous +---------------- + +This License represents the complete agreement concerning the subject +matter hereof. If any provision of this License is held to be +unenforceable, such provision shall be reformed only to the extent +necessary to make it enforceable. Any law or regulation which provides +that the language of a contract shall be construed against the drafter +shall not be used to construe this License against a Contributor. + +10. Versions of the License +--------------------------- + +10.1. New Versions + +Mozilla Foundation is the license steward. Except as provided in Section +10.3, no one other than the license steward has the right to modify or +publish new versions of this License. Each version will be given a +distinguishing version number. + +10.2. Effect of New Versions + +You may distribute the Covered Software under the terms of the version +of the License under which You originally received the Covered Software, +or under the terms of any subsequent version published by the license +steward. + +10.3. Modified Versions + +If you create software not governed by this License, and you want to +create a new license for such software, you may create and use a +modified version of this License if you rename the license and remove +any references to the name of the license steward (except to note that +such modified license differs from this License). + +10.4. Distributing Source Code Form that is Incompatible With Secondary +Licenses + +If You choose to distribute Source Code Form that is Incompatible With +Secondary Licenses under the terms of this version of the License, the +notice described in Exhibit B of this License must be attached. + +Exhibit A - Source Code Form License Notice +------------------------------------------- + + This Source Code Form is subject to the terms of the Mozilla Public + License, v. 2.0. If a copy of the MPL was not distributed with this + file, You can obtain one at http://mozilla.org/MPL/2.0/. + +If it is not possible or desirable to put the notice in a particular +file, then You may include the notice in a location (such as a LICENSE +file in a relevant directory) where a recipient would be likely to look +for such a notice. + +You may add additional accurate notices of copyright ownership. + +Exhibit B - "Incompatible With Secondary Licenses" Notice +--------------------------------------------------------- + + This Source Code Form is "Incompatible With Secondary Licenses", as + defined by the Mozilla Public License, v. 2.0. \ No newline at end of file diff --git a/README.md b/README.md index ca41ae10..7a9f1c8a 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,20 @@ Detailed documentation including installation can be found in [Documentation](ht This project is licensed under the MPL-2.0 license. See the LICENSE file or https://www.mozilla.org/en-US/MPL/ for details. +Two files contain code from other projects and keep their original licenses: + +- `include/aare/ProducerConsumerQueue.hpp` is Apache-2.0, from + [folly](https://github.com/facebook/folly) by Meta Platforms. +- `src/NumpyHelpers.cpp` is MIT, from + [libnpy](https://github.com/llohse/libnpy) by Leon Merten Lohse. + +The full license texts are in the `LICENSES` directory. + +The Python wheels and conda packages also contain Minuit2 +(LGPL-2.1-or-later), {fmt} and nlohmann/json (MIT), and pybind11 +(BSD-3-Clause), which CMake downloads at build time. See +`THIRD-PARTY-NOTICES.txt` for their notices. + ## Build and install Prerequisites diff --git a/RELEASE.md b/RELEASE.md index 638b8b3b..88686f53 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,9 +1,134 @@ # Release notes -## Next +## 2026.9.30 + +### New Features: + +- Added ``pedestal_ready`` to ``ClusterFinder`` and ``ClusterFinderMT`` in C++ + and Python to check whether enough pedestal frames have been pushed to find + clusters. +- Added the Python ``FrameDiscardPolicy`` enum with ``NoDiscard``, ``Discard``, + and ``DiscardPartial``, enabling access to ``RawMasterFile.frame_discard_policy``. +- Added the Python ``Pedestal`` factory with ``dtype`` selection, matching + ``FastPedestal`` and defaulting to ``float64`` output. +- Added ``FastPedestal`` in C++ and Python for per-pixel running mean + and population standard deviation. It supports exponentially +- Added ``ClusterFile.frames()`` and ``ClusterFile.chunks()`` in C++ and + Python for iteration from the current file position. Frames preserve empty + frames and frame numbers; chunks support an optional positive size override. + Python's default file iteration continues to yield chunks. +- Added ``FastPedestal`` in C++ and Python for per-pixel running mean, + population variance, and standard deviation. It supports exponentially + weighted updates, initialization from files, direct subtraction from NumPy + arrays, and ``float64``, ``float32``, and ``int16`` output types. +- Added the ``Pedestal_i16`` Python binding alongside + ``FastPedestal_d``, ``FastPedestal_f``, and ``FastPedestal_i16``. +- ``PedestalTrackingPixelHistogram.fill_from_file()`` now uses parallel, + double-buffered file reading and accepts ``reader_threads`` and + ``reader_chunk_size`` tuning parameters. +- Added the ``AARE_TUNE_LOCAL`` CMake option to build with ``-march=native`` + and ``-mtune=native`` when supported. Binaries built with this option are + specific to the local CPU and may not be portable. +- Added string representator in python for Cluster and Eta +- Added roi slice method in python for easy slicing of numpy arrays ``array[roi.slice()]``. +- added context manager for ``aare.RawMasterFile`` +- Added a configurable minimizer for the fit models, selected with the + ``minimizer`` constructor argument or property in Python, or + ``FitModel::SetMinimizer`` in C++. ``Minimizer.Migrad`` remains the default. + ``Minimizer.Fumili`` is Minuit2's Gauss-Newton minimizer, which uses the + analytic model derivatives and needs far fewer function evaluations. + ``Minimizer.LevenbergMarquardt`` is a built-in, dependency-free + Levenberg-Marquardt solver with the analytic Jacobian; it reflects steps at + parameter limits and fits data cubes without per-pixel allocations. A fit + that stalls away from a minimum, or whose width collapses below the spacing + of the scan points, is reported as failed (all zeros) rather than returned + as a result, so noisy or signal-free pixels may fail. With + ``compute_errors``, Fumili and LevenbergMarquardt report parameter errors + from their linearised covariance instead of running Hesse. Pixels for which + Fumili does not reach a valid minimum, which Minuit2's implementation cannot + once a two-sided limit becomes active, are refitted with Migrad. +- ``Minimizer.LevenbergMarquardt`` is about twice as fast for every model: it + evaluates the Jacobian at the trial point instead of re-evaluating accepted + points, accumulates the normal equations with fixed-size loops, needs a + single factorisation per iteration, and the erf-based models compute one + exponential per point instead of two. Results are unchanged, except that + ``GaussianChargeSharing`` and ``GaussianChargeSharingKb`` now evaluate the + erf from the Gaussian's exponential, a rounding-level change. One iteration + costs one model evaluation of the ``max_calls`` budget, so pixels that used + to exhaust the default budget of 100 may now converge. +- Added ``Minimizer.VarPro``, a built-in variable projection + solver. Every bundled model now declares the parameters it is linear in + (``linear_par`` and ``basis_and_grad`` in C++, checked against the model + derivatives by the tests). Each trial point solves those exactly and the + Levenberg-Marquardt iteration runs over the nonlinear parameters only, so + the fit needs fewer evaluations and does not depend on start values of the + linear parameters. When the basis functions are nearly collinear, as for a + polynomial on a narrow range far from x = 0, one more pass over the data + refines the linear solution to the precision of the other minimizers. + Limits on linear parameters, pixels that do not converge + and models without the separable structure fall back to + ``LevenbergMarquardt``. Parameter errors are the same Gauss-Newton + estimates. The Gaussian, plateau, charge-sharing and S-curve models also + provide their basis functions for all scan points at once + (``basis_columns``), a loop that compiles to vector instructions with the + new ``model::fast_exp``; ``eval`` and ``eval_and_grad``, and with them the + other minimizers, are unchanged. ``Minimizer.Migrad`` remains the + default. +- ``NDArray``/``NDView`` expressions now support scalar operands + (``2 * a + b / 4``) and can be assigned to an existing ``NDArray``, reusing + its buffer. ### API Changes: +- The C++ ``NDArray`` view constructor and ``NDArray::copy_from()`` take + ``NDView``. Mutable views still convert implicitly, so + existing callers are unaffected, and read-only views can now be copied. +- Added C++ overloads of ``adc_sar_05_06_07_08decode64to16``, + ``adc_sar_05_decode64to16`` and ``adc_sar_04_decode64to16`` that take the + packed samples as ``NDView``. Words are assembled with + ``memcpy``, so the buffer does not need to be 8-byte aligned. The existing + ``NDView`` overloads are unchanged. +- ``NDArray`` ``+ - * /`` with a scalar now returns a lazy expression instead + of an ``NDArray`` and no longer converts the scalar to the element type. +- ``FitModel`` is now plain data without a pimpl (C++ users need to rebuild); + it gained ``strategy()``, ``lower_limit()``, ``upper_limit()`` and + ``value()`` accessors and lost ``impl()``. Bad parameter indices raise + ``std::out_of_range`` (``IndexError`` in Python) and ``SetParLimits`` + rejects ``lo >= hi``. The models in ``Models.hpp`` no longer provide the + Minuit specific ``compute_steps`` and ``compute_ranges`` helpers. +- Fit behaviour common to all minimizers: user start and fixed values are + applied before the validity check and free start values are clamped into + their limits; one-sided limits are open-ended for ``LevenbergMarquardt`` + while the Minuit2 minimizers use a wide two-sided range as before; with + ``compute_errors``, fixed parameters and parameters ending on a limit report + an error of 0 (fixed parameters previously reported 1.0 with Minuit2). + +- ``FastPedestal`` variance is now a private ``double`` intermediate. Removed + the C++ ``variance()``/``variance_unchecked()`` APIs and Python ``var()``. + Standard deviation is calculated before conversion to the output type, + avoiding overflow of intermediate variance for ``int16`` output. Negative + variance from floating-point roundoff is clamped to zero. +- ``Pedestal`` now always accumulates sums and sums of squares in ``double``, + like ``FastPedestal``. Mean and standard deviation output types + are unchanged. Removed the C++ ``get_sum()``/``get_sum2()`` getters and + Python ``sum``/``sum2`` properties; internal sums are no longer exposed. +- Removed the public ``Pedestal.variance()`` and ``cached_std()`` APIs and + C++ ``update_std()``. Use ``std()`` to calculate the current population + standard deviation; variance is now an internal implementation detail. +- Added ``ClusterFile::read_frame(ClusterVector&)`` for allocation-reusing C++ + reads. It returns ``false`` at a clean end of file. The value-returning C++ + overload now returns ``std::optional>`` and + Python ``read_frame()`` returns ``None`` at end of file. Incomplete frames + still raise an error. +- Added an explicit boolean conversion to the C++ ``FilePtr`` type. +- Removed the public C++ ``ClusterFile::open`` method. Construct a new + ``ClusterFile`` to reopen a file or change its mode. +- ``ClusterFinder`` now uses ``FastPedestal``. It must receive 1000 pedestal + frames before cluster finding; ``find_clusters()`` raises + an error until initialization is complete. Added ``update_threshold()`` to + recompute the per-pixel detection thresholds. +- Added the ``queue_depth`` constructor argument to ``ClusterFinderMT`` to + configure the number of preallocated frame buffers per worker thread. - Exposed ``ClusterVector.empty()`` in the Python API. - Added ClusterVector.estimate_n_clusters - Removed the lmfit dependency and the legacy ``fit_gaus``, ``fit_pol1``, @@ -16,8 +141,120 @@ - ``NDView`` now converts to ``NDView``; ``expand4to8bit`` and ``expand24to32bit`` accept const input views. +- ``RawMasterFile::geometry()`` is deprecetad and returns full detector geometry information including module geometry. Use +``RawMasterFile::module_layout()`` to get num_modules in x an y +- ``RawMasterFile::rois()`` always returns a list of rois (no optional). Per default it returns a list of one ROI element spawing the entire detector +- ``TimingMode::Auto`` changed to ``TimingMode::AUTO_TIMING``, ``TimingMode::Trigger`` changed to ``TimingMode::TRIGGER_EXPOSURE`` + + ### Bugfixes: +- Fitting a 3D data cube in Python without ``y_err`` now returns ``par_err`` + when ``compute_errors`` is set, as fitting a single pixel already did. +- Samples with a zero error are ignored by every minimizer, as documented, + also when they hold NaN. They are now removed before the start values are + estimated; previously a masked NaN at a sample used by the estimate, such + as the first or last point of ``Pol1``, made even Migrad fail. A pixel with + fewer remaining samples than free parameters is reported as failed. +- Fitting a single pixel checks that ``x``, ``y`` and ``y_err`` have the same + size and raises ``RuntimeError`` otherwise. Previously a shorter ``y`` or + ``y_err`` was read past its end. +- Setting nSigma on ``ClusterFinder`` and ``ClusterFinderMT`` before the + pedestal is ready no longer raises an error. The new value is used when the + threshold is initialized. Previously ``ClusterFinderMT.set_nSigma()`` also + left the worker threads with different nSigma values. +- The Python ``Cluster`` constructors validate that the data array holds + exactly one value per pixel and raise ``ValueError`` otherwise. Previously a + longer array wrote past the end of the cluster data. +- The Python CTB decoding helpers ``adc_sar_*decode64to16``, + ``apply_custom_weights``, ``expand24to32bit``, ``expand4to8bit`` and + ``decode_my302`` validate their input through ``make_view``. They require + C-contiguous arrays of the documented rank and reject other dtypes with + ``TypeError`` instead of converting a copy. The ADC SAR decoders no longer + reinterpret the byte buffer as aligned 64-bit words; a row length that is + not a multiple of 8 bytes raises ``ValueError`` instead of dropping the + trailing bytes. Read-only arrays are accepted where the input is only read. + ``decode_my302`` raises ``ValueError`` instead of ``RuntimeError`` for a + wrong input size. +- Mismatched operators inhibited vectorization in gcc of NDArray math operators +- ``NDArray``/``NDView`` math operators and expressions were not vectorized + for ``uint8_t`` and 64 bit integers, up to 30x slower than a plain loop. +- Removed the move constructor and move assignment of + ``ProducerConsumerQueue``. They left the moved-from queue with a null + buffer, crashing its destructor if it still held elements, and leaked + the target's buffer on assignment. The queue is now non-movable, as in + the upstream folly implementation; hold it in a ``unique_ptr`` to move + it around. +- ``ProducerConsumerQueue`` rejects sizes below two with + ``std::invalid_argument`` in all build types. The previous assertion was + compiled out of Release builds, so a zero-size queue overflowed its buffer + on the first write. +- Corrected the license metadata for vendored third-party code. + ``ProducerConsumerQueue.hpp`` is tagged ``Apache-2.0`` like upstream folly + and ``NumpyHelpers.cpp`` is tagged ``MIT`` like upstream libnpy. The MPL 2.0, + Apache 2.0, and MIT texts ship in ``LICENSES/``, and both the conda package + and the Python wheel declare ``MPL-2.0 AND Apache-2.0 AND MIT`` with all + three license files. Building the wheel now requires scikit-build-core 0.11 + or newer. +- The wheel and conda package now ship ``THIRD-PARTY-NOTICES.txt`` and the + LGPL 2.1 text for the libraries compiled into the extension: Minuit2 + (LGPL-2.1-or-later), {fmt} and nlohmann/json (MIT), and pybind11 + (BSD-3-Clause). Their license expressions include these licenses. + Minuit2 is now fetched at tag ``v6-40-02`` instead of ``master``, and + libzmq at ``v4.3.5`` (MPL-2.0) instead of ``v4.3.4`` (LGPL-3.0 with a + static-linking exception). +- ``RawFile`` and ``File`` reject raw files with frame padding disabled unless + the frame discard policy is ``discardpartial``. The constructor reports the + master path before opening data subfiles. Legacy ``.raw`` master files now + parse the frame discard policy so unpadded ``discardpartial`` files remain + readable. +- Fixed ``CtbRawFile.read_frame(index)`` and reads after ``seek(index)`` at + subfile boundaries skipping a subfile, returning the wrong frame or raising + ``Subfile index out of range``. +- ``Pedestal`` reports mismatched frame shapes with exceptions in all push + overloads, including Debug builds, instead of aborting on assertions. +- Python ``Pedestal`` constructors reject negative dimensions and sample + counts instead of converting them to large unsigned values. +- ``Pedestal`` clamps negative variance from floating-point roundoff to zero, + preventing NaN standard deviations for nearly constant inputs. +- Python ``Pedestal.push()`` now requires C-contiguous ``uint16`` frames + without implicit conversion. Both ``push()`` and ``push_with_threshold()`` + validate that frames and thresholds are two-dimensional before constructing + views, preventing incorrect results from unsupported array layouts or ranks. +- Python ``ClusterFinder.push_pedestal_frame()`` and ``find_clusters()`` now + require C-contiguous ``uint16`` frames without implicit conversion, raising + ``TypeError`` instead of silently copying and casting the input. The frame + argument can also be passed by keyword as ``frame``. + +- Fixed a leaked empty ``ClusterVector`` at the end of Python ``ClusterFile`` + iteration. Chunk iteration now rejects a zero chunk size. +- ``ClusterFile::read_clusters`` and Python iteration now report incomplete + frame headers and cluster records instead of treating truncated files as a + clean end of file, with or without ROI or noise filtering. +- ``ClusterFile::write_frame`` now reports incomplete writes instead of + silently continuing with a truncated file. +- Gain-map application now checks the complete cluster footprint, preventing + out-of-bounds access for cluster sizes larger than 3x3. +- ``RawFile`` now derives its frame count from the shortest selected raw + subfile series across all ROIs. Frame-number reads use the same bounds, and + Python ``len(reader)`` returns the adjusted count. A warning is printed when + subfile counts differ or their minimum differs from the recorded master + count. The expected frame count is not used and master metadata is preserved. + Warning-level logging is now enabled in non-verbose builds. +- Raw frame, batch, ROI, and frame-number read errors now include the attempted + frame index and file path in C++ and Python, preserving subfile error details. + Subfile errors omit redundant master-file context and C++ source locations; + out-of-range errors include the available frame count. + Synchronization errors identify the last raw data file for the affected module. + Top-level frame bounds errors state the total frame count and that indices + are zero-based. +- Fixed ``ClusterVector`` move operations to transfer storage instead of + copying every cluster. +- Validate that ``ClusterVector`` masks are one-dimensional, C-contiguous + Boolean arrays. +- Preserve signed ``ClusterVector`` frame numbers when filtering or reducing + cluster dimensions. - Fixed broken reading of old (pre reordering) Moench03 +- Supports reading all timing modes supported in slsDetectorPackage (auto, trigger, gating, burst_trigger, trigger_gating) ## 2026.7.2 @@ -33,6 +270,7 @@ - ``aare.transfrom.Matterhorn10Transform`` reshapes data such that first dimension is number of counters - Added support for len() for files. Returns the number of frames - Added support for direct subtraction of Pedestal from numpy array +- Added support to read files with disabled udp ports ### Bugfixes: @@ -164,8 +402,3 @@ dhanya.thattil@psi.ch - - - - - diff --git a/THIRD-PARTY-NOTICES.txt b/THIRD-PARTY-NOTICES.txt new file mode 100644 index 00000000..3d07d9c7 --- /dev/null +++ b/THIRD-PARTY-NOTICES.txt @@ -0,0 +1,137 @@ +Third-party notices for aare binary packages +============================================ + +The aare Python wheels and conda packages compile the following libraries into +the aare extension module. They are not part of the aare source repository; +CMake downloads them at build time. Their license notices follow below. + + Minuit2 v6-40-02 LGPL-2.1-or-later https://github.com/GooFit/Minuit2 + {fmt} 12.1.0 MIT https://github.com/fmtlib/fmt + nlohmann/json 3.11.3 MIT https://github.com/nlohmann/json + pybind11 3.0.3 BSD-3-Clause https://github.com/pybind/pybind11 + +Minuit2 is linked statically. To use aare with a modified Minuit2, build aare +from source (https://github.com/slsdetectorgroup/aare) with +-DAARE_FETCH_MINUIT2=OFF and make your Minuit2 build findable by CMake's +find_package(Minuit2). + +------------------------------------------------------------------------------ +Minuit2 +------------------------------------------------------------------------------ + +Copyright (c) 2005 LCG ROOT Math team, CERN/PH-SFT + +This library is free software; you can redistribute it and/or +modify it under the terms of the GNU Lesser General Public +License as published by the Free Software Foundation; either +version 2.1 of the License, or (at your option) any later version. + +This library is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +Lesser General Public License for more details. + +You should have received a copy of the GNU Lesser General Public +License along with this library; if not, write to the Free Software +Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + +See LGPL2_1.txt next to this file for the text of the LGPL version 2.1. +For questions regarding the ROOT license, please contact rootdev@cern.ch. + +The precompiled binary distributions provided as a service to our community +may feature packages released with licenses different from LGPL. + +In aare packages the text of the LGPL version 2.1 is in +LICENSES/LGPL-2.1-or-later.txt. + +------------------------------------------------------------------------------ +{fmt} +------------------------------------------------------------------------------ + +Copyright (c) 2012 - present, Victor Zverovich and {fmt} contributors + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +--- Optional exception to the license --- + +As an exception, if, as a result of your compiling your source code, portions +of this Software are embedded into a machine-executable object form of such +source code, you may redistribute such embedded portions in such object form +without including the above copyright and permission notices. + +------------------------------------------------------------------------------ +nlohmann/json +------------------------------------------------------------------------------ + +MIT License + +Copyright (c) 2013-2022 Niels Lohmann + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +------------------------------------------------------------------------------ +pybind11 +------------------------------------------------------------------------------ + +Copyright (c) 2016 Wenzel Jakob , All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND +ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +Please also refer to the file .github/CONTRIBUTING.md, which clarifies licensing of +external contributions to this project including patches, pull requests, etc. diff --git a/VERSION b/VERSION index 2398eb24..966e58e3 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2026.7.2 \ No newline at end of file +2026.9.30 \ No newline at end of file diff --git a/benchmarks/CMakeLists.txt b/benchmarks/CMakeLists.txt index ef378e4d..e874a003 100644 --- a/benchmarks/CMakeLists.txt +++ b/benchmarks/CMakeLists.txt @@ -18,8 +18,9 @@ FetchContent_MakeAvailable(benchmark) add_executable(benchmarks) target_sources( - benchmarks PRIVATE ndarray_benchmark.cpp calculateeta_benchmark.cpp - reduce_benchmark.cpp) + benchmarks + PRIVATE ndarray_benchmark.cpp ndview_benchmark.cpp calculateeta_benchmark.cpp + reduce_benchmark.cpp clusterfile_benchmark.cpp) # Link Google Benchmark and other necessary libraries target_link_libraries(benchmarks PRIVATE benchmark::benchmark aare_core diff --git a/benchmarks/clusterfile_benchmark.cpp b/benchmarks/clusterfile_benchmark.cpp new file mode 100644 index 00000000..7d757532 --- /dev/null +++ b/benchmarks/clusterfile_benchmark.cpp @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/ClusterFile.hpp" + +#include +#include +#include +#include +#include + +class ClusterFileFixture : public benchmark::Fixture { + using Cluster = aare::Cluster; + using File = aare::ClusterFile; + using Vector = aare::ClusterVector; + + std::filesystem::path m_path; + static constexpr int32_t frame_count = 256; + static constexpr size_t chunk_size = 1000; + + static void observe(Vector &clusters) { + benchmark::DoNotOptimize(clusters.data()); + benchmark::DoNotOptimize(clusters.size()); + } + + public: + void SetUp(const benchmark::State &state) override { + const auto stamp = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_path = std::filesystem::temp_directory_path() / + ("aare-cluster-benchmark-" + std::to_string(stamp) + ".clust"); + File writer(m_path, chunk_size, "w"); + Vector frame(0); + frame.resize(state.range(0)); + for (int32_t number = 0; number < frame_count; ++number) { + frame.set_frame_number(number); + writer.write_frame(frame); + } + } + + void TearDown(const benchmark::State &) override { + std::error_code error; + std::filesystem::remove(m_path, error); + } + + template void read(benchmark::State &state) { + for (auto _ : state) { + File reader(m_path, chunk_size); + if constexpr (Iterate) { + auto consume = [](auto range) { + for (auto &clusters : range) { + observe(clusters); + } + }; + if constexpr (Frames) { + consume(reader.frames()); + } else { + consume(reader.chunks()); + } + } else if constexpr (Frames) { + Vector frame(0); + while (reader.read_frame(frame)) { + observe(frame); + } + } else { + while (true) { + auto chunk = reader.read_clusters(chunk_size); + if (chunk.empty()) { + break; + } + observe(chunk); + } + } + } + state.SetBytesProcessed(state.iterations() * frame_count * + state.range(0) * sizeof(Cluster)); + } +}; + +BENCHMARK_DEFINE_F(ClusterFileFixture, ReadFrames)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, IterateFrames)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, ReadChunks)(benchmark::State &state) { + read(state); +} +BENCHMARK_DEFINE_F(ClusterFileFixture, IterateChunks)(benchmark::State &state) { + read(state); +} + +BENCHMARK_REGISTER_F(ClusterFileFixture, ReadFrames)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, IterateFrames)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, ReadChunks)->Arg(64)->Arg(1024); +BENCHMARK_REGISTER_F(ClusterFileFixture, IterateChunks)->Arg(64)->Arg(1024); diff --git a/benchmarks/fit_benchmark.cpp b/benchmarks/fit_benchmark.cpp index 626bde8d..8e522f80 100644 --- a/benchmarks/fit_benchmark.cpp +++ b/benchmarks/fit_benchmark.cpp @@ -3,6 +3,7 @@ #include "aare/FitModel.hpp" #include "aare/Models.hpp" +#include #include #include #include @@ -75,52 +76,31 @@ static void report_accuracy(benchmark::State &state, const TestCase &tc, // Benchmarks // ---------- -// Minuit2, analytic gradient (no Hesse) -static void BM_FitGausMinuitGrad(benchmark::State &state) { - const auto &tc = get_test_cases()[state.range(0)]; - auto data = generate_gaussian_data(tc); - auto xv = data.x.view(); - auto yv = data.y.view(); - - const auto model = aare::FitModel( - /*strategy = */ 0, - /*max_calls = */ 500, // increase for noisy signals - /*tolerance = */ 0.5, - /*compute_errors = */ false); - - aare::NDArray result; - for (auto _ : state) { - result = aare::fit_pixel(model, xv, yv); - benchmark::DoNotOptimize(result.data()); - } - - report_accuracy(state, tc, result); - state.SetLabel(tc.name); -} - -// Minuit2, analytic gradient + Hesse -static void BM_FitGausMinuitGradHesse(benchmark::State &state) { +static void run_gaussian_fit(benchmark::State &state, aare::Minimizer minimizer, + bool compute_errors) { const auto &tc = get_test_cases()[state.range(0)]; auto data = generate_gaussian_data(tc); auto xv = data.x.view(); auto yv = data.y.view(); auto ev = data.y_err.view(); - const auto model = aare::FitModel( - 0, 500, 0.5, true); // compute_errors = true -> Runs Hesse and provides - // errors on fitted params + const aare::FitModel model( + /*strategy = */ 0, + /*max_calls = */ 500, // increase for noisy signals + /*tolerance = */ 0.5, compute_errors, minimizer); aare::NDArray result; for (auto _ : state) { - result = aare::fit_pixel(model, xv, yv, ev); + result = compute_errors + ? aare::fit_pixel(model, xv, yv, ev) + : aare::fit_pixel(model, xv, yv); benchmark::DoNotOptimize(result.data()); } - // result has 6 elements: [A, mu, sig, err_A, err_mu, err_sig] report_accuracy(state, tc, result); - // Also report Hesse uncertainties - if (result.size() >= 6) { + // With errors the result is [A, mu, sig, err_A, err_mu, err_sig, chi2] + if (compute_errors && result.size() >= 6) { state.counters["errA"] = result(3); state.counters["errMu"] = result(4); state.counters["errSig"] = result(5); @@ -128,12 +108,372 @@ static void BM_FitGausMinuitGradHesse(benchmark::State &state) { state.SetLabel(tc.name); } -BENCHMARK(BM_FitGausMinuitGrad) +// Migrad, analytic gradient (no Hesse) +static void BM_FitGausMigrad(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::Migrad, false); +} + +// Migrad, analytic gradient + Hesse for the parameter errors +static void BM_FitGausMigradHesse(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::Migrad, true); +} + +// Fumili, analytic gradient and linearised Hessian +static void BM_FitGausFumili(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::Fumili, false); +} + +// Fumili, parameter errors from the linearised covariance +static void BM_FitGausFumiliErrors(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::Fumili, true); +} + +// Built-in Levenberg-Marquardt, analytic Jacobian +static void BM_FitGausLM(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::LevenbergMarquardt, false); +} + +// Built-in Levenberg-Marquardt, Gauss-Newton parameter errors +static void BM_FitGausLMErrors(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::LevenbergMarquardt, true); +} + +// Built-in variable projection: A solved exactly, iteration over mu and sigma +static void BM_FitGausVarPro(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::VarPro, false); +} + +static void BM_FitGausVarProErrors(benchmark::State &state) { + run_gaussian_fit(state, aare::Minimizer::VarPro, true); +} + +// Gaussian with sigma fixed: exercises the fixed-parameter path of the +// minimizers on the Moderate_noise case. +static void run_gaussian_fixed_sigma_fit(benchmark::State &state, + aare::Minimizer minimizer) { + const auto &tc = get_test_cases()[1]; + auto data = generate_gaussian_data(tc); + auto xv = data.x.view(); + auto yv = data.y.view(); + + aare::FitModel model(0, 500, 0.5, false, minimizer); + model.FixParameter(2, tc.true_sig); + + aare::NDArray result; + for (auto _ : state) { + result = aare::fit_pixel(model, xv, yv); + benchmark::DoNotOptimize(result.data()); + } + report_accuracy(state, tc, result); +} + +static void BM_FitGausFixedSigmaMigrad(benchmark::State &state) { + run_gaussian_fixed_sigma_fit(state, aare::Minimizer::Migrad); +} +static void BM_FitGausFixedSigmaFumili(benchmark::State &state) { + run_gaussian_fixed_sigma_fit(state, aare::Minimizer::Fumili); +} +static void BM_FitGausFixedSigmaLM(benchmark::State &state) { + run_gaussian_fixed_sigma_fit(state, aare::Minimizer::LevenbergMarquardt); +} +static void BM_FitGausFixedSigmaVarPro(benchmark::State &state) { + run_gaussian_fixed_sigma_fit(state, aare::Minimizer::VarPro); +} + +// ---------------------------------------------------------------- +// Rising S-curve (6 parameters), typical for threshold scans +// ---------------------------------------------------------------- +static constexpr ssize_t N_SCURVE_POINTS = 100; + +static void run_scurve_fit(benchmark::State &state, aare::Minimizer minimizer, + bool weighted = false) { + const std::vector truth = {5.0, 0.1, 50.0, 4.0, 200.0, 0.5}; + const double noise_sigma = 0.02 * truth[4]; + + aare::NDArray x({N_SCURVE_POINTS}); + aare::NDArray y({N_SCURVE_POINTS}); + aare::NDArray y_err({N_SCURVE_POINTS}, noise_sigma); + std::mt19937 rng(SEED); + std::normal_distribution noise(0.0, noise_sigma); + for (ssize_t i = 0; i < N_SCURVE_POINTS; ++i) { + x[i] = static_cast(i); + y[i] = aare::model::RisingScurve::eval(x[i], truth) + noise(rng); + } + auto xv = x.view(); + auto yv = y.view(); + auto ev = y_err.view(); + + const aare::FitModel model(0, 500, 0.5, false, + minimizer); + + aare::NDArray result; + for (auto _ : state) { + result = + weighted + ? aare::fit_pixel(model, xv, yv, ev) + : aare::fit_pixel(model, xv, yv); + benchmark::DoNotOptimize(result.data()); + } + + state.counters["dMu"] = result(2) - truth[2]; + state.counters["dSig"] = result(3) - truth[3]; + state.counters["dA"] = result(4) - truth[4]; + state.counters["chi2"] = result(6); +} + +static void BM_FitScurveMigrad(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::Migrad); +} + +static void BM_FitScurveFumili(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::Fumili); +} + +static void BM_FitScurveLM(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::LevenbergMarquardt); +} + +static void BM_FitScurveVarPro(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::VarPro); +} + +// Weighted fits take the per-point uncertainties into the residuals. +static void BM_FitScurveWeightedMigrad(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::Migrad, true); +} +static void BM_FitScurveWeightedFumili(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::Fumili, true); +} +static void BM_FitScurveWeightedLM(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::LevenbergMarquardt, true); +} +static void BM_FitScurveWeightedVarPro(benchmark::State &state) { + run_scurve_fit(state, aare::Minimizer::VarPro, true); +} + +// ---------------------------------------------------------------- +// Pol2 (3 parameters, cheap model): exposes the solver overhead per +// iteration rather than the model evaluation. +// ---------------------------------------------------------------- +static void run_pol2_fit(benchmark::State &state, aare::Minimizer minimizer) { + constexpr ssize_t n = 51; + const std::vector truth = {12.0, -0.8, 0.05}; + + aare::NDArray x({n}); + aare::NDArray y({n}); + std::mt19937 rng(SEED); + std::normal_distribution noise(0.0, 2.0); + for (ssize_t i = 0; i < n; ++i) { + x[i] = static_cast(i); + y[i] = aare::model::Pol2::eval(x[i], truth) + noise(rng); + } + auto xv = x.view(); + auto yv = y.view(); + + const aare::FitModel model(0, 500, 0.5, false, + minimizer); + + aare::NDArray result; + for (auto _ : state) { + result = aare::fit_pixel(model, xv, yv); + benchmark::DoNotOptimize(result.data()); + } + state.counters["dp2"] = result(2) - truth[2]; +} + +static void BM_FitPol2Migrad(benchmark::State &state) { + run_pol2_fit(state, aare::Minimizer::Migrad); +} +static void BM_FitPol2Fumili(benchmark::State &state) { + run_pol2_fit(state, aare::Minimizer::Fumili); +} +static void BM_FitPol2LM(benchmark::State &state) { + run_pol2_fit(state, aare::Minimizer::LevenbergMarquardt); +} +static void BM_FitPol2VarPro(benchmark::State &state) { + run_pol2_fit(state, aare::Minimizer::VarPro); +} + +// ---------------------------------------------------------------- +// GaussianChargeSharingKb (8 parameters, two exponentials and two erfs per +// point): the most expensive model per evaluation. +// ---------------------------------------------------------------- +static void run_charge_sharing_kb_fit(benchmark::State &state, + aare::Minimizer minimizer) { + constexpr ssize_t n = 200; + const std::vector truth = {20.0, 0.05, 110.0, 6.0, + 1500.0, 0.1, 1.1, 0.1}; + + aare::NDArray x({n}); + aare::NDArray y({n}); + std::mt19937 rng(SEED); + std::normal_distribution noise(0.0, 1.0); + for (ssize_t i = 0; i < n; ++i) { + x[i] = static_cast(i); + const double clean = + aare::model::GaussianChargeSharingKb::eval(x[i], truth); + y[i] = clean + std::sqrt(std::max(clean, 1.0)) * noise(rng); + } + auto xv = x.view(); + auto yv = y.view(); + + const aare::FitModel model( + 0, 500, 0.5, false, minimizer); + + aare::NDArray result; + for (auto _ : state) { + result = aare::fit_pixel(model, + xv, yv); + benchmark::DoNotOptimize(result.data()); + } + state.counters["dMu"] = result(2) - truth[2]; + state.counters["dN"] = result(4) - truth[4]; + state.counters["chi2"] = result(8); +} + +static void BM_FitChargeSharingKbMigrad(benchmark::State &state) { + run_charge_sharing_kb_fit(state, aare::Minimizer::Migrad); +} +static void BM_FitChargeSharingKbFumili(benchmark::State &state) { + run_charge_sharing_kb_fit(state, aare::Minimizer::Fumili); +} +static void BM_FitChargeSharingKbLM(benchmark::State &state) { + run_charge_sharing_kb_fit(state, aare::Minimizer::LevenbergMarquardt); +} +static void BM_FitChargeSharingKbVarPro(benchmark::State &state) { + run_charge_sharing_kb_fit(state, aare::Minimizer::VarPro); +} + +// ---------------------------------------------------------------- +// Data cube through fit_3d on one thread: exposes the per-pixel overhead +// of each minimizer on top of the minimisation itself. +// ---------------------------------------------------------------- +static constexpr ssize_t CUBE_ROWS = 32; +static constexpr ssize_t CUBE_COLS = 32; + +static void run_cube_fit(benchmark::State &state, aare::Minimizer minimizer, + bool compute_errors) { + const auto &tc = get_test_cases()[1]; // Moderate_noise + const double x_min = tc.true_mu - 5.0 * tc.true_sig; + const double x_max = tc.true_mu + 5.0 * tc.true_sig; + const double dx = (x_max - x_min) / (N_POINTS - 1); + const double noise_sigma = tc.noise_frac * tc.true_A; + + aare::NDArray x({N_POINTS}); + aare::NDArray y({CUBE_ROWS, CUBE_COLS, N_POINTS}); + aare::NDArray y_err({CUBE_ROWS, CUBE_COLS, N_POINTS}, + noise_sigma); + std::mt19937 rng(SEED); + std::normal_distribution noise(0.0, noise_sigma); + for (ssize_t i = 0; i < N_POINTS; ++i) + x[i] = x_min + i * dx; + for (ssize_t row = 0; row < CUBE_ROWS; ++row) { + for (ssize_t col = 0; col < CUBE_COLS; ++col) { + for (ssize_t i = 0; i < N_POINTS; ++i) { + const double clean = + tc.true_A * std::exp(-std::pow(x[i] - tc.true_mu, 2) / + (2.0 * std::pow(tc.true_sig, 2))); + y(row, col, i) = clean + noise(rng); + } + } + } + + aare::NDArray par({CUBE_ROWS, CUBE_COLS, 3}); + aare::NDArray err({CUBE_ROWS, CUBE_COLS, 3}); + aare::NDArray chi2({CUBE_ROWS, CUBE_COLS}); + const aare::FitModel model( + 0, 500, 0.5, compute_errors, minimizer); + + for (auto _ : state) { + if (compute_errors) { + aare::fit_3d(model, x.view(), y.view(), y_err.view(), par.view(), + err.view(), chi2.view(), 1); + } else { + aare::fit_3d(model, x.view(), y.view(), aare::NDView{}, + par.view(), aare::NDView{}, chi2.view(), 1); + } + benchmark::DoNotOptimize(par.data()); + } + + state.SetItemsProcessed(state.iterations() * CUBE_ROWS * CUBE_COLS); + state.counters["failed"] = + static_cast(std::count(chi2.begin(), chi2.end(), 0.0)); +} + +static void BM_FitCubeMigrad(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::Migrad, false); +} +static void BM_FitCubeFumili(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::Fumili, false); +} +static void BM_FitCubeLM(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::LevenbergMarquardt, false); +} +static void BM_FitCubeMigradErrors(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::Migrad, true); +} +static void BM_FitCubeFumiliErrors(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::Fumili, true); +} +static void BM_FitCubeLMErrors(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::LevenbergMarquardt, true); +} +static void BM_FitCubeVarPro(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::VarPro, false); +} +static void BM_FitCubeVarProErrors(benchmark::State &state) { + run_cube_fit(state, aare::Minimizer::VarPro, true); +} + +BENCHMARK(BM_FitGausMigrad)->DenseRange(0, 5)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFumili)->DenseRange(0, 5)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausLM)->DenseRange(0, 5)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausVarPro)->DenseRange(0, 5)->Unit(benchmark::kMicrosecond); + +BENCHMARK(BM_FitGausMigradHesse) + ->DenseRange(0, 5) + ->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFumiliErrors) + ->DenseRange(0, 5) + ->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausLMErrors)->DenseRange(0, 5)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausVarProErrors) ->DenseRange(0, 5) ->Unit(benchmark::kMicrosecond); -BENCHMARK(BM_FitGausMinuitGradHesse) - ->DenseRange(0, 5) - ->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFixedSigmaMigrad)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFixedSigmaFumili)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFixedSigmaLM)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitGausFixedSigmaVarPro)->Unit(benchmark::kMicrosecond); + +BENCHMARK(BM_FitScurveMigrad)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveFumili)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveLM)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveVarPro)->Unit(benchmark::kMicrosecond); + +BENCHMARK(BM_FitScurveWeightedMigrad)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveWeightedFumili)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveWeightedLM)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitScurveWeightedVarPro)->Unit(benchmark::kMicrosecond); + +BENCHMARK(BM_FitPol2Migrad)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitPol2Fumili)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitPol2LM)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitPol2VarPro)->Unit(benchmark::kMicrosecond); + +BENCHMARK(BM_FitChargeSharingKbMigrad)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitChargeSharingKbFumili)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitChargeSharingKbLM)->Unit(benchmark::kMicrosecond); +BENCHMARK(BM_FitChargeSharingKbVarPro)->Unit(benchmark::kMicrosecond); + +// fit_3d runs in worker threads, so measure wall time. +BENCHMARK(BM_FitCubeMigrad)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeFumili)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeLM)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeMigradErrors)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeFumiliErrors)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeLMErrors)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeVarPro)->Unit(benchmark::kMillisecond)->UseRealTime(); +BENCHMARK(BM_FitCubeVarProErrors)->Unit(benchmark::kMillisecond)->UseRealTime(); BENCHMARK_MAIN(); diff --git a/benchmarks/ndarray_benchmark.cpp b/benchmarks/ndarray_benchmark.cpp index 456f4d40..b3a3f610 100644 --- a/benchmarks/ndarray_benchmark.cpp +++ b/benchmarks/ndarray_benchmark.cpp @@ -130,4 +130,80 @@ BENCHMARK_F(TwoArrays, MultiplyAddDivideWithIndex)(benchmark::State &st) { } } +BENCHMARK_F(TwoArrays, ScalarMultiplyAddWithOperator)(benchmark::State &st) { + for (auto _ : st) { + // This code gets timed + NDArray res = a * 2 + b; + benchmark::DoNotOptimize(res); + } +} +BENCHMARK_F(TwoArrays, ScalarMultiplyAddWithIndex)(benchmark::State &st) { + for (auto _ : st) { + // This code gets timed + NDArray res(a.shape()); + for (uint32_t i = 0; i < a.size(); i++) { + res(i) = a(i) * 2 + b(i); + } + benchmark::DoNotOptimize(res); + } +} + +BENCHMARK_F(TwoArrays, AddToExistingWithOperator)(benchmark::State &st) { + NDArray res(a.shape()); + for (auto _ : st) { + // This code gets timed + res = a + b; + benchmark::DoNotOptimize(res); + } +} +BENCHMARK_F(TwoArrays, AddToExistingWithIndex)(benchmark::State &st) { + NDArray res(a.shape()); + for (auto _ : st) { + // This code gets timed + for (uint32_t i = 0; i < a.size(); i++) { + res(i) = a(i) + b(i); + } + benchmark::DoNotOptimize(res); + } +} + +// A store to uint8_t or a 64 bit integer can alias the members of NDArray. +// Compare with a loop over raw pointers to check that this does not block +// vectorization. +template void InPlaceAddWithOperator(benchmark::State &st) { + NDArray a({size, size}, 1); + NDArray b({size, size}, 2); + for (auto _ : st) { + a += b; + benchmark::DoNotOptimize(a); + } +} +template void InPlaceAddWithPointer(benchmark::State &st) { + NDArray a({size, size}, 1); + NDArray b({size, size}, 2); + for (auto _ : st) { + T *dst = a.data(); + const T *src = b.data(); + for (ssize_t i = 0, n = a.size(); i < n; ++i) { + dst[i] += src[i]; + } + benchmark::DoNotOptimize(a); + } +} +template void AddToExistingWithOperator(benchmark::State &st) { + NDArray a({size, size}, 1); + NDArray b({size, size}, 2); + NDArray res({size, size}, 0); + for (auto _ : st) { + res = a + b; + benchmark::DoNotOptimize(res); + } +} +BENCHMARK_TEMPLATE(InPlaceAddWithOperator, uint8_t); +BENCHMARK_TEMPLATE(InPlaceAddWithPointer, uint8_t); +BENCHMARK_TEMPLATE(AddToExistingWithOperator, uint8_t); +BENCHMARK_TEMPLATE(InPlaceAddWithOperator, int64_t); +BENCHMARK_TEMPLATE(InPlaceAddWithPointer, int64_t); +BENCHMARK_TEMPLATE(AddToExistingWithOperator, int64_t); + BENCHMARK_MAIN(); \ No newline at end of file diff --git a/benchmarks/ndview_benchmark.cpp b/benchmarks/ndview_benchmark.cpp new file mode 100644 index 00000000..b0653193 --- /dev/null +++ b/benchmarks/ndview_benchmark.cpp @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/NDArray.hpp" +#include "aare/NDView.hpp" +#include + +using aare::NDArray; +using aare::NDView; + +static void BM_CreateNDView(benchmark::State &st) { + NDArray arr{{1024, 1024}, 0}; + for (auto _ : st) { + // This code gets timed + auto res = arr.view(); + benchmark::DoNotOptimize(res); + } +} +BENCHMARK(BM_CreateNDView); \ No newline at end of file diff --git a/cmake/FindSphinx.cmake b/cmake/FindSphinx.cmake index a052e1d4..b980c93d 100644 --- a/cmake/FindSphinx.cmake +++ b/cmake/FindSphinx.cmake @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MPL-2.0 + # Look for an executable called sphinx-build find_program( SPHINX_EXECUTABLE diff --git a/cmake/helpers.cmake b/cmake/helpers.cmake index 595f9530..2fe1d7cf 100644 --- a/cmake/helpers.cmake +++ b/cmake/helpers.cmake @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 function(default_build_type val) if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES) message(STATUS "No build type selected, default to Release") diff --git a/cmake/package_config.cmake b/cmake/package_config.cmake index d018a7f0..0d9c0dcb 100644 --- a/cmake/package_config.cmake +++ b/cmake/package_config.cmake @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MPL-2.0 + # This cmake code creates the configuration that is found and used by # find_package() of another cmake project diff --git a/cmake/project-config.cmake.in b/cmake/project-config.cmake.in index 793a373f..b3f0d569 100644 --- a/cmake/project-config.cmake.in +++ b/cmake/project-config.cmake.in @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 # Config file for @PROJECT_NAME_LOWER@ # # It defines the following variables: diff --git a/conda-recipe/meta.yaml b/conda-recipe/meta.yaml index 66fbe2e3..d2d39b53 100644 --- a/conda-recipe/meta.yaml +++ b/conda-recipe/meta.yaml @@ -26,7 +26,7 @@ requirements: - python - pip - numpy >=2.1 - - scikit-build-core + - scikit-build-core >=0.11 - pybind11 >=3.0.3 - matplotlib # needed in host to solve the environment for run @@ -50,5 +50,11 @@ test: - python -m pytest python/tests about: - license: SPDX-License-Identifier MPL-2.0 + license: MPL-2.0 AND Apache-2.0 AND MIT AND BSD-3-Clause AND LGPL-2.1-or-later + license_file: + - LICENSES/MPL-2.0.txt + - LICENSES/Apache-2.0.txt + - LICENSES/MIT.txt + - LICENSES/LGPL-2.1-or-later.txt + - THIRD-PARTY-NOTICES.txt summary: Data analysis library for hybrid pixel detectors from PSI diff --git a/docs/CMakeLists.txt b/docs/CMakeLists.txt index df589419..d83b6f29 100644 --- a/docs/CMakeLists.txt +++ b/docs/CMakeLists.txt @@ -36,7 +36,7 @@ add_custom_target( docs COMMAND ${DOXYGEN_EXECUTABLE} ${DOXYGEN_OUT} COMMAND - ${SPHINX_EXECUTABLE} -a -b html + ${SPHINX_EXECUTABLE} -W -a -b html -Dbreathe_projects.aare=${CMAKE_CURRENT_BINARY_DIR}/xml -c "${SPHINX_BUILD}" ${SPHINX_BUILD}/src ${SPHINX_BUILD}/html COMMENT "Generating documentation with Sphinx") diff --git a/docs/conf.py.in b/docs/conf.py.in index ad735752..1e0b5a71 100644 --- a/docs/conf.py.in +++ b/docs/conf.py.in @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full diff --git a/docs/src/ClusterFile.rst b/docs/src/ClusterFile.rst index a2ee162b..d764b687 100644 --- a/docs/src/ClusterFile.rst +++ b/docs/src/ClusterFile.rst @@ -1,8 +1,40 @@ ClusterFile ============= +Use ``frames()`` to iterate over complete frames, including empty frames and +frames whose clusters are all rejected by ROI or noise filtering. Each result +keeps its stored frame number; missing frame numbers are not synthesized. + +.. code-block:: cpp + + using ClusterType = aare::Cluster; + aare::ClusterFile file("clusters.clust"); + for (auto &frame : file.frames()) { + process_frame(frame.frame_number(), frame); + } + +Use ``chunks()`` for the constructor's chunk size, or ``chunks(10000)`` to +request a different size for that traversal. The size must be positive and +counts selected clusters after filtering. Chunks may split or combine frames, +so their frame numbers are not reliable per-cluster metadata. Empty chunks are +not yielded; only the final chunk can contain fewer clusters than requested. +Both ranges apply the configured gain map and report incomplete files as +errors, just like the explicit read methods. + +Ranges borrow the file and consume its current position without rewinding. +Constructing a range does not read; ``begin()`` reads the first result and +increment reads the next. Breaking a loop does not read ahead. Use one +traversal at a time and separate files for independent cursors. Switching to +frame reads after a chunk stops partway through a frame raises an error until +the remaining clusters in that frame have been read. + +The file must outlive its ranges and iterators and must not be moved while +they are in use. Iterators are move only and support C++17 range-based loops, +not algorithms requiring copyable STL input iterators. References to a result +last until advancement or iterator destruction. Move a result out with +``auto retained = std::move(frame)`` to retain it. Frame iteration reuses the +current vector's storage when it has not been moved out. + .. doxygenclass:: aare::ClusterFile :members: :undoc-members: - :private-members: - diff --git a/docs/src/NumpyArrays.rst b/docs/src/NumpyArrays.rst new file mode 100644 index 00000000..b837771c --- /dev/null +++ b/docs/src/NumpyArrays.rst @@ -0,0 +1,147 @@ +**************** +Numpy arrays +**************** + +Most Python functions in aare take NumPy arrays as input. The bindings in +``python/src/`` receive them as ``py::array_t`` and wrap the buffer in an +``NDView`` before calling the C++ implementation. No data is copied, so the +C++ code reads and writes the memory that the caller's array owns. This page +explains the choices we make when declaring such an argument and what they +mean for the user. + +The preferred pattern +~~~~~~~~~~~~~~~~~~~~~~~~ + +Declare the argument with only the element type, take it by value, mark it +``noconvert`` in the ``py::arg`` list and let ``make_view`` check the rest: + +.. code-block:: cpp + + m.def( + "push", + [](Pedestal &self, py::array_t frame) { + auto view = make_view_2d(frame); + self.push(view); + }, + py::arg("frame").noconvert()); + +The responsibilities are split between pybind11 and ``make_view``: + +- pybind11 checks that the argument is a NumPy array with exactly the + requested dtype. Anything else fails with a ``TypeError``, and since the + argument is ``noconvert`` pybind11 never tries to build a matching array + from it. +- ``make_view`` (``python/src/np_helper.hpp``) checks that the array has the + expected number of dimensions and is C-contiguous. It raises a + ``ValueError`` with a message that tells the user what to fix, for example + ``Array is not C-contiguous. Use np.ascontiguousarray(arr)``. Writing + functions use ``make_view`` which also rejects read-only arrays. Read-only + input goes through ``make_const_view``. + +The result is a binding that either operates on the caller's memory or +refuses the call. It never silently works on a temporary copy. + +Take the array by value +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``py::array_t`` is a handle to the Python object, not the data. Passing it by +value copies the handle, which increments the reference count of the NumPy +array, and nothing else. The pixel data stays where it is. We therefore +declare the parameter as ``py::array_t frame`` rather than +``py::array_t &frame``: + +- A reference saves one reference-count increment per call, which is not + measurable next to the work done on a frame. +- A by-value parameter is a plain lvalue inside the lambda, so + ``make_view``, which calls ``mutable_data()``, works without further + qualification. A ``const py::array_t &`` does not compile with it. +- It matches the usual pybind11 style for ``py::object`` and its subclasses, + so the binding reads the same as examples in the pybind11 documentation. + +Why not let pybind11 convert +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Without ``noconvert``, pybind11 tries every overload once with conversions +disabled and then once more with conversions enabled. In the second pass a +``py::array_t`` argument accepts anything NumPy can turn into an array of +type ``T``: an array with another dtype, a non-contiguous array, or even a +Python list. pybind11 creates a new array, hands the converted copy to the +binding and drops it when the call returns. For detector data this has +several consequences: + +- The copy costs time and memory, and the user gets no indication that it + happened. For a stack of frames this can be gigabytes. +- Writes are lost. A function that fills or updates its input, for example + a pedestal subtraction in place, writes into the temporary copy. +- Values are cast without warning. ``forcecast`` allows unsafe casts, so a + ``float64`` frame is truncated to ``uint16`` instead of being rejected. +- Overloads stop working as intended. ``apply_calibration`` is registered for + both ``double`` and ``float``. With conversions allowed the first overload + would accept everything by casting and the ``float`` overload would never be + chosen. + +Argument conversion is only worth it for small inputs where convenience beats +control, for example the parameter vector passed to a fit. Even then the +function must not write into the argument. + +.. note:: + + When a binding intentionally lets pybind11 convert, say so with a comment + next to the argument. Otherwise a missing ``noconvert`` looks like an + oversight and the next person to touch the binding will add it. + + .. code-block:: cpp + + // Conversion allowed: small read-only vector, accept lists and any dtype + py::array_t par + +What the template flags mean +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``py::array_t`` takes an optional second template argument. The +flags do two things: they add requirements that the array must fulfil for the +*non-converting* check to pass, and they tell NumPy what to produce when a +conversion *is* allowed. + +``py::array::c_style`` + The array must be C-contiguous. With ``noconvert`` a non-contiguous array + is rejected by pybind11 with a ``TypeError`` listing the accepted + signatures. Without ``noconvert`` pybind11 makes a contiguous copy. + Since ``make_view`` already enforces contiguity and gives a more helpful + ``ValueError``, the flag is redundant in the preferred pattern. It is + still present in some older bindings and is harmless there. + +``py::array::f_style`` + Same as above for Fortran order. ``NDView`` assumes C order, so we do not + use it. + +``py::array::forcecast`` + Allows NumPy to perform unsafe casts, such as ``float64`` to ``uint16``, + when converting. It is the **default** value of the template argument, so + ``py::array_t`` and ``py::array_t`` + are the same type. The flag has no effect together with ``noconvert`` + because no conversion takes place. It only matters when conversions are + allowed, where it decides whether a lossy cast is accepted or the call is + rejected. + +The common spelling ``py::array_t`` +therefore means: when converting, produce a contiguous array of type ``T`` and +cast whatever you were given. Combined with ``noconvert`` it collapses to +"C-contiguous array of exactly type ``T``", which is what the plain +``py::array_t`` plus ``make_view`` already gives us. + +Summary +~~~~~~~~~~~~~~~~~~ + +- Declare the argument as ``py::array_t`` by value and add + ``.noconvert()`` to its ``py::arg``. The handle is reference counted, so + passing it by value does not copy the data. +- Create the view with ``make_view_1d/2d/3d`` or ``make_const_view_*`` and + let it validate dimensions and contiguity. Add an explicit check only when + the binding accepts several shapes, as ``apply_calibration`` does for 2D + and 3D pedestals. +- Return data to Python with ``return_image_data`` or ``return_vector`` so that + the NumPy array owns the C++ allocation. +- Only allow conversion for small, read-only inputs where accepting lists or + other dtypes is a real convenience for the user, and mark it with a comment + in the code. diff --git a/docs/src/RawFile.rst b/docs/src/RawFile.rst index 4b7e03ae..8ecd30e9 100644 --- a/docs/src/RawFile.rst +++ b/docs/src/RawFile.rst @@ -1,8 +1,48 @@ RawFile =============== +Reading requires frame padding to be enabled +or ``Frame Discard Policy`` to be ``discardpartial``. +The constructor checks the +parsed master metadata before opening data subfiles and throws +``std::runtime_error`` with the master path if this requirement is not met. +This applies to JSON and legacy ``.raw`` masters, including when opened through +``aare::File``. ``RawMasterFile`` can still be used to inspect their metadata. + +``total_frames()`` is the minimum actual frame count across all selected +``RawSubFile`` objects in all ROIs. Each subfile count includes its complete +series of ``.raw`` files and is determined when the file is opened. Disabled +UDP ports are excluded. An empty subfile makes the count zero; opening a file +with no selected subfiles raises an error. + +A warning is logged once on opening if the subfile counts differ or their +minimum differs from ``master().frames_in_file()``. It reports the subfile count +range, the master count, and the count used for reading. The master metadata +is preserved. Its expected frame count does not affect the count used for +reading. + +Frame, batch, ROI, and frame-number reads use this minimum as their bound. +Frame-number synchronization may still fail within this bound if detector +frame numbers differ between subfiles. + +Read errors include the attempted zero-based frame index and the failing file +path. Errors from a data subfile identify that subfile's path and preserve +the underlying failure reason without adding a master-file wrapper or C++ +source location. Out-of-range subfile errors also report the available frame +count across that subfile's series. This applies to frame, batch, ROI, and +frame-number reads, including reads through ``aare::File``. + +The frame index is the position in the file, not the frame number recorded by +the detector. Batch errors report the index of the frame that failed. + +Synchronization errors identify the last ``.raw`` data file read for the module +that ran out of frames. Requests rejected as out of range before accessing a +subfile report the master path, since there is no single failing data file. +The top-level frame bounds error states the number of frames in the file and +that indices are zero-based. + .. doxygenclass:: aare::RawFile :members: :undoc-members: - :private-members: \ No newline at end of file + :private-members: diff --git a/docs/src/index.rst b/docs/src/index.rst index d60fc78e..4f0151b7 100644 --- a/docs/src/index.rst +++ b/docs/src/index.rst @@ -30,6 +30,7 @@ AARE python/file/index python/experimental/index python/histogram/index + python/pedestal/index pyFit @@ -67,3 +68,4 @@ AARE Philosophy Workflow Tests + NumpyArrays diff --git a/docs/src/pyFit.rst b/docs/src/pyFit.rst index 257f7414..142b1ca6 100644 --- a/docs/src/pyFit.rst +++ b/docs/src/pyFit.rst @@ -4,11 +4,18 @@ Fitting .. py:currentmodule:: aare Aare fits one-dimensional scans and three-dimensional pixel data with -Minuit2. Create a model object and call its :meth:`fit` method:: +Minuit2's Migrad and Fumili or with the built-in Levenberg-Marquardt and +variable projection solvers. Create a model object and call its :meth:`fit` +method:: model = Gaussian(compute_errors=True) result = model.fit(x, y, y_err) +``y_err`` holds the standard deviation of each point and weights the +residuals; points with ``y_err == 0`` are left out of the fit, which skips +empty histogram bins when ``y_err = sqrt(y)``. Without ``y_err`` every point +has unit weight. + The model object is also callable, which evaluates it at the supplied points:: fitted_y = model(x, result["par"]) @@ -20,11 +27,127 @@ accepts the same model objects when a functional interface is preferred. For three-dimensional data, pass an array with shape ``(rows, columns, scan_points)`` and select the worker count with -``n_threads``:: +``n_threads`` (default 4):: result = model.fit(x, image_data, image_errors, n_threads=8) -The result dictionary contains ``par`` and ``chi2``. It also contains -``par_err`` when ``compute_errors`` is enabled. +The result dictionary contains ``par`` and ``chi2``, with shapes ``(npar,)`` +and ``(1,)`` for a scan or ``(rows, columns, npar)`` and ``(rows, columns)`` +for a cube. It also contains ``par_err`` when ``compute_errors`` is enabled. +A pixel that fails to converge returns zeros for all three. + +.. note:: + + The fit works on ``float64`` data. If ``x``, ``y`` or ``y_err`` is not + already a C-contiguous ``float64`` array, the Python bindings convert it + before fitting, which allocates a full copy. Integer or ``float32`` + input therefore works, but for a large three-dimensional ``y`` the + copy can be several times the size of the original array. Pass + ``float64`` arrays to avoid it. A ``float64`` C-contiguous array is used + in place without copying. + +Choosing the minimizer +~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + The built-in VarPro solver is up to ten times faster than Migrad. It and + the built-in Levenberg-Marquardt solver do not offer Minuit2's strategy + setting or Hesse errors. Try VarPro first and fall back to Migrad if you + run into problems. Minuit2's Fumili and the built-in Levenberg-Marquardt + solver are included for evaluation and may turn out to be redundant. + +Fits use Minuit2's Migrad by default. Three alternatives share the same +interface and are selected with the ``minimizer`` argument or property:: + + model = Gaussian(minimizer=Minimizer.LevenbergMarquardt) + model.minimizer = Minimizer.Fumili + +- ``Minimizer.Migrad``: Minuit2's variable-metric minimizer, driven by the + analytic gradient of the chi-squared. Parameter errors come from Hesse. +- ``Minimizer.Fumili``: Minuit2's Gauss-Newton minimizer, which controls its + steps with a trust region by default. It takes the gradient and the + linearised Hessian from the analytic derivatives of the model and typically + needs far fewer function evaluations than Migrad. The linearised Hessian at + the minimum also provides the parameter covariance, so no Hesse step runs. + + Parameter limits make Minuit2 map the bounded external parameters onto + unbounded internal ones. On an active limit the derivative of that mapping + vanishes, and Fumili's linearised Hessian, which contains only this first + derivative, loses the curvature along the bounded direction. The Hessian + can then become singular and the iteration fails to converge. Pixels for + which Fumili does not return a valid minimum are refitted with Migrad. + +- ``Minimizer.LevenbergMarquardt``: a built-in damped Gauss-Newton solver with + the analytic Jacobian of the model. It reflects steps at parameter limits + and reuses its buffers between pixels, so data cubes are fitted without + per-pixel allocations. Parameter errors are ``sqrt(diag((J^T W J)^-1))`` + with ``W`` the weights from ``y_err``. +- ``Minimizer.VarPro``: a built-in variable projection solver. + Every bundled model is linear in some of its parameters: every parameter of + a polynomial, the amplitude of a Gaussian, the amplitude and plateau of + ``GaussianErfcPlateau``, the background and amplitude of the charge-sharing + models, and the background, amplitude and charge-sharing term of the + S-curves. Each trial point solves those exactly and the Levenberg-Marquardt + iteration runs over the remaining nonlinear parameters only, so it needs + fewer evaluations than LevenbergMarquardt and is several times faster than + Migrad. Start values of the linear parameters are not used, so + ``SetParameter`` has no effect on them; ``FixParameter`` does. A pixel + whose linear solution violates a limit on a linear parameter, or that does + not converge, is refitted with LevenbergMarquardt with a fresh + ``max_calls`` budget. Parameter errors are the same Gauss-Newton estimates. + On data that barely constrains a model, VarPro can settle at a point where + an amplitude is driven to zero and the parameters attached to it lose + their meaning; such a pixel reports a valid fit with a poor ``chi2``. + +All four converge to the same minimum on well-behaved data; the ``[fit]`` +test suite checks that. The bundled ``fit_benchmark`` compares their speed on +Gaussian peaks, S-curves, polynomials, charge-sharing spectra and a data +cube. On ill-conditioned data (few points, many parameters, or parameters +that nearly trade off against each other) the minimizers stop at different +points of a flat chi-squared valley, and Migrad in particular can settle on a +parameter limit with a higher ``chi2`` than the others. Compare ``chi2`` +rather than the parameters when they disagree, fix or bound the poorly +constrained parameters, and provide start values for the rest. + +Parameters and errors +~~~~~~~~~~~~~~~~~~~~~ + +Start values are estimated from the data. ``SetParameter`` overrides the +estimate, ``FixParameter`` keeps a parameter at a given value, and +``SetParLimits`` restricts it to a range where ``-inf`` and ``inf`` leave a +side open (the Minuit2 minimizers turn an open side into a wide finite one); +free start values are clamped into their limits. The constructor arguments +control the minimizer: + +- ``max_calls`` (default 100) caps the work per pixel: Minuit2 function calls + for Migrad and Fumili, model evaluations for LevenbergMarquardt and VarPro, + which spend one evaluation per iteration (two when a step crosses a limit). + ``0`` selects Minuit's default max calls of ``200 + 100 * npar + 5 * npar**2``. + A pixel that does not converge within the budget returns zeros; the six- + and eight-parameter models on noisy data need several hundred calls, so + raise the default when such pixels fail. +- ``tolerance`` is the EDM tolerance in Minuit's convention. Migrad, + LevenbergMarquardt and VarPro stop when the estimated distance + to the minimum is below ``0.002 * tolerance``, Fumili below + ``1e-4 * tolerance``. The EDM is measured in units of ``chi2``, not + relative to the data: without ``y_err`` every sample counts with an error + of 1, so a fit of small values (amplitudes well below 1) meets the + tolerance while still far from the minimum. Pass ``y_err`` or lower + ``tolerance`` for such data. +- ``strategy`` is the Minuit2 strategy (0 is fast, 1 is Minuit2's default). + LevenbergMarquardt and VarPro ignore it. +- ``compute_errors`` adds ``par_err`` to the result. Fixed parameters and + parameters that end on a limit report an error of 0 with every minimizer. + Migrad's Hesse errors include the residual-weighted second derivatives of + the model, which the Gauss-Newton estimates of the other three omit; the + two agree when the model describes the data (``chi2`` per degree of freedom + near 1) and the Gauss-Newton errors come out smaller for a poorer fit. + +Unknown parameter names raise ``RuntimeError``, bad indices ``IndexError``, +and ``SetParLimits`` rejects a lower limit at or above the upper one. .. autofunction:: fit + +.. autoclass:: Minimizer + :members: Migrad, Fumili, LevenbergMarquardt, VarPro diff --git a/docs/src/python/cluster/pyClusterVector.rst b/docs/src/python/cluster/pyClusterVector.rst index 1aaf1b78..fc9508c3 100644 --- a/docs/src/python/cluster/pyClusterVector.rst +++ b/docs/src/python/cluster/pyClusterVector.rst @@ -3,17 +3,16 @@ ClusterVector ================ -The ClusterVector, holds clusters from the ClusterFinder. Since it is templated -in C++ we use a suffix indicating the type of cluster it holds. The suffix follows -the same pattern as for ClusterFile i.e. ``ClusterVector_Cluster3x3i`` -for a vector holding 3x3 integer clusters. +A ClusterVector stores fixed-size clusters contiguously. Since it is templated +in C++, each bound class has a suffix indicating the cluster type. The suffix +follows the same pattern as ClusterFile; for example, +``ClusterVector_Cluster3x3i`` stores 3x3 clusters with 32-bit integer pixels. -At the moment the functionality from python is limited and it is not supported -to push_back clusters to the vector. The intended use case is to pass it to -C++ functions that support the ClusterVector or to view it as a numpy array. +The intended use case is to pass a ClusterVector to C++ functions that support +it or to view it as a NumPy array. -**View ClusterVector as numpy array** +**View ClusterVector as a NumPy array** .. code:: python @@ -25,7 +24,16 @@ C++ functions that support the ClusterVector or to view it as a numpy array. clusters = np.array(cluster_vector) # Avoid copying the data by passing copy=False - clusters = np.array(cluster_vector, copy = False) + clusters = np.array(cluster_vector, copy=False) + +.. warning:: + + A NumPy array created with ``copy=False`` is a view of the ClusterVector's + current storage. Do not call ``push_back`` or otherwise change the + ClusterVector while using the view. A ``push_back`` that reallocates the + backing buffer leaves existing NumPy views pointing to invalid memory, and + appending without reallocation does not update their shape. Use + ``copy=True`` if the ClusterVector may change after creating the array. .. py:currentmodule:: aare @@ -35,7 +43,8 @@ C++ functions that support the ClusterVector or to view it as a numpy array. :undoc-members: :inherited-members: -Below is the API of the ClusterVector_Cluster3x3i but all variants share the same API. +Below is the API of ``ClusterVector_Cluster3x3i``. All variants share the same +API. .. autoclass:: aare._aare.ClusterVector_Cluster3x3i :special-members: __init__, __call__ @@ -45,14 +54,39 @@ Below is the API of the ClusterVector_Cluster3x3i but all variants share the sam :inherited-members: -**Free Functions:** +**Free Functions:** -.. autofunction:: reduce_to_3x3 +.. py:function:: hitmap(image_size, clusters) :noindex: - Reduce a single Cluster to 3x3 by taking the 3x3 subcluster with highest photon energy. + Count cluster centers into an ``int32`` image. ``image_size`` is given as + ``(rows, columns)``, and output element ``[y, x]`` contains the number of + photon hits at that coordinate. Out-of-bounds hits are ignored. All registered + ClusterVector variants are accepted. -.. autofunction:: reduce_to_2x2 + :param tuple[int, int] image_size: Shape of the output image. + :param ClusterVector clusters: Clusters whose centers are counted. + :return: Hit counts with shape ``image_size``. + :rtype: numpy.ndarray + +.. py:function:: reduce_to_3x3(clustervector) :noindex: - Reduce a single Cluster to 2x2 by taking the 2x2 subcluster with highest photon energy. + Return a new vector containing the central 3x3 block of every input cluster. + Cluster order, coordinates, frame number, and pixel dtype are preserved. + + :param ClusterVector clustervector: Input clusters with both dimensions at + least 3 and at least one dimension greater than 3. + :return: Reduced 3x3 clusters. + :rtype: ClusterVector + +.. py:function:: reduce_to_2x2(clustervector) + :noindex: + + Return a new vector containing the highest-sum center-adjacent 2x2 block of + every input cluster. Cluster order, coordinates, frame number, and pixel + dtype are preserved. + + :param ClusterVector clustervector: Input clusters of size 2x2 or larger. + :return: Reduced 2x2 clusters. + :rtype: ClusterVector diff --git a/docs/src/python/file/pyClusterFile.rst b/docs/src/python/file/pyClusterFile.rst index cd391e0e..b406cab3 100644 --- a/docs/src/python/file/pyClusterFile.rst +++ b/docs/src/python/file/pyClusterFile.rst @@ -1,26 +1,78 @@ ClusterFile -============ +=========== +The :func:`ClusterFile` factory is the main interface for reading and writing +cluster files. Use ``mode="r"`` to read, ``mode="w"`` to truncate and +write, or ``mode="a"`` to append. -The :class:`ClusterFile` class is the main interface to read and write clusters in aare. Unfortunately the -old file format does not include metadata like the cluster size and the data type. This means that the -user has to know this information from other sources. Specifying the wrong cluster size or data type -will lead to garbage data being read. +The format does not store the cluster dimensions, value type, coordinate type, +padding, or byte order. The ``cluster_size`` and ``dtype`` arguments must match +the writer; otherwise the bytes are interpreted incorrectly. + +Use ``frames()`` when frame boundaries and the frame number matter: + +.. code-block:: python + + with ClusterFile("clusters.clust") as file: + for frame in file.frames(): + print(frame.frame_number, frame.size) + +Each stored frame is yielded, including empty frames and frames whose clusters +are all rejected by filtering. Missing frame numbers are not synthesized. +The explicit ``read_frame()`` method returns one frame or ``None`` at EOF. + +Use ``chunks()`` to iterate over chunks of ``chunk_size`` clusters. Per default +the chunk size defined in the constructor is used. Alternatively one can pass +an explicit chunk size to the function for that specific traversal: + +.. code-block:: python + + with ClusterFile("clusters.clust", chunk_size=1000) as file: + for clusters in file.chunks(10000): + process(clusters) + +Chunk sizes must be positive and count selected clusters after filtering. +``file.chunks(10000)`` does not overwrite the constructor argument ``chunk_size`` +. Iterating directly +over ``file`` or calling ``next(file)`` still reads chunks using that default. +Chunks and ``read_clusters(n_clusters)`` may split or combine frames, so their +frame numbers are not reliable per-cluster metadata. Empty chunks are not +yielded. +Chunk reads and iteration raise an error if they encounter an incomplete frame +header or cluster record, including when ROI or noise filtering is enabled. +They return fewer clusters than requested only at a clean end of file. + +Both iterators consume the current file position without rewinding. Creating +an iterator does not read; each ``next()`` reads one result without prefetching +the following result. Use one traversal at a time and separate readers for +independent cursors. Switching to frames after a chunk stops partway through a +frame raises an error until the remaining clusters in that frame have been +read. Exhausted iterators continue to raise ``StopIteration``. + +Iterators keep their file object alive, but explicitly closing the file or +leaving its ``with`` block prevents further reads. Every yielded +``ClusterVector`` owns its storage and can be retained after iteration advances +or the file closes. NumPy views also remain valid across those operations; +as usual, resizing or modifying the allocation of their backing vector can +invalidate them. + +When a gain map is configured, it is applied to every cluster whose complete +footprint lies inside the map. Clusters whose footprint extends beyond the +gain-map boundaries remain in the returned cluster vector, but all their data +values are set to zero. .. py:currentmodule:: aare -.. autoclass:: ClusterFile - :members: - :undoc-members: - :inherited-members: +.. autofunction:: ClusterFile -Below is the API of the ClusterFile_Cluster3x3i but all variants share the same API. +Below is the API of ``ClusterFile_Cluster3x3i``; all compiled variants share the +same API. .. autoclass:: aare._aare.ClusterFile_Cluster3x3i :special-members: __init__ :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/file/pyCtbRawFile.rst b/docs/src/python/file/pyCtbRawFile.rst index 75d83550..f0676e6d 100644 --- a/docs/src/python/file/pyCtbRawFile.rst +++ b/docs/src/python/file/pyCtbRawFile.rst @@ -6,6 +6,10 @@ Read analog, digital and transceiver samples from a raw file containing data from the Chip Test Board. Uses :mod:`aare.transform` to decode the data into a format that the user can work with. +``read_frame(index)`` reads the frame at the zero-based index across the raw +subfiles and advances the position to ``index + 1``. A subsequent +``read_frame()`` reads the following frame, including across subfile boundaries. + .. code:: python import aare @@ -22,4 +26,4 @@ data into a format that the user can work with. :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/file/pyRawFile.rst b/docs/src/python/file/pyRawFile.rst index b81f4138..a128d2fe 100644 --- a/docs/src/python/file/pyRawFile.rst +++ b/docs/src/python/file/pyRawFile.rst @@ -3,8 +3,45 @@ RawFile .. py:currentmodule:: aare +Reading requires frame padding to be enabled +or ``Frame Discard Policy`` to be ``discardpartial``. Construction raises +``RuntimeError`` with the master path before opening data subfiles if this +requirement is not met. This applies to JSON and legacy ``.raw`` masters, +including when opened through ``File``. ``RawMasterFile`` can still be used +to inspect their metadata. + +``total_frames`` and ``len(reader)`` report the minimum actual frame count +across all selected subfiles and ROIs, including each subfile's complete +``.raw`` series. ``read()`` uses this count, and ``read_n()`` and +``read_n_with_roi()`` limit batches to the remaining frames. Counts are +determined when the reader is constructed. +An empty subfile makes the count zero, and construction fails if no subfiles +are selected. + +The C++ logger prints one warning to stderr on opening if the subfile counts +differ or their minimum differs from ``reader.master.frames_in_file``. The +warning includes the count range, the recorded master count, and the count +used for reading. Master metadata is preserved; its expected frame count +does not affect reading bounds. Detector frame-number mismatches can still +cause synchronization errors within these bounds. + +Reading errors raise ``RuntimeError`` with the attempted zero-based frame index +and the failing file path. If a data subfile fails, its path and the underlying +failure reason are reported. Out-of-range subfile errors include the available frame count across +that subfile's series. This applies to ``read_frame()``, ``read_n()``, +and the ROI reading methods. The frame index is the position in the file, not +the frame number recorded by the detector. + +Synchronization errors also identify the last ``.raw`` data file read for the +module that ran out of frames. An out-of-range request rejected before accessing +a subfile reports the master path. + +For a file containing five frames, the valid indices are 0 through 4. Attempting +to read index 5 raises an error stating: ``Frame index 5 is out of range: file +contains 5 frames (indices are zero-based)``. + .. autoclass:: RawFile :members: :undoc-members: :show-inheritance: - :inherited-members: \ No newline at end of file + :inherited-members: diff --git a/docs/src/python/histogram/pyPedestalTrackingPixelHistogram.rst b/docs/src/python/histogram/pyPedestalTrackingPixelHistogram.rst index 18d438cd..66fe092d 100644 --- a/docs/src/python/histogram/pyPedestalTrackingPixelHistogram.rst +++ b/docs/src/python/histogram/pyPedestalTrackingPixelHistogram.rst @@ -15,6 +15,14 @@ Use ``push_pedestal_no_update()`` to seed the pedestal estimate, then asynchronous fills are drained by ``flush()``, and snapshot methods such as ``values()`` and ``pedestal_mean()`` return numpy arrays. +``fill_from_file()`` uses parallel file-reader workers and a double-buffered +pipeline. After the initial batch has been read, histogram processing of one +batch overlaps reading of the next. ``reader_threads`` and +``reader_chunk_size`` tune the I/O stage independently of the histogram worker +count. Two fixed-capacity buffers are allocated once and reused by alternating +their read and histogram roles. Their approximate memory use is ``2 * +reader_threads * reader_chunk_size * rows * cols * sizeof(uint16)``. + .. py:currentmodule:: aare .. autoclass:: PedestalTrackingPixelHistogram diff --git a/docs/src/python/pedestal/index.rst b/docs/src/python/pedestal/index.rst new file mode 100644 index 00000000..4e0a05e4 --- /dev/null +++ b/docs/src/python/pedestal/index.rst @@ -0,0 +1,9 @@ +Pedestal +======== + +.. toctree:: + :caption: Pedestal + :maxdepth: 1 + + pyFastPedestal + pyPedestal diff --git a/docs/src/python/pedestal/pyFastPedestal.rst b/docs/src/python/pedestal/pyFastPedestal.rst new file mode 100644 index 00000000..fcc57d33 --- /dev/null +++ b/docs/src/python/pedestal/pyFastPedestal.rst @@ -0,0 +1,86 @@ +FastPedestal +============ + +``FastPedestal`` calculates a running mean and standard deviation for each pixel in a +series of frames. The python binding only exposes ``uint16`` input but the underlying +C++ class is templated. Initialize it with ``n_samples`` frames using +``add_init_frame()``. Once ``ready`` is true, use ``push_ema()`` to update the exponential +moving average initialized by the mean and with smoothing factor 1/n_samples. + +.. warning:: + + FastPedestal is not usable until you have added ``n_samples`` initial frames with ``add_init_frame(raw)``. + You can check the state with ``ready``. + +The public factory selects the bound C++ specialization from ``dtype``: + +* ``numpy.float64`` creates ``FastPedestal_d`` +* ``numpy.float32`` creates ``FastPedestal_f`` +* ``numpy.int16`` creates ``FastPedestal_i16`` + +Internal moments and variance are calculated in double precision. Variance is +private and stays in double precision through the square root; the cached mean +and on-demand standard deviation are returned in the specified type. Negative +variance caused by floating-point roundoff is clamped to zero. + +Factory +------- + +.. py:currentmodule:: aare + +.. autofunction:: FastPedestal + +Loading from a file +------------------- + +``FastPedestal.from_file()`` initializes the pedestal from ``n_samples`` +frames after ``skip_first``, then applies steady-state updates for any frames +remaining in the file. The input frames must contain ``uint16`` data; ``dtype`` +selects the output type of the pedestal statistics. + +.. autofunction:: aare.FastPedestal.from_file + +.. code-block:: python + + pedestal = FastPedestal.from_file( + "frames.npy", n_samples=100, skip_first=10, dtype=np.float32 + ) + +Example +------- + +.. code-block:: python + + import numpy as np + from aare import FastPedestal + + pedestal = FastPedestal(512, 1024, n_samples=100, dtype=np.float32) + + # Initialize with n_samples frames + for frame in initialization_frames: + pedestal.add_init_frame(frame) + + # Now we can push a frame for pedestal update + if pedestal.ready: + pedestal.push_ema(next_frame) + + # Mean and std are also ready + mean = pedestal.mean() + noise = pedestal.std() + + # Direct pedestal subtraction is also supported + for frame in raw_data: + image = frame - pedestal + +Complete API +------------ + +The API below is for the ``float64`` specialization. All dtype variants share +the same API. + +.. autoclass:: aare._aare.FastPedestal_d + :special-members: __init__ + :members: + :undoc-members: + :show-inheritance: + :inherited-members: diff --git a/docs/src/python/pedestal/pyPedestal.rst b/docs/src/python/pedestal/pyPedestal.rst new file mode 100644 index 00000000..00000b19 --- /dev/null +++ b/docs/src/python/pedestal/pyPedestal.rst @@ -0,0 +1,69 @@ +Pedestal +======== + +``Pedestal`` calculates a running mean and population standard deviation for +each pixel in a series of ``uint16`` frames. ``push()`` updates the cached mean +immediately; ``std()`` calculates the noise from the current statistics. + +``push()`` and ``push_with_threshold()`` require C-contiguous, two-dimensional +NumPy frames with dtype ``uint16`` and shape matching the pedestal. +``push_with_threshold()`` also requires a C-contiguous, two-dimensional +threshold array with the pedestal's output dtype and shape. Noncontiguous +inputs raise ``TypeError``; inputs with the wrong number of dimensions raise +``ValueError``. Mismatched frame or threshold shapes raise ``RuntimeError``. +Use ``numpy.ascontiguousarray()`` to copy a sliced or transposed array into the +required layout when needed. + +Internal sums and sums of squares always use ``float64``. Three +specializations are available from :mod:`aare` for the mean and standard +deviation output types: + +* ``Pedestal_d`` returns ``float64`` +* ``Pedestal_f`` returns ``float32`` +* ``Pedestal_i16`` returns ``int16`` + +The public ``Pedestal`` factory selects the specialization from ``dtype``, +defaulting to ``numpy.float64``. + +Constructor dimensions and ``n_samples`` must be positive integers. Negative +values raise ``TypeError`` and zero values raise ``RuntimeError``. +Internally, negative variance caused by floating-point roundoff is clamped to +zero before taking its square root, keeping the standard deviation finite for +nearly constant inputs. Only the final standard deviation is converted to the +output dtype. + +Factory +------- + +.. py:currentmodule:: aare + +.. autofunction:: Pedestal + +Example +------- + +.. code-block:: python + + import numpy as np + from aare import Pedestal + + pedestal = Pedestal(512, 1024, n_samples=100, dtype=np.float32) + + for frame in initialization_frames: + pedestal.push(frame) + + mean = pedestal.mean() + noise = pedestal.std() + +Complete API +------------ + +The API below is for the ``float64`` specialization. All dtype variants share +the same API. + +.. autoclass:: aare._aare.Pedestal_d + :special-members: __init__ + :members: + :undoc-members: + :show-inheritance: + :inherited-members: diff --git a/etc/add_license.py b/etc/add_license.py index f43b6cdb..ca02ba43 100644 --- a/etc/add_license.py +++ b/etc/add_license.py @@ -4,13 +4,17 @@ import fnmatch import os from pathlib import Path -CPP_PATTERNS = ["*.h", "*.hpp", "*.cpp"] -PY_PATTERNS = ["*.py"] -CMAKE_PATTERNS = ["CMakeLists.txt"] +CPP_PATTERNS = ["*.h", "*.hpp", "*.cpp", "*.hpp.in"] +PY_PATTERNS = ["*.py", "*.py.in"] +CMAKE_PATTERNS = ["CMakeLists.txt", "*.cmake", "*.cmake.in"] FILE_PATTERNS = CPP_PATTERNS + PY_PATTERNS + CMAKE_PATTERNS LICENSE_TEXT = "SPDX-License-Identifier: MPL-2.0" +# Text found in license notices of code copied from other projects. Such files +# keep their original license and must be tagged by hand. +THIRD_PARTY_MARKERS = ["Copyright", "Permission is hereby granted", "Licensed under"] + def get_comment_prefix(filename: str) -> str | None: if any(fnmatch.fnmatch(filename, p) for p in CPP_PATTERNS): @@ -43,6 +47,10 @@ def process_file(filepath: Path) -> bool: if any("SPDX-License-Identifier" in line for line in lines): return False + if any(marker in line for line in lines for marker in THIRD_PARTY_MARKERS): + print(f"⚠️ Skipped {filepath}: contains a license notice, tag it by hand") + return False + insert_index = 0 # For Python, keep shebang on the very first line @@ -50,6 +58,11 @@ def process_file(filepath: Path) -> bool: if lines[0].startswith("#!"): insert_index = 1 + # cmake-format merges consecutive comment lines into one paragraph + if any(fnmatch.fnmatch(filename, p) for p in CMAKE_PATTERNS): + if len(lines) > insert_index and lines[insert_index].startswith("#"): + lines.insert(insert_index, "\n") + lines.insert(insert_index, license_line) try: @@ -69,7 +82,7 @@ def main() -> None: parser.add_argument( "path", help="Root directory to recursively process " - "(*.h, *.cpp, *.py, and CMakeLists.txt).", + "(C++, Python, and CMake sources, and their .in templates).", ) args = parser.parse_args() diff --git a/etc/update_version.py b/etc/update_version.py index 78eea2a3..a939d9f7 100644 --- a/etc/update_version.py +++ b/etc/update_version.py @@ -1,5 +1,4 @@ # SPDX-License-Identifier: MPL-2.0 -# Copyright (C) 2021 Contributors to the Aare Package """ Script to update VERSION file with semantic versioning if provided as an argument, or with 0.0.0 if no argument is provided. """ diff --git a/include/aare/ArrayExpr.hpp b/include/aare/ArrayExpr.hpp index 368f691f..8520208d 100644 --- a/include/aare/ArrayExpr.hpp +++ b/include/aare/ArrayExpr.hpp @@ -5,6 +5,8 @@ #include #include #include +#include +#include namespace aare { @@ -20,80 +22,95 @@ template class ArrayExpr { } }; -template -class ArrayAdd : public ArrayExpr, Ndim> { - const A &arr1_; - const B &arr2_; - - public: - ArrayAdd(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { - assert(arr1.size() == arr2.size()); - } - auto operator[](int i) const { return arr1_[i] + arr2_[i]; } - size_t size() const { return arr1_.size(); } - std::array shape() const { return arr1_.shape(); } -}; - -template -class ArraySub : public ArrayExpr, Ndim> { - const A &arr1_; - const B &arr2_; - - public: - ArraySub(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { - assert(arr1.size() == arr2.size()); - } - auto operator[](int i) const { return arr1_[i] - arr2_[i]; } - size_t size() const { return arr1_.size(); } - std::array shape() const { return arr1_.shape(); } -}; - -template -class ArrayMul : public ArrayExpr, Ndim> { - const A &arr1_; - const B &arr2_; - - public: - ArrayMul(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { - assert(arr1.size() == arr2.size()); - } - auto operator[](int i) const { return arr1_[i] * arr2_[i]; } - size_t size() const { return arr1_.size(); } - std::array shape() const { return arr1_.shape(); } -}; - -template -class ArrayDiv : public ArrayExpr, Ndim> { - const A &arr1_; - const B &arr2_; - - public: - ArrayDiv(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { - assert(arr1.size() == arr2.size()); - } - auto operator[](int i) const { return arr1_[i] / arr2_[i]; } - size_t size() const { return arr1_.size(); } - std::array shape() const { return arr1_.shape(); } -}; - -template -auto operator+(const ArrayExpr &arr1, const ArrayExpr &arr2) { - return ArrayAdd, ArrayExpr, Ndim>(arr1, arr2); +/** + * @brief What an expression stores for one of its operands. An NDArray is + * replaced by a view, everything else is copied. The expression then holds its + * data pointers by value, so the compiler can keep them in registers even if + * the evaluation writes to a type that may alias them (e.g. uint8_t). + */ +template +auto operand(const ArrayExpr &expr) { + if constexpr (E::is_leaf) + return static_cast(expr).view(); + else + return static_cast(expr); } -template -auto operator-(const ArrayExpr &arr1, const ArrayExpr &arr2) { - return ArraySub, ArrayExpr, Ndim>(arr1, arr2); +template +class ArrayBinaryOp : public ArrayExpr, Ndim> { + A arr1_; + B arr2_; + + public: + ArrayBinaryOp(const A &arr1, const B &arr2) : arr1_(arr1), arr2_(arr2) { + assert(arr1.shape() == arr2.shape()); + } + auto operator[](size_t i) const { return Op{}(arr1_[i], arr2_[i]); } + size_t size() const { return arr1_.size(); } + std::array shape() const { return arr1_.shape(); } +}; + +/** @brief Scalar operand, takes size and shape from the expression it is + * combined with. */ +template +class ArrayScalar : public ArrayExpr, Ndim> { + T value_; + size_t size_; + std::array shape_; + + public: + template + ArrayScalar(T value, const ArrayExpr &like) + : value_(value), size_(like.size()), shape_(like.shape()) {} + T operator[](size_t) const { return value_; } + size_t size() const { return size_; } + std::array shape() const { return shape_; } +}; + +// Builds the expression for expr op expr, expr op scalar and scalar op expr +template +auto binary_op(const ArrayExpr &lhs, const ArrayExpr &rhs) { + auto a = operand(lhs); + auto b = operand(rhs); + return ArrayBinaryOp(a, b); } -template -auto operator*(const ArrayExpr &arr1, const ArrayExpr &arr2) { - return ArrayMul, ArrayExpr, Ndim>(arr1, arr2); +template >> +auto binary_op(const ArrayExpr &lhs, T rhs) { + return binary_op(lhs, ArrayScalar(rhs, lhs)); } -template -auto operator/(const ArrayExpr &arr1, const ArrayExpr &arr2) { - return ArrayDiv, ArrayExpr, Ndim>(arr1, arr2); +template >> +auto binary_op(T lhs, const ArrayExpr &rhs) { + return binary_op(ArrayScalar(lhs, rhs), rhs); +} + +// The operators only take part in overload resolution if binary_op accepts +// the operands, meaning at least one of them is an ArrayExpr +template +auto operator+(const L &lhs, const R &rhs) + -> decltype(binary_op>(lhs, rhs)) { + return binary_op>(lhs, rhs); +} + +template +auto operator-(const L &lhs, const R &rhs) + -> decltype(binary_op>(lhs, rhs)) { + return binary_op>(lhs, rhs); +} + +template +auto operator*(const L &lhs, const R &rhs) + -> decltype(binary_op>(lhs, rhs)) { + return binary_op>(lhs, rhs); +} + +template +auto operator/(const L &lhs, const R &rhs) + -> decltype(binary_op>(lhs, rhs)) { + return binary_op>(lhs, rhs); } } // namespace aare \ No newline at end of file diff --git a/include/aare/Backoff.hpp b/include/aare/Backoff.hpp new file mode 100644 index 00000000..617c8aae --- /dev/null +++ b/include/aare/Backoff.hpp @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include +#include + +namespace aare { + +/** + * @brief Hint the CPU that this is a spin-wait. Prefer this over yield() when + * the wait is expected to be short. + */ +inline void cpu_relax() noexcept { +#if defined(__x86_64__) || defined(__i386__) + __builtin_ia32_pause(); +#elif defined(__aarch64__) + asm volatile("yield" ::: "memory"); +#else + std::this_thread::yield(); +#endif +} + +/** + * @brief Escalating wait for producer/consumer idle and backpressure loops. + * + * Starts with a pause instruction (sub-microsecond, no syscall), then yield, + * then a short sleep. Call reset() whenever work arrives so a busy pipeline + * never leaves the pause tier. + */ +class Backoff { + int m_count{0}; + + public: + void reset() noexcept { m_count = 0; } + + void pause() noexcept { + if (m_count < 64) { + cpu_relax(); + } else if (m_count < 256) { + std::this_thread::yield(); + } else { + std::this_thread::sleep_for(std::chrono::microseconds(50)); + } + ++m_count; + } +}; + +} // namespace aare diff --git a/include/aare/CircularFifo.hpp b/include/aare/CircularFifo.hpp index e0f4842b..fb275b72 100644 --- a/include/aare/CircularFifo.hpp +++ b/include/aare/CircularFifo.hpp @@ -1,11 +1,13 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once +#include #include #include #include #include #include +#include #include "aare/ProducerConsumerQueue.hpp" @@ -29,6 +31,23 @@ template class CircularFifo { } } + /** + * @brief Construct a fifo and seed the free list using a factory. + * @param size number of items circulating in the fifo + * @param make_item callable invoked as make_item(i) for each slot index i + * + * Use this instead of CircularFifo(size) when the items need to be + * initialized, for example to hold a preallocated buffer or to carry + * their own slot index. + */ + template + CircularFifo(uint32_t size, F make_item) + : fifo_size(size), free_slots(size + 1), filled_slots(size + 1) { + for (size_t i = 0; i < fifo_size; ++i) { + free_slots.write(make_item(i)); + } + } + bool next() { // TODO! avoid default constructing ItemType ItemType it; @@ -47,6 +66,13 @@ template class CircularFifo { auto numFreeSlots() const noexcept { return free_slots.sizeGuess(); } auto isFull() const noexcept { return filled_slots.isFull(); } + /** + * @brief True if there are no filled slots waiting to be consumed. + * @note Prefer this over numFilledSlots() == 0 since sizeGuess() may + * under-report when called from the producing thread. + */ + auto isEmpty() const noexcept { return filled_slots.isEmpty(); } + ItemType pop_free() { ItemType v; while (!free_slots.read(v)) @@ -75,8 +101,22 @@ template class CircularFifo { ItemType *frontPtr() { return filled_slots.frontPtr(); } - // TODO! Add function to move item from filled to free to be used - // with the frontPtr function + /** + * @brief Return the front filled item to the free list. To be used + * together with frontPtr() once the item has been consumed in place. + * @warning The fifo must not be empty when calling this. + * + * The item is written to the free list before it is popped from the + * filled list, so it can never be dropped. The write cannot fail: both + * queues hold size + 1 slots while only size items circulate. + */ + void recycle_front() { + ItemType *it = filled_slots.frontPtr(); + assert(it != nullptr); + [[maybe_unused]] const bool ok = free_slots.write(std::move(*it)); + assert(ok); + filled_slots.popFront(); + } template void push_value(Args &&...recordArgs) { while (!filled_slots.write(std::forward(recordArgs)...)) diff --git a/include/aare/Cluster.hpp b/include/aare/Cluster.hpp index aa56defe..b08a2878 100755 --- a/include/aare/Cluster.hpp +++ b/include/aare/Cluster.hpp @@ -52,9 +52,9 @@ struct Cluster { // TODO: handle 1 dimensional clusters /** - * @brief sum of 2x2 subcluster with highest energy - * @return photon energy of subcluster, 2x2 subcluster index relative to - * cluster center + * @brief Find the highest-sum center-adjacent 2x2 subcluster. + * @return Sum and corner index of the selected 2x2 subcluster relative to + * the cluster center */ Sum_index_pair max_sum_2x2() const { @@ -117,13 +117,10 @@ struct Cluster { }; /** - * @brief Reduce a cluster to a 2x2 cluster by selecting the 2x2 block with the - * highest sum. + * @brief Reduce a cluster to its highest-sum center-adjacent 2x2 block. * @param c Cluster to reduce - * @return reduced cluster + * @return Reduced cluster with the input coordinates * @note The cluster is filled using row major ordering starting at the top-left - * (thus for a max subcluster in the top left cornern the photon hit is at - * the fourth position) */ template @@ -194,17 +191,21 @@ Cluster reduce_to_2x2(const Cluster &c) { } /** - * @brief Reduce a cluster to a 3x3 cluster + * @brief Reduce a cluster to the 3x3 block around its center index. * @param c Cluster to reduce - * @return reduced cluster + * @pre ClusterSizeX and ClusterSizeY must both be at least 3, and at least one + * must be greater than 3. + * @return Reduced cluster with the input coordinates are preserved. */ template Cluster reduce_to_3x3(const Cluster &c) { - static_assert(ClusterSizeX >= 3 && ClusterSizeY >= 3, - "Cluster sizes must be at least 3x3 for reduction to 3x3"); + static_assert(ClusterSizeX >= 3 && ClusterSizeY >= 3 && + (ClusterSizeX > 3 || ClusterSizeY > 3), + "Cluster sizes must both be at least 3, and at least one " + "must be greater than 3 for reduction to 3x3"); Cluster result{}; diff --git a/include/aare/ClusterCollector.hpp b/include/aare/ClusterCollector.hpp index 23666b7d..aa4e334c 100644 --- a/include/aare/ClusterCollector.hpp +++ b/include/aare/ClusterCollector.hpp @@ -2,7 +2,11 @@ #pragma once #include #include +#include +#include +#include +#include "aare/Backoff.hpp" #include "aare/ClusterFinderMT.hpp" #include "aare/ClusterVector.hpp" #include "aare/ProducerConsumerQueue.hpp" @@ -13,48 +17,64 @@ namespace aare { template >> class ClusterCollector { - ProducerConsumerQueue> *m_source; + using SourceQueue = ProducerConsumerQueue>; + + SourceQueue *m_source; std::atomic m_stop_requested{false}; std::atomic m_stopped{true}; - std::chrono::milliseconds m_default_wait{1}; std::thread m_thread; std::vector> m_clusters; void process() { - m_stopped = false; + m_stopped.store(false, std::memory_order_release); fmt::print("ClusterCollector started\n"); - while (!m_stop_requested || !m_source->isEmpty()) { + Backoff backoff; + while (true) { if (ClusterVector *clusters = m_source->frontPtr(); clusters != nullptr) { + backoff.reset(); m_clusters.push_back(std::move(*clusters)); m_source->popFront(); - } else { - std::this_thread::sleep_for(m_default_wait); + continue; } + + if (m_stop_requested.load(std::memory_order_acquire)) + break; + + backoff.pause(); } fmt::print("ClusterCollector stopped\n"); - m_stopped = true; + m_stopped.store(true, std::memory_order_release); } public: - ClusterCollector(ClusterFinderMT *source) { - m_source = source->sink(); + explicit ClusterCollector(SourceQueue *source) : m_source(source) { m_thread = std::thread(&ClusterCollector::process, this); // only one process does that so why isnt it // automatically written to m_cluster in collect // - instead of writing first to m_sink? } + + template ().sink()), + SourceQueue *>, + int> = 0> + explicit ClusterCollector(Finder *source) + : ClusterCollector(source->sink()) {} + void stop() { - m_stop_requested = true; - m_thread.join(); + m_stop_requested.store(true, std::memory_order_release); + if (m_thread.joinable()) + m_thread.join(); } std::vector> steal_clusters() { - if (!m_stopped) { + if (!m_stopped.load(std::memory_order_acquire)) { throw std::runtime_error("ClusterCollector is still running"); } return std::move(m_clusters); } }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/ClusterFile.hpp b/include/aare/ClusterFile.hpp index 4e416c23..507e2c9c 100644 --- a/include/aare/ClusterFile.hpp +++ b/include/aare/ClusterFile.hpp @@ -3,46 +3,39 @@ #include "aare/Cluster.hpp" #include "aare/ClusterVector.hpp" +#include "aare/FilePtr.hpp" #include "aare/GainMap.hpp" #include "aare/NDArray.hpp" -#include "aare/defs.hpp" +#include "aare/ROI.hpp" #include "aare/logger.hpp" #include #include #include +#include +#include namespace aare { -/* -Binary cluster file. Expects data to be laid out as: -int32_t frame_number -uint32_t number_of_clusters -int16_t x, int16_t y, int32_t data[9] x number_of_clusters -int32_t frame_number -uint32_t number_of_clusters -.... -*/ - -// TODO: change to support any type of clusters, e.g. header line with -// clsuter_size_x, cluster_size_y, /** - * @brief Class to read and write cluster files - * Expects data to be laid out as: + * @brief Read and write legacy binary cluster files. * + * Each frame is stored as: * * int32_t frame_number * uint32_t number_of_clusters - * int16_t x, int16_t y, int32_t data[9] * number_of_clusters - * int32_t frame_number - * uint32_t number_of_clusters - * etc. + * ClusterType clusters[number_of_clusters] + * + * The format stores clusters as their native in-memory representation and has + * no metadata describing the cluster dimensions, value type, coordinate type, + * padding, or byte order. Readers must therefore use the same ClusterType and + * a compatible platform ABI as the writer. */ template >> class ClusterFile { - FILE *fp{}; - const std::string m_filename{}; + FilePtr m_fp; + std::string m_filename{}; uint32_t m_num_left{}; /*Number of photons left in frame*/ size_t m_chunk_size{}; /*Number of clusters to read at a time*/ std::string m_mode; /*Mode to open the file in*/ @@ -54,48 +47,150 @@ class ClusterFile { public: /** - * @brief Construct a new Cluster File object - * @param fname path to the file - * @param chunk_size number of clusters to read at a time when iterating - * over the file - * @param mode mode to open the file in. "r" for reading, "w" for writing, - * "a" for appending - * @throws std::runtime_error if the file could not be opened + * @brief Single-pass range over frames or chunks of selected clusters. + * + * The range borrows the file and shares its current position with all + * other reads. Use only one traversal at a time. The file must outlive the + * range and its iterators and must not be moved while they are in use. + * Constructing a range does not read; begin() reads the first result. + * Calling begin() again starts at the file's then-current position. + */ + template class ReadRange { + ClusterFile *m_file; + size_t m_chunk_size; + + friend class ClusterFile; + ReadRange(ClusterFile &file, size_t chunk_size) + : m_file(&file), m_chunk_size(chunk_size) {} + + public: + struct Sentinel {}; + + /** + * @brief Move-only iterator for C++17 range-based loops. + * + * References to the current vector are valid until the iterator is + * advanced or destroyed. Move the vector out to retain its storage. + * Frame iteration reuses storage when the vector is not moved out. + * This is not a copyable STL input iterator. + */ + class Iterator { + ClusterFile *m_file; + size_t m_chunk_size; + ClusterVector m_clusters{0}; + + friend class ReadRange; + Iterator(ClusterFile &file, size_t chunk_size) + : m_file(&file), m_chunk_size(chunk_size) { + ++*this; + } + + public: + using value_type = ClusterVector; + + Iterator(const Iterator &) = delete; + Iterator &operator=(const Iterator &) = delete; + Iterator(Iterator &&) noexcept = default; + Iterator &operator=(Iterator &&) noexcept = default; + ~Iterator() = default; + + value_type &operator*() { return m_clusters; } + value_type *operator->() { return &m_clusters; } + + Iterator &operator++() { + if (m_file) { + if constexpr (ByFrame) { + if (!m_file->read_frame(m_clusters)) { + m_file = nullptr; + } + } else { + m_clusters = m_file->read_clusters(m_chunk_size); + if (m_clusters.empty()) { + m_file = nullptr; + } + } + } + return *this; + } + + void operator++(int) { ++*this; } + + friend bool operator==(const Iterator &it, Sentinel /*end*/) { + return it.m_file == nullptr; + } + friend bool operator!=(const Iterator &it, Sentinel end) { + return !(it == end); + } + friend bool operator==(Sentinel end, const Iterator &it) { + return it == end; + } + friend bool operator!=(Sentinel end, const Iterator &it) { + return !(it == end); + } + }; + + Iterator begin() const { return Iterator(*m_file, m_chunk_size); } + Sentinel end() const { return {}; } + }; + + using FrameRange = ReadRange; + using ChunkRange = ReadRange; + + /** + * @brief Open a cluster file. + * @param fname Path to the file. + * @param chunk_size Maximum number of selected clusters per chunk step. + * @param mode File mode: "r" to read, "w" to truncate and write, or "a" + * to append. + * @throws std::runtime_error If the mode is unsupported or the file cannot + * be opened. */ ClusterFile(const std::filesystem::path &fname, size_t chunk_size = 1000, const std::string &mode = "r") - : m_filename(fname.string()), m_chunk_size(chunk_size), m_mode(mode) { - - if (mode == "r") { - fp = fopen(m_filename.c_str(), "rb"); - if (!fp) { - throw std::runtime_error("Could not open file for reading: " + - m_filename); - } - } else if (mode == "w") { - fp = fopen(m_filename.c_str(), "wb"); - if (!fp) { - throw std::runtime_error("Could not open file for writing: " + - m_filename); - } - } else if (mode == "a") { - fp = fopen(m_filename.c_str(), "ab"); - if (!fp) { - throw std::runtime_error("Could not open file for appending: " + - m_filename); - } - } else { - throw std::runtime_error("Unsupported mode: " + mode); - } + : m_filename(fname.string()), m_chunk_size(chunk_size) { + open(mode); } - ~ClusterFile() { close(); } + /** + * @brief Iterate over complete frames from the current file position. + * + * Empty and fully filtered frames are yielded with their stored frame + * numbers. Reading stops only at a clean end of file. Advancing uses + * read_frame(), including its error for a prior partial-frame read. + */ + FrameRange frames() & { return FrameRange(*this, 0); } + FrameRange frames() && = delete; + + /** @brief Iterate using the chunk size supplied to the constructor. */ + ChunkRange chunks() & { return chunks(m_chunk_size); } + ChunkRange chunks() && = delete; /** - * @brief Read n_clusters clusters from the file discarding - * frame numbers. If EOF is reached the returned vector will - * have less than n_clusters clusters + * @brief Iterate over chunks from the current file position. + * @param chunk_size Maximum number of selected clusters per step. + * @throws std::invalid_argument If chunk_size is zero. + * @note Chunks can span frames; their frame numbers are not per-cluster + * metadata. Advancing uses read_clusters() and propagates its errors. + * Only the final chunk can be short. Empty chunks are not yielded. + */ + ChunkRange chunks(size_t chunk_size) & { + if (chunk_size == 0) { + throw std::invalid_argument("Chunk size must be greater than zero"); + } + return ChunkRange(*this, chunk_size); + } + ChunkRange chunks(size_t) && = delete; + + /** + * @brief Read up to n_clusters without preserving frame boundaries. + * @param n_clusters Maximum number of selected clusters to return. + * @return A cluster vector that may contain fewer clusters at a clean end + * of file. + * @note The returned vector may combine data from several frames, so its + * frame number must not be used as per-cluster metadata. + * @throws std::runtime_error If the file is not open for reading, an I/O + * error occurs, or an incomplete frame header or cluster record is read. */ ClusterVector read_clusters(size_t n_clusters) { if (m_mode != "r") { @@ -109,76 +204,116 @@ class ClusterFile { } /** - * @brief Read a single frame from the file and return the - * clusters. The cluster vector will have the frame number - * set. - * @throws std::runtime_error if the file is not opened for - * reading or the file pointer not at the beginning of a - * frame + * @brief Read the next complete frame. + * @return The selected clusters with the stored frame number set, or + * std::nullopt at a clean end of file before the next frame. + * @throws std::runtime_error If the file is not open for reading, a prior + * partial-frame read left clusters unread, or the frame is incomplete. + * @note A complete frame produces an engaged optional even when it contains + * no clusters or all of its clusters are removed by the configured filters. */ - ClusterVector read_frame() { + std::optional> read_frame() { + ClusterVector clusters(0); + if (!read_frame(clusters)) { + return std::nullopt; + } + return std::optional>{std::move(clusters)}; + } + + /** + * @brief Read the next complete frame into an existing cluster vector. + * @param clusters Destination whose storage is reused when large enough. + * Existing clusters are replaced after a frame header is read and remain + * unchanged at a clean end of file. + * @return true when a complete frame was read, or false at a clean end of + * file before the next frame. + * @throws std::runtime_error If the file is not open for reading, a prior + * partial-frame read left clusters unread, or a frame is incomplete. + * @note A complete frame is a successful read even when it contains no + * clusters or all of its clusters are removed by the configured filters. + */ + bool read_frame(ClusterVector &clusters) { if (m_mode != "r") { throw std::runtime_error(LOCATION + "File not opened for reading"); } - if (m_noise_map || m_roi) { - return read_frame_with_cut(); - } else { - return read_frame_without_cut(); + if (m_num_left) { + throw std::runtime_error( + LOCATION + "There are still clusters left in the last frame"); } + + if (m_noise_map || m_roi) { + return read_frame_with_cut(clusters); + } + return read_frame_without_cut(clusters); } + /** + * @brief Write one frame to the file. + * @param clusters Clusters to write, including their frame number. + * @throws std::runtime_error If the file is not open for writing or any + * part of the frame cannot be written completely. + */ void write_frame(const ClusterVector &clusters) { if (m_mode != "w" && m_mode != "a") { throw std::runtime_error("File not opened for writing"); } int32_t frame_number = clusters.frame_number(); - fwrite(&frame_number, sizeof(frame_number), 1, fp); + if (fwrite(&frame_number, sizeof(frame_number), 1, m_fp.get()) != 1) { + throw std::runtime_error(LOCATION + "Could not write frame number"); + } + uint32_t n_clusters = clusters.size(); - fwrite(&n_clusters, sizeof(n_clusters), 1, fp); - fwrite(clusters.data(), clusters.item_size(), clusters.size(), fp); + if (fwrite(&n_clusters, sizeof(n_clusters), 1, m_fp.get()) != 1) { + throw std::runtime_error(LOCATION + + "Could not write number of clusters"); + } + + if (fwrite(clusters.data(), clusters.item_size(), clusters.size(), + m_fp.get()) != clusters.size()) { + throw std::runtime_error(LOCATION + "Could not write clusters"); + } } /** - * @brief Return the chunk size + * @brief Return the default number of selected clusters per chunk step. */ size_t chunk_size() const { return m_chunk_size; } /** - * @brief Estimate the number of clusters in the file from its size + * @brief Estimate the number of clusters in the file from its size. * - * Frame headers are included in the estimate, so the result may be slightly - * larger than the actual number of clusters. The file position is not - * changed. - * - * @throws std::runtime_error if the file is not opened for reading + * Frame-header bytes are included in the estimate, so it may exceed the + * actual number of clusters. The file position is not changed. */ size_t estimate_n_clusters() const { return std::filesystem::file_size(m_filename) / sizeof(ClusterType); } /** - * @brief Set the region of interest to use when reading - * clusters. If set only clusters within the ROI will be - * read. + * @brief Select clusters by their center coordinate when reading. + * @param roi Half-open region of interest: [xmin, xmax) x [ymin, ymax). */ void set_roi(ROI roi) { m_roi = roi; } /** - * @brief Set the noise map to use when reading clusters. If - * set clusters below the noise level will be discarded. - * Selection criteria one of: Central pixel above noise, - * highest 2x2 sum above 2 * noise, total sum above 3 * - * noise. + * @brief Discard clusters that do not pass the noise thresholds. + * @param noise_map Per-pixel noise indexed as [y, x]. The map is copied. + * A cluster is retained only when its central pixel exceeds the local + * noise, its highest 2x2 sum exceeds twice the noise, and its total sum + * exceeds three times the noise. + * @warning The map must cover every cluster center coordinate in the file. */ void set_noise_map(const NDView noise_map) { m_noise_map = NDArray(noise_map); } /** - * @brief Set the gain map to use when reading clusters. If set the gain map - * will be applied to the clusters that pass ROI and noise_map selection. - * The gain map is expected to be in ADU/energy. + * @brief Apply a gain map to clusters selected while reading. + * @param gain_map Per-pixel gain in ADU/energy, indexed as [y, x]. The map + * is copied and inverted internally. + * @note Clusters whose complete footprint extends beyond the gain map are + * retained with all cluster data values set to zero. */ void set_gain_map(const NDView gain_map) { m_gain_map = InvertedGainMap(gain_map); @@ -193,65 +328,53 @@ class ClusterFile { } /** - * @brief Close the file. If not closed the file will be - * closed in the destructor + * @brief Close the file. + * + * Calling close more than once is safe. The destructor closes an open file + * automatically. */ void close() { - if (fp) { - fclose(fp); - fp = nullptr; - } + m_fp = FilePtr{}; + m_mode = ""; } /** - * @brief Return the current position in the file (bytes) + * @brief Return the current byte position in the file. + * @throws std::runtime_error If the file is closed or its position cannot + * be determined. */ int64_t tell() { - if (!fp) { + if (!m_fp.get()) { throw std::runtime_error(LOCATION + "File not opened"); } - return ftell(fp); + return m_fp.tell(); } + private: /** @brief Open the file in specific mode * */ void open(const std::string &mode) { - if (fp) { - close(); - } + close(); if (mode == "r") { - fp = fopen(m_filename.c_str(), "rb"); - if (!fp) { - throw std::runtime_error("Could not open file for reading: " + - m_filename); - } + m_fp = FilePtr(m_filename, "rb"); m_mode = "r"; } else if (mode == "w") { - fp = fopen(m_filename.c_str(), "wb"); - if (!fp) { - throw std::runtime_error("Could not open file for writing: " + - m_filename); - } + m_fp = FilePtr(m_filename, "wb"); m_mode = "w"; } else if (mode == "a") { - fp = fopen(m_filename.c_str(), "ab"); - if (!fp) { - throw std::runtime_error("Could not open file for appending: " + - m_filename); - } + m_fp = FilePtr(m_filename, "ab"); m_mode = "a"; } else { throw std::runtime_error("Unsupported mode: " + mode); } } - - private: ClusterVector read_clusters_with_cut(size_t n_clusters); ClusterVector read_clusters_without_cut(size_t n_clusters); - ClusterVector read_frame_with_cut(); - ClusterVector read_frame_without_cut(); + bool read_frame_header(int32_t &frame_number, uint32_t &n_clusters); + bool read_frame_with_cut(ClusterVector &clusters); + bool read_frame_without_cut(ClusterVector &clusters); bool is_selected(ClusterType &cl); ClusterType read_one_cluster(); }; @@ -281,25 +404,31 @@ ClusterFile::read_clusters_without_cut(size_t n_clusters) { } else { nn = nph; } - nph_read += fread((buf + nph_read), clusters.item_size(), nn, fp); + const auto count = + fread((buf + nph_read), clusters.item_size(), nn, m_fp.get()); + if (count != nn) { + throw std::runtime_error(LOCATION + "Could not read clusters"); + } + nph_read += count; m_num_left = nph - nn; // write back the number of photons left } if (nph_read < n_clusters) { // keep on reading frames and photons until reaching n_clusters - while (fread(&iframe, sizeof(iframe), 1, fp)) { + while (read_frame_header(iframe, nph)) { clusters.set_frame_number(iframe); - // read number of clusters in frame - if (fread(&nph, sizeof(nph), 1, fp)) { - if (nph > (n_clusters - nph_read)) - nn = n_clusters - nph_read; - else - nn = nph; + if (nph > (n_clusters - nph_read)) + nn = n_clusters - nph_read; + else + nn = nph; - nph_read += - fread((buf + nph_read), clusters.item_size(), nn, fp); - m_num_left = nph - nn; + const auto count = + fread((buf + nph_read), clusters.item_size(), nn, m_fp.get()); + if (count != nn) { + throw std::runtime_error(LOCATION + "Could not read clusters"); } + nph_read += count; + m_num_left = nph - nn; if (nph_read >= n_clusters) break; } @@ -339,16 +468,15 @@ ClusterFile::read_clusters_with_cut(size_t n_clusters) { } int32_t frame_number = 0; // frame number needs to be 4 bytes! - while (fread(&frame_number, sizeof(frame_number), 1, fp)) { - if (fread(&m_num_left, sizeof(m_num_left), 1, fp)) { - clusters.set_frame_number( - frame_number); // cluster vector will hold the last - // frame number - while (m_num_left && clusters.size() < n_clusters) { - ClusterType c = read_one_cluster(); - if (is_selected(c)) { - clusters.push_back(c); - } + uint32_t n_clusters_in_frame = 0; + while (read_frame_header(frame_number, n_clusters_in_frame)) { + m_num_left = n_clusters_in_frame; + clusters.set_frame_number( + frame_number); // cluster vector will hold the last frame number + while (m_num_left && clusters.size() < n_clusters) { + ClusterType c = read_one_cluster(); + if (is_selected(c)) { + clusters.push_back(c); } } @@ -366,7 +494,7 @@ ClusterFile::read_clusters_with_cut(size_t n_clusters) { template ClusterType ClusterFile::read_one_cluster() { ClusterType c; - auto rc = fread(&c, sizeof(c), 1, fp); + auto rc = fread(&c, sizeof(c), 1, m_fp.get()); if (rc != 1) { throw std::runtime_error(LOCATION + "Could not read cluster"); } @@ -375,73 +503,70 @@ ClusterType ClusterFile::read_one_cluster() { } template -ClusterVector -ClusterFile::read_frame_without_cut() { - if (m_mode != "r") { - throw std::runtime_error(LOCATION + "File not opened for reading"); +bool ClusterFile::read_frame_header(int32_t &frame_number, + uint32_t &n_clusters) { + const auto frame_number_bytes = + fread(&frame_number, 1, sizeof(frame_number), m_fp.get()); + if (frame_number_bytes == 0 && feof(m_fp.get()) && !ferror(m_fp.get())) { + return false; } - if (m_num_left) { - throw std::runtime_error( - LOCATION + "There are still photons left in the last frame"); - } - int32_t frame_number; - if (fread(&frame_number, sizeof(frame_number), 1, fp) != 1) { - if (feof(fp)) - throw std::runtime_error(LOCATION + "Unexpected end of file"); - else if (ferror(fp)) + if (frame_number_bytes != sizeof(frame_number)) { + if (ferror(m_fp.get())) { throw std::runtime_error(LOCATION + "Error reading from file"); - - throw std::runtime_error( - LOCATION + - "Unexpected error (not feof or ferror) when reading frame number"); + } + throw std::runtime_error(LOCATION + "Incomplete frame number"); } + const auto cluster_count_bytes = + fread(&n_clusters, 1, sizeof(n_clusters), m_fp.get()); + if (cluster_count_bytes != sizeof(n_clusters)) { + if (ferror(m_fp.get())) { + throw std::runtime_error(LOCATION + "Error reading from file"); + } + throw std::runtime_error(LOCATION + "Incomplete number of clusters"); + } + return true; +} + +template +bool ClusterFile::read_frame_without_cut( + ClusterVector &clusters) { + int32_t frame_number; uint32_t n_clusters; - if (fread(&n_clusters, sizeof(n_clusters), 1, fp) != 1) { - throw std::runtime_error(LOCATION + - "Could not read number of clusters"); + if (!read_frame_header(frame_number, n_clusters)) { + return false; } LOG(logDEBUG1) << "Reading " << n_clusters << " clusters from frame " << frame_number; - ClusterVector clusters(n_clusters); clusters.set_frame_number(frame_number); clusters.resize(n_clusters); LOG(logDEBUG1) << "clusters.item_size(): " << clusters.item_size(); - if (fread(clusters.data(), clusters.item_size(), n_clusters, fp) != + if (fread(clusters.data(), clusters.item_size(), n_clusters, m_fp.get()) != static_cast(n_clusters)) { throw std::runtime_error(LOCATION + "Could not read clusters"); } if (m_gain_map) m_gain_map->apply_gain_map(clusters); - return clusters; + return true; } template -ClusterVector -ClusterFile::read_frame_with_cut() { - if (m_mode != "r") { - throw std::runtime_error("File not opened for reading"); - } - if (m_num_left) { - throw std::runtime_error( - "There are still photons left in the last frame"); - } +bool ClusterFile::read_frame_with_cut( + ClusterVector &clusters) { int32_t frame_number; - if (fread(&frame_number, sizeof(frame_number), 1, fp) != 1) { - throw std::runtime_error("Could not read frame number"); + uint32_t n_clusters; + if (!read_frame_header(frame_number, n_clusters)) { + return false; } - if (fread(&m_num_left, sizeof(m_num_left), 1, fp) != 1) { - throw std::runtime_error("Could not read number of clusters"); - } - - ClusterVector clusters; - clusters.reserve(m_num_left); + m_num_left = n_clusters; + clusters.resize(0); + clusters.reserve(n_clusters); clusters.set_frame_number(frame_number); while (m_num_left) { ClusterType c = read_one_cluster(); @@ -451,7 +576,7 @@ ClusterFile::read_frame_with_cut() { } if (m_gain_map) m_gain_map->apply_gain_map(clusters); - return clusters; + return true; } template diff --git a/include/aare/ClusterFinder.hpp b/include/aare/ClusterFinder.hpp index 83b5c780..ca78ad06 100644 --- a/include/aare/ClusterFinder.hpp +++ b/include/aare/ClusterFinder.hpp @@ -3,6 +3,7 @@ #include "aare/ClusterFile.hpp" #include "aare/ClusterVector.hpp" #include "aare/Dtype.hpp" +#include "aare/FastPedestal.hpp" #include "aare/NDArray.hpp" #include "aare/NDView.hpp" #include "aare/Pedestal.hpp" @@ -18,6 +19,13 @@ struct no_2x2_cluster { ClusterType::cluster_size_x > 2 && ClusterType::cluster_size_y > 2; }; +/** + * @brief Find fixed-size photon clusters using a per-pixel pedestal and noise + * threshold. + * @tparam ClusterType Output cluster type; both dimensions must exceed 2. + * @tparam FRAME_TYPE Input pixel type. + * @tparam PEDESTAL_TYPE Type used for pedestal and threshold calculations. + */ template , typename FRAME_TYPE = uint16_t, typename PEDESTAL_TYPE = double, typename = std::enable_if_t::value>> @@ -26,51 +34,108 @@ class ClusterFinder { PEDESTAL_TYPE m_nSigma; const PEDESTAL_TYPE c2; const PEDESTAL_TYPE c3; - Pedestal m_pedestal; + FastPedestal m_pedestal; ClusterVector m_clusters; static const uint8_t ClusterSizeX = ClusterType::cluster_size_x; static const uint8_t ClusterSizeY = ClusterType::cluster_size_y; using CT = typename ClusterType::value_type; + NDArray m_threshold; + NDArray m_pd_corrected_frame; + public: /** - * @brief Construct a new ClusterFinder object - * @param image_size size of the image - * @param cluster_size size of the cluster (x, y) - * @param nSigma number of sigma above the pedestal to consider a photon - * @param capacity initial capacity of the cluster vector - * + * @brief Construct a cluster finder with an empty pedestal. + * @param image_size Image shape as (rows, columns). + * @param nSigma Per-pixel noise threshold multiplier. + * @param capacity Initial cluster-vector capacity. + * @param min_pedestal_samples Minimum number of pedestal frames to + * accumulate to get reasonable statistics */ ClusterFinder(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, - size_t capacity = 1000000) + size_t capacity = 1000000, size_t min_pedestal_samples = 1000) : m_image_size(image_size), m_nSigma(nSigma), c2(sqrt((ClusterSizeY + 1) / 2 * (ClusterSizeX + 1) / 2)), c3(sqrt(ClusterSizeX * ClusterSizeY)), - m_pedestal(image_size[0], image_size[1]), m_clusters(capacity) { + m_pedestal(image_size[0], image_size[1], min_pedestal_samples), + m_clusters(capacity), m_threshold({image_size[0], image_size[1]}, 0), + m_pd_corrected_frame({image_size[0], image_size[1]}, 0) { LOG(logDEBUG) << "ClusterFinder: " << "image_size: " << image_size[0] << "x" << image_size[1] - << ", nSigma: " << nSigma << ", capacity: " << capacity; + << ", nSigma: " << nSigma << ", capacity: " << capacity + << ", min_pedestal_samples: " << min_pedestal_samples; } - void set_nSigma(PEDESTAL_TYPE nSigma) { m_nSigma = nSigma; } + /** + * @brief Set the noise multiplier used for threshold calculation. + * + * The threshold is recomputed immediately if the pedestal is ready, + * otherwise when the pedestal becomes ready. + * @param nSigma New per-pixel noise multiplier. + */ + void set_nSigma(PEDESTAL_TYPE nSigma) { + m_nSigma = nSigma; + if (m_pedestal.ready()) { + update_threshold(); + } + } + /** @brief Return the current noise multiplier used for threshold + * calculation. */ PEDESTAL_TYPE get_nSigma() const { return m_nSigma; } + /** + * @brief Add a dark frame to the pedestal estimator. + * + * The threshold is initialized automatically when the pedestal first + * becomes ready. Later frames update the ready pedestal. + * @param frame Dark frame matching the configured image shape. + * @throws std::runtime_error if the frame shape does not match. + */ void push_pedestal_frame(NDView frame) { - m_pedestal.push(frame); + if (!m_pedestal.ready()) { + m_pedestal.add_init_frame(frame); + // Initialize the threshold when the pedestal becomes ready. + if (m_pedestal.ready()) { + update_threshold(); + } + } else { + m_pedestal.push_ema(frame); + } } + /** @brief Return a copy of the per-pixel pedestal mean. */ NDArray pedestal() { return m_pedestal.mean(); } + + /** @brief Return the per-pixel pedestal standard deviation (noise). */ NDArray noise() { return m_pedestal.std(); } + + /** @brief Clear the pedestal and mark it as not ready. */ void clear_pedestal() { m_pedestal.clear(); } /** - * @brief Move the clusters from the ClusterVector in the ClusterFinder to a - * new ClusterVector and return it. - * @param realloc_same_capacity if true the new ClusterVector will have the - * same capacity as the old one - * + * @brief Return whether enough pedestal frames have been pushed to find + * clusters. + */ + bool pedestal_ready() const { return m_pedestal.ready(); } + + /** + * @brief Recompute the threshold as noise multiplied by nSigma. + * @throws std::runtime_error if the pedestal is not ready. + */ + void update_threshold() { + if (!m_pedestal.ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot update threshold"); + } + m_threshold = m_pedestal.std() * m_nSigma; + } + + /** + * @brief Move out all accumulated clusters and reset the internal vector. + * @param realloc_same_capacity Preserve the previous capacity when true. + * @return The accumulated clusters and their frame metadata. */ ClusterVector steal_clusters(bool realloc_same_capacity = false) { @@ -81,109 +146,179 @@ class ClusterFinder { m_clusters = ClusterVector{}; return tmp; } + + private: + /** + * @brief Process a single pixel: scan its cluster window, decide whether it + * is a photon or a pedestal value, and store the cluster if needed. + * @tparam CheckBounds Skip out-of-image neighbours when true; assume the + * complete window is in bounds when false. Skipped cluster values remain 0. + */ + template + void process_pixel(const NDView &frame, const int iy, + const int ix) { + + constexpr int dy = ClusterSizeY / 2; + constexpr int dx = ClusterSizeX / 2; + constexpr int has_center_pixel_x = ClusterSizeX % 2; + constexpr int has_center_pixel_y = ClusterSizeY % 2; + + PEDESTAL_TYPE max = std::numeric_limits::lowest(); + PEDESTAL_TYPE total = 0; + + const int cols = static_cast(frame.shape(1)); + const int rows = static_cast(frame.shape(0)); + const auto center = + (static_cast(iy) * static_cast(cols)) + + static_cast(ix); + const auto *corrected = m_pd_corrected_frame.data(); + const PEDESTAL_TYPE threshold = m_threshold.data()[center]; + const PEDESTAL_TYPE value = corrected[center]; + + if (value < -threshold) + return; // NEGATIVE_PEDESTAL, nothing to do for this pixel + // TODO! No pedestal update??? + + for (int ir = -dy; ir < dy + has_center_pixel_y; ir++) { + const int y = iy + ir; + if constexpr (CheckBounds) { + if (y < 0 || y >= rows) + continue; + } + + const auto *row = corrected + static_cast(y) * cols; + for (int ic = -dx; ic < dx + has_center_pixel_x; ic++) { + const int x = ix + ic; + if constexpr (CheckBounds) { + if (x < 0 || x >= cols) + continue; + } + + const PEDESTAL_TYPE val = row[x]; + total += val; + max = std::max(max, val); + } + } + + if ((max > threshold)) { + if (value < max) + return; // Not max go to the next pixel, no pedestal update + } else if (total > c3 * threshold) { + // pass, store the cluster below + } else { + m_pedestal.push_ema_unchecked(center, frame.data()[center]); + return; // It was a pedestal value nothing to store + } + + // Store cluster + if (value == max) { + ClusterType cluster{}; + cluster.x = ix; + cluster.y = iy; + + int i = 0; + for (int ir = -dy; ir < dy + has_center_pixel_y; ir++) { + const int y = iy + ir; + for (int ic = -dx; ic < dx + has_center_pixel_x; ic++, i++) { + const int x = ix + ic; + if constexpr (CheckBounds) { + if (x < 0 || x >= cols || y < 0 || y >= rows) + continue; + } + + const PEDESTAL_TYPE corrected_value = + corrected[(static_cast(y) * cols) + x]; + // If the cluster type is an integral type, and the + // pedestal is a floating point type then we need to + // round the value before storing it + if constexpr (std::is_integral_v && + std::is_floating_point_v) { + cluster.data[i] = + static_cast(std::lround(corrected_value)); + } + // On the other hand if both are floating point or both + // are integral then we can just static cast directly + else { + cluster.data[i] = static_cast(corrected_value); + } + } + } + + // Add the cluster to the output ClusterVector + m_clusters.push_back(cluster); + } + } + + public: + /** + * @brief Find clusters in one frame and update eligible pedestal pixels. + * @param frame Input frame matching the configured image shape. + * @param frame_number Metadata assigned to the accumulated clusters. + * @pre frame has the same shape as image_size passed to the constructor. + * @throws std::runtime_error if the pedestal is not ready. + * @note Clusters accumulate until steal_clusters() is called. + */ void find_clusters(NDView frame, uint64_t frame_number = 0) { + // // TODO! deal with even size clusters // // currently 3,3 -> +/- 1 // // 4,4 -> +/- 2 - int dy = ClusterSizeY / 2; - int dx = ClusterSizeX / 2; - int has_center_pixel_x = - ClusterSizeX % - 2; // for even sized clusters there is no proper cluster center and - // even amount of pixels around the center - int has_center_pixel_y = ClusterSizeY % 2; + if (!m_pedestal.ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot find clusters"); + } + + constexpr int dy = ClusterSizeY / 2; + constexpr int dx = ClusterSizeX / 2; + constexpr int has_center_pixel_x = ClusterSizeX % 2; + constexpr int has_center_pixel_y = ClusterSizeY % 2; + + // Largest neighbour offset below/right of the current pixel. Pixels + // further than this from an edge have their whole window in bounds. + constexpr int down = dy + has_center_pixel_y - 1; + constexpr int right = dx + has_center_pixel_x - 1; m_clusters.set_frame_number(frame_number); - for (int iy = 0; iy < frame.shape(0); iy++) { - for (int ix = 0; ix < frame.shape(1); ix++) { - PEDESTAL_TYPE max = std::numeric_limits::min(); - PEDESTAL_TYPE total = 0; + const int rows = static_cast(frame.shape(0)); + const int cols = static_cast(frame.shape(1)); - // What can we short circuit here? - PEDESTAL_TYPE rms = m_pedestal.std(iy, ix); - PEDESTAL_TYPE value = (frame(iy, ix) - m_pedestal.mean(iy, ix)); + // TODO! See if we can get the same performace using the operator- + // m_pd_corrected_frame = frame - m_pedestal.view(); - if (value < -m_nSigma * rms) - continue; // NEGATIVE_PEDESTAL go to next pixel - // TODO! No pedestal update??? + // here we should be able to safely assume that the frame and corrected + // frame have the same size + auto n_pixels = frame.size(); + auto pd = m_pedestal.view().data(); + auto corrected = m_pd_corrected_frame.data(); + auto frame_data = frame.data(); + for (ssize_t i = 0; i < n_pixels; i++) { + corrected[i] = static_cast(frame_data[i]) - pd[i]; + } - for (int ir = -dy; ir < dy + has_center_pixel_y; ir++) { - for (int ic = -dx; ic < dx + has_center_pixel_x; ic++) { - if (ix + ic >= 0 && ix + ic < frame.shape(1) && - iy + ir >= 0 && iy + ir < frame.shape(0)) { - PEDESTAL_TYPE val = - frame(iy + ir, ix + ic) - - m_pedestal.mean(iy + ir, ix + ic); + // Interior pixels can skip the per-neighbour bounds checks; pixels + // within dx/dy of an edge take the bounds-checked path. Iteration order + // (row-major, increasing ix) is preserved so results are identical. + const int ix_begin = dx; + const int ix_end = cols - right; // exclusive - total += val; - max = std::max(max, val); - } - } - } + for (int iy = 0; iy < rows; iy++) { + const bool interior_row = iy >= dy && iy < rows - down; - if ((max > m_nSigma * rms)) { - if (value < max) - continue; // Not max go to the next pixel - // but also no pedestal update - } else if (total > c3 * m_nSigma * rms) { - // pass - } else { - // m_pedestal.push(iy, ix, frame(iy, ix)); // Safe option - m_pedestal.push_fast( - iy, ix, - frame(iy, - ix)); // Assume we have reached n_samples in the - // pedestal, slight performance improvement - continue; // It was a pedestal value nothing to store - } - - // Store cluster - if (value == max) { - ClusterType cluster{}; - cluster.x = ix; - cluster.y = iy; - - // Fill the cluster data since we have a photon to store - // It's worth redoing the look since most of the time we - // don't have a photon - int i = 0; - for (int ir = -dy; ir < dy + has_center_pixel_y; ir++) { - for (int ic = -dx; ic < dx + has_center_pixel_x; ic++) { - if (ix + ic >= 0 && ix + ic < frame.shape(1) && - iy + ir >= 0 && iy + ir < frame.shape(0)) { - - // If the cluster type is an integral type, and - // the pedestal is a floating point type then we - // need to round the value before storing it - if constexpr (std::is_integral_v && - std::is_floating_point_v< - PEDESTAL_TYPE>) { - auto tmp = std::lround( - frame(iy + ir, ix + ic) - - m_pedestal.mean(iy + ir, ix + ic)); - cluster.data[i] = static_cast(tmp); - } - // On the other hand if both are floating point - // or both are integral then we can just static - // cast directly - else { - auto tmp = - frame(iy + ir, ix + ic) - - m_pedestal.mean(iy + ir, ix + ic); - cluster.data[i] = static_cast(tmp); - } - } - i++; - } - } - - // Add the cluster to the output ClusterVector - m_clusters.push_back(cluster); - } + if (!interior_row || ix_begin >= ix_end) { + for (int ix = 0; ix < cols; ix++) + process_pixel(frame, iy, ix); + continue; } + + for (int ix = 0; ix < ix_begin; ix++) + process_pixel(frame, iy, ix); + for (int ix = ix_begin; ix < ix_end; ix++) + process_pixel(frame, iy, ix); + for (int ix = ix_end; ix < cols; ix++) + process_pixel(frame, iy, ix); } } }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/ClusterFinderMT.hpp b/include/aare/ClusterFinderMT.hpp index d4ff0018..f57fec88 100644 --- a/include/aare/ClusterFinderMT.hpp +++ b/include/aare/ClusterFinderMT.hpp @@ -1,16 +1,19 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once +#include #include #include #include #include #include +#include "aare/Backoff.hpp" +#include "aare/CircularFifo.hpp" #include "aare/ClusterFinder.hpp" #include "aare/NDArray.hpp" #include "aare/ProducerConsumerQueue.hpp" #include "aare/logger.hpp" - +#include namespace aare { enum class FrameType { @@ -18,10 +21,40 @@ enum class FrameType { PEDESTAL, }; -struct FrameWrapper { - FrameType type; - uint64_t frame_number; - NDArray data; +/** + * @brief Ticket identifying a frame buffer in a FramePool. Trivially + * copyable, the buffer itself never travels through the queues. + */ +struct FrameRef { + FrameType type{}; + uint32_t slot{}; + uint64_t frame_number{}; +}; + +/** + * @brief Fixed set of frame buffers allocated once at construction. + * + * Buffers are addressed by slot index and are never moved or reallocated, so + * the queues only need to pass a FrameRef around. This keeps the per frame + * allocation out of the hot path entirely. + */ +class FramePool { + std::vector> m_buffers; + + public: + FramePool(size_t depth, Shape<2> shape) { + m_buffers.reserve(depth); // no reallocation, slots stay stable + for (size_t i = 0; i < depth; ++i) { + m_buffers.emplace_back(shape); + } + } + + NDArray &operator[](uint32_t slot) { return m_buffers[slot]; } + const NDArray &operator[](uint32_t slot) const { + return m_buffers[slot]; + } + + size_t size() const { return m_buffers.size(); } }; /** @@ -42,17 +75,18 @@ class ClusterFinderMT { size_t m_current_thread{0}; size_t m_n_threads{0}; using Finder = ClusterFinder; - using InputQueue = ProducerConsumerQueue; + using InputQueue = CircularFifo; using OutputQueue = ProducerConsumerQueue>; std::vector> m_input_queues; std::vector> m_output_queues; + std::vector> m_frame_pools; OutputQueue m_sink{1000}; // All clusters go into this queue std::vector> m_cluster_finders; std::vector m_threads; std::thread m_collect_thread; - std::chrono::milliseconds m_default_wait{1}; + std::chrono::microseconds m_default_wait{50}; private: std::atomic m_stop_requested{false}; @@ -65,53 +99,79 @@ class ClusterFinderMT { void process(int thread_id) { auto cf = m_cluster_finders[thread_id].get(); auto q = m_input_queues[thread_id].get(); - bool realloc_same_capacity = true; + auto *pool = m_frame_pools[thread_id].get(); + Backoff backoff; while (!m_stop_requested || !q->isEmpty()) { - if (FrameWrapper *frame = q->frontPtr(); frame != nullptr) { + if (FrameRef *ref = q->frontPtr(); ref != nullptr) { + backoff.reset(); + auto view = (*pool)[ref->slot].view(); - switch (frame->type) { - case FrameType::DATA: - cf->find_clusters(frame->data.view(), frame->frame_number); + switch (ref->type) { + case FrameType::DATA: { + cf->find_clusters(view, ref->frame_number); + // Steal before the write so a failed write cannot drop the + // clusters by re-stealing an empty vector on retry. + auto clusters = cf->steal_clusters(true); while (!m_output_queues[thread_id]->write( - cf->steal_clusters(realloc_same_capacity))) { - std::this_thread::sleep_for(m_default_wait); + std::move(clusters))) { + backoff.pause(); } - break; - + } case FrameType::PEDESTAL: - m_cluster_finders[thread_id]->push_pedestal_frame( - frame->data.view()); + m_cluster_finders[thread_id]->push_pedestal_frame(view); break; } - // frame is processed now discard it - m_input_queues[thread_id]->popFront(); + // frame is processed, hand the buffer back to the free list + q->recycle_front(); } else { + backoff.pause(); + } + } + } + + void wait_for_input_queues() const { + for (auto &q : m_input_queues) { + while (!q->isEmpty()) { std::this_thread::sleep_for(m_default_wait); } } } + bool output_queues_are_empty() const { + for (auto &q : m_output_queues) { + if (!q->isEmpty()) { + return false; + } + } + return true; + } + /** * @brief Collect all the clusters from the output queues and write them to * the sink */ void collect() { - bool empty = true; - while (!m_stop_requested || !empty || !m_processing_threads_stopped) { - empty = true; + Backoff backoff; + while (!m_stop_requested || !output_queues_are_empty() || + !m_processing_threads_stopped) { + bool moved_any = false; for (auto &queue : m_output_queues) { - if (!queue->isEmpty()) { - - while (!m_sink.write(std::move(*queue->frontPtr()))) { - std::this_thread::sleep_for(m_default_wait); + if (auto *front = queue->frontPtr(); front != nullptr) { + while (!m_sink.write(std::move(*front))) { + backoff.pause(); } queue->popFront(); - empty = false; + moved_any = true; } } + if (moved_any) { + backoff.reset(); + } else { + backoff.pause(); + } } } @@ -124,25 +184,40 @@ class ClusterFinderMT { * @param capacity initial capacity of the cluster vector. Should match * expected number of clusters in a frame per frame. * @param n_threads number of threads to use + * @param queue_depth number of frame buffers per thread. These are + * allocated once and recycled, so the total resident frame memory is + * n_threads * queue_depth * frame size. Keeping the in flight data below + * the L3 size keeps the per frame copy cheap. + * @param min_pedestal_samples minimum number of pedestal samples to + * accumulate before using the pedestal */ ClusterFinderMT(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, - size_t capacity = 2000, size_t n_threads = 3) + size_t capacity = 2000, size_t n_threads = 3, + size_t queue_depth = 16, size_t min_pedestal_samples = 1000) : m_n_threads(n_threads) { LOG(logDEBUG1) << "ClusterFinderMT: " << "image_size: " << image_size[0] << "x" << image_size[1] << ", nSigma: " << nSigma << ", capacity: " << capacity - << ", n_threads: " << n_threads; + << ", min_pedestal_samples: " << min_pedestal_samples + << ", n_threads: " << n_threads + << ", queue_depth: " << queue_depth; for (size_t i = 0; i < n_threads; i++) { m_cluster_finders.push_back( std::make_unique< ClusterFinder>( - image_size, nSigma, capacity)); + image_size, nSigma, capacity, min_pedestal_samples)); } for (size_t i = 0; i < n_threads; i++) { - m_input_queues.emplace_back(std::make_unique(200)); + m_frame_pools.emplace_back( + std::make_unique(queue_depth, image_size)); + m_input_queues.emplace_back(std::make_unique( + static_cast(queue_depth), [](size_t slot) { + return FrameRef{FrameType::DATA, + static_cast(slot), 0}; + })); m_output_queues.emplace_back(std::make_unique(200)); } // TODO! Should we start automatically? @@ -212,13 +287,22 @@ class ClusterFinderMT { * expected to be dark. No photon finding is done. Just pedestal update. */ void push_pedestal_frame(NDView frame) { - FrameWrapper fw{FrameType::PEDESTAL, 0, - NDArray(frame)}; // TODO! copies the data! + for (size_t i = 0; i < m_n_threads; ++i) { + auto *q = m_input_queues[i].get(); + Backoff backoff; - for (auto &queue : m_input_queues) { - while (!queue->write(fw)) { - std::this_thread::sleep_for(m_default_wait); + FrameRef ref; + while (!q->try_pop_free(ref)) { + backoff.pause(); } + + ref.type = FrameType::PEDESTAL; + ref.frame_number = 0; + (*m_frame_pools[i])[ref.slot].copy_from(frame); + + // Cannot fail, the free list is what limits how many frames are + // in flight so there is always room in the filled list. + q->try_push_value(ref); } } @@ -228,11 +312,29 @@ class ClusterFinderMT { * @note Spin locks with a default wait if the queue is full. */ void find_clusters(NDView frame, uint64_t frame_number = 0) { - FrameWrapper fw{FrameType::DATA, frame_number, - NDArray(frame)}; // TODO! copies the data! - while (!m_input_queues[m_current_thread % m_n_threads]->write(fw)) { - std::this_thread::sleep_for(m_default_wait); + const size_t tid = m_current_thread % m_n_threads; + auto *q = m_input_queues[tid].get(); + Backoff backoff; + + FrameRef ref; + while (!q->try_pop_free(ref)) { + backoff.pause(); } + + ref.type = FrameType::DATA; + ref.frame_number = frame_number; + + // DualTimer dt; + (*m_frame_pools[tid])[ref.slot].copy_from(frame); + // auto [wall_ns, cpu_ns] = dt.elapsed(); + // std::cerr << "ClusterFinderMT: find_clusters: copied frame " + // << frame_number << " took " << wall_ns/1000.0 << " wall_us + // and " + // << cpu_ns/1000.0 << " cpu_us" << std::endl; + + // Cannot fail, the free list is what limits how many frames are in + // flight so there is always room in the filled list. + q->try_push_value(ref); m_current_thread++; } @@ -245,6 +347,19 @@ class ClusterFinderMT { } } + /** + * @brief Recompute the threshold (nSigma * pedestal std) on all cluster + * finders. Requires the processing threads to be stopped. + */ + void update_threshold() { + if (!m_processing_threads_stopped) { + throw std::runtime_error("ClusterFinderMT is still running"); + } + for (auto &cf : m_cluster_finders) { + cf->update_threshold(); + } + } + /** * @brief Return the pedestal currently used by the cluster finder * @param thread_index index of the thread @@ -286,21 +401,28 @@ class ClusterFinderMT { // } /** - * @brief Set the nSigma value for all the cluster finders. + * @brief Set the nSigma value for all the cluster finders. Waits for the + * queued frames to be processed first. * @param nSigma number of sigma above the pedestal to consider a photon * during cluster finding. */ void set_nSigma(const PEDESTAL_TYPE nSigma) { - // Wait for all queues to be empty before changing the sigma - for (auto &q : m_input_queues) { - while (!q->isEmpty()) { - std::this_thread::sleep_for(m_default_wait); - } - } + wait_for_input_queues(); for (auto &cf : m_cluster_finders) { cf->set_nSigma(nSigma); } } + + /** + * @brief Return whether all cluster finders have received enough pedestal + * frames to find clusters. Waits for the queued frames to be processed + * first. + */ + bool pedestal_ready() { + wait_for_input_queues(); + return std::all_of(m_cluster_finders.begin(), m_cluster_finders.end(), + [](const auto &cf) { return cf->pedestal_ready(); }); + } }; } // namespace aare \ No newline at end of file diff --git a/include/aare/ClusterVector.hpp b/include/aare/ClusterVector.hpp index 0adcb5e3..37956933 100644 --- a/include/aare/ClusterVector.hpp +++ b/include/aare/ClusterVector.hpp @@ -20,14 +20,19 @@ template > { /** * @brief Construct a new ClusterVector object - * @param capacity initial capacity of the buffer in number of clusters + * @param capacity minimum initial capacity in number of clusters * @param frame_number frame number of the clusters. Default is 0, which is * also used to indicate that the clusters come from many frames */ - ClusterVector(size_t capacity = 1024, uint64_t frame_number = 0) + ClusterVector(size_t capacity = 1024, int32_t frame_number = 0) : m_frame_number(frame_number) { m_data.reserve(capacity); } - // Move constructor - ClusterVector(ClusterVector &&other) noexcept - : m_data(other.m_data), m_frame_number(other.m_frame_number) { - other.m_data.clear(); - } + ClusterVector(ClusterVector &&other) noexcept = default; + ClusterVector &operator=(ClusterVector &&other) noexcept = default; /** - * @brief Create a copy of the clustervector by filtering clusters in the - * ClusterVector using a boolean mask. - * @param mask boolean 1d mask - * @return ClusterVector containing only the clusters where the mask is - * true + * @brief Return a filtered copy selected by a one-dimensional Boolean mask. + * @param mask boolean 1d mask true if element in ClusterVector will be + * included + * @return ClusterVector containing the selected clusters in their original + * order and with the original frame number + * @throws std::runtime_error if the mask length differs from size() */ ClusterVector operator()(NDView mask) { if (static_cast(mask.size()) != m_data.size()) { throw std::runtime_error( LOCATION + "Mask size does not match number of clusters"); } - ClusterVector result(capacity(), frame_number()); + if (m_data.empty()) { + return ClusterVector(0, frame_number()); + } + const auto selected = + static_cast(std::count(mask.begin(), mask.end(), true)); + ClusterVector result(selected, frame_number()); for (size_t i = 0; i < m_data.size(); ++i) { if (mask(i)) { result.push_back(m_data[i]); @@ -80,20 +88,9 @@ class ClusterVector> { return result; } - // Move assignment operator - ClusterVector &operator=(ClusterVector &&other) noexcept { - if (this != &other) { - m_data = other.m_data; - m_frame_number = other.m_frame_number; - other.m_data.clear(); - other.m_frame_number = 0; - } - return *this; - } - /** - * @brief Sum the pixels in each cluster - * @return std::vector vector of sums for each cluster + * @brief Sum the pixels in each cluster. + * @return One sum for every cluster, in container order */ std::vector sum() { std::vector sums(m_data.size()); @@ -106,9 +103,10 @@ class ClusterVector> { } /** - * @brief Sum the pixels in the 2x2 subcluster with the biggest pixel sum in - * each cluster - * @return vector of sums index pairs for each cluster + * @brief Find the highest-sum center-adjacent 2x2 subcluster in each + * cluster. + * @return One sum and corner-index pair for every cluster, in container + * order */ std::vector> sum_2x2() { std::vector> sums_2x2(m_data.size()); @@ -128,10 +126,29 @@ class ClusterVector> { */ void reserve(size_t capacity) { m_data.reserve(capacity); } + /** + * @brief Change the number of stored clusters. + * @param size new number of clusters + * @note Growing the vector value-initializes new clusters and can + * invalidate pointers, references, and iterators. + */ void resize(size_t size) { m_data.resize(size); } + /** + * @brief Append a cluster to the vector. + * @param cluster cluster to append + * @note Reallocation invalidates pointers, references, iterators, and + * zero-copy NumPy views of the storage. + */ void push_back(const ClusterType &cluster) { m_data.push_back(cluster); } + /** + * @brief Append all clusters from another vector. + * @param other vector whose clusters are appended + * @return Reference to this vector + * @note The frame number of this vector is unchanged. + * @warning other must not refer to this vector. + */ ClusterVector &operator+=(const ClusterVector &other) { m_data.insert(m_data.end(), other.begin(), other.end()); @@ -148,8 +165,10 @@ class ClusterVector> { */ bool empty() const { return m_data.empty(); } + /** @brief Return the cluster size in the x dimension. */ uint8_t cluster_size_x() const { return ClusterSizeX; } + /** @brief Return the cluster size in the y dimension. */ uint8_t cluster_size_y() const { return ClusterSizeY; } /** @@ -164,7 +183,7 @@ class ClusterVector> { auto end() const { return m_data.end(); } /** - * @brief Return the size in bytes of a single cluster + * @brief Return the size in bytes of one stored cluster, including padding. */ size_t item_size() const { return sizeof(ClusterType); // 2 * sizeof(CoordType) + ClusterSizeX * @@ -175,8 +194,8 @@ class ClusterVector> { ClusterType const *data() const { return m_data.data(); } /** - * @brief Return a reference to the i-th cluster casted to type V - * @tparam V type of the cluster + * @brief Return a reference to the i-th cluster without bounds checking. + * @param i zero-based cluster index */ ClusterType &operator[](size_t i) { return m_data[i]; } @@ -188,16 +207,20 @@ class ClusterVector> { */ int32_t frame_number() const { return m_frame_number; } + /** + * @brief Set the signed 32-bit frame number associated with the clusters. + * @param frame_number frame number, or 0 for clusters from multiple frames + */ void set_frame_number(int32_t frame_number) { m_frame_number = frame_number; } }; /** - * @brief Reduce a cluster to a 2x2 cluster by selecting the 2x2 block with the - * highest sum. - * @param cv Clustervector containing clusters to reduce - * @return Clustervector with reduced clusters + * @brief Reduce every cluster to its highest-sum center-adjacent 2x2 block. + * @param cv ClusterVector containing clusters to reduce + * @return ClusterVector of 2x2 clusters in the original order and with the + * original frame number * @note The cluster is filled using row major ordering starting at the top-left * (thus for a max subcluster in the top left cornern the photon hit is at * the fourth position) @@ -207,7 +230,8 @@ template > reduce_to_2x2( const ClusterVector> &cv) { - ClusterVector> result; + ClusterVector> result(cv.size(), + cv.frame_number()); for (const auto &c : cv) { result.push_back(reduce_to_2x2(c)); } @@ -215,20 +239,25 @@ ClusterVector> reduce_to_2x2( } /** - * @brief Reduce a cluster to a 3x3 cluster - * @param cv Clustervector containing clusters to reduce - * @return Clustervector with reduced clusters + * @brief Reduce every cluster to the 3x3 block around its center index. + * @param cv ClusterVector containing clusters to reduce + * @pre ClusterSizeX and ClusterSizeY must both be at least 3, and at least one + * must be greater than 3. + * @return ClusterVector of 3x3 clusters in the original order and with the + * original frame number + * @note Coordinates are preserved. */ template ClusterVector> reduce_to_3x3( const ClusterVector> &cv) { - ClusterVector> result; + ClusterVector> result(cv.size(), + cv.frame_number()); for (const auto &c : cv) { result.push_back(reduce_to_3x3(c)); } return result; } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/DetectorGeometry.hpp b/include/aare/DetectorGeometry.hpp index 8aa60783..e6718ffd 100644 --- a/include/aare/DetectorGeometry.hpp +++ b/include/aare/DetectorGeometry.hpp @@ -1,8 +1,7 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once +#include "aare/ROI.hpp" #include "aare/ROIGeometry.hpp" -#include "aare/RawMasterFile.hpp" //ROI refactor away -#include "aare/defs.hpp" #include namespace aare { @@ -97,6 +96,8 @@ class DetectorGeometry { const xy udp_interfaces_per_module = xy{1, 1}, const bool quad = false); + DetectorGeometry() = default; + ~DetectorGeometry() = default; /** @@ -114,6 +115,8 @@ class DetectorGeometry { size_t modules_x() const; size_t modules_y() const; + xy udp_interfaces_per_module() const; + const std::vector &get_module_geometries() const; const ModuleGeometry &get_module_geometries(const size_t index) const; @@ -125,6 +128,7 @@ class DetectorGeometry { size_t m_modules_y{}; size_t m_pixels_x{}; size_t m_pixels_y{}; + xy m_udp_interfaces_per_module{}; static constexpr ModuleConfig cfg{0, 0}; // TODO: maybe remove - should be a member in ROIGeometry - in particular diff --git a/include/aare/FastPedestal.hpp b/include/aare/FastPedestal.hpp new file mode 100644 index 00000000..1a3579e8 --- /dev/null +++ b/include/aare/FastPedestal.hpp @@ -0,0 +1,386 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include "aare/File.hpp" +#include "aare/Frame.hpp" +#include "aare/NDArray.hpp" +#include "aare/NDView.hpp" +#include +#include +#include + +namespace aare { + +/** + * @brief Maintain per-pixel mean and population standard deviation. + * + * Initialization accumulates exactly n_samples frames. Subsequent push_ema() + * update the exponential moving average initialized with the mean using + * a smoothing factor of (1/ n_samples). Internal moments are stored + * in double precision. Variance is a private double-precision intermediate. + * @tparam PEDESTAL_TYPE Type returned for the mean and standard deviation. + */ +template class FastPedestal { + + // Did we accumulate enough samples and updated the mean? + bool m_ready = false; + + uint32_t m_rows; + uint32_t m_cols; + + uint32_t m_samples; + double m_inv_samples; // precompute 1/m_samples for faster division + uint32_t m_cur_samples = 0; // number of samples accumulated so far + + // For cache locality we want to keep sum and sum2 close. Should + // improve performance for random access. + struct Entry { + double sum; + double sum2; + }; + NDArray m_sum; + + // Cache mean since it is used over and over in the ClusterFinder + // This optimization is related to the access pattern of the ClusterFinder + // Relies on having more reads than pushes to the pedestal + // But also makes sense when subtracting the pedestal from the frame + NDArray m_mean; + + // Helper function to convert row and column indices to a flat index + // used to provide both row column and flat index access to the pedestal + size_t rc_to_index(uint32_t row, uint32_t col) const { + return (static_cast(row) * m_cols) + col; + } + + public: + /** + * @brief Construct an empty pedestal that becomes ready after n_samples + * initialization frames. + * @param rows Number of image rows. + * @param cols Number of image columns. + * @param n_samples Number of initialization frames and reciprocal of the + * weight assigned to each subsequent value. + * @throws std::runtime_error if rows, cols, or n_samples is zero. + */ + FastPedestal(uint32_t rows, uint32_t cols, uint32_t n_samples = 1000) + : m_rows(rows), m_cols(cols), m_samples(n_samples), + m_inv_samples(1.0 / n_samples), m_sum({rows, cols}, Entry{0, 0}), + m_mean({rows, cols}, PEDESTAL_TYPE(0)) { + if (!(rows > 0 && cols > 0 && n_samples > 0)) { + throw std::runtime_error( + fmt::format("Invalid parameters for FastPedestal: rows={}, " + "cols={}, n_samples={} need to be positive", + rows, cols, n_samples)); + } + } + + ~FastPedestal() = default; + + /** + * @brief Return a non-owning view of the cached mean. + * @throws std::runtime_error if ready() is false. + * @note The caller must treat the data as read-only and must not retain the + * view after this object is destroyed, moved, or assigned. + */ + NDView view() const { + if (!ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot return view"); + } + return m_mean.view(); + } + + /** + * @brief Return a copy of the cached mean. + * @throws std::runtime_error if ready() is false. + */ + NDArray mean() const { + if (!ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot return mean"); + } + return m_mean; + } + + /** + * @brief Return the cached mean at (row, col). + * @throws std::runtime_error if ready() is false or either index is out of + * range. + */ + PEDESTAL_TYPE mean(uint32_t row, uint32_t col) const { + if (!ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot return mean"); + } + if (row >= m_rows || col >= m_cols) { + throw std::runtime_error( + fmt::format("Invalid indices for FastPedestal mean: row={}, " + "col={} must be in [0, {}), [0, {})", + row, col, m_rows, m_cols)); + } + return m_mean(row, col); + } + + /** + * @brief Return the cached mean at a flat row-major index. + * @pre ready() is true, the cache is current, and index is valid; the index + * is not checked. + */ + PEDESTAL_TYPE mean_unchecked(ssize_t index) const { return m_mean[index]; } + + /** + * @brief Calculate and return the population standard deviation of every + * pixel. + * @throws std::runtime_error if ready() is false. + */ + NDArray std() const { + if (!ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot return std"); + } + NDArray res({m_rows, m_cols}); + for (ssize_t i = 0; i < m_sum.size(); ++i) { + res[i] = std_unchecked(i); + } + return res; + } + + /** + * @brief Calculate the population standard deviation at (row, col). + * @throws std::runtime_error if ready() is false or either index is out of + * range. + */ + PEDESTAL_TYPE std(const uint32_t row, const uint32_t col) const { + if (!ready()) { + throw std::runtime_error( + "Pedestal is not ready, cannot return std"); + } + if (row >= m_rows || col >= m_cols) { + throw std::runtime_error( + fmt::format("Invalid indices for FastPedestal std: row={}, " + "col={} must be in [0, {}), [0, {})", + row, col, m_rows, m_cols)); + } + return std_unchecked(rc_to_index(row, col)); + } + + /** + * @brief Calculate the population standard deviation at a flat row-major + * index. + * @pre ready() is true and index is valid; the index is not checked. + */ + PEDESTAL_TYPE std_unchecked(ssize_t index) const { + return std::sqrt(variance_unchecked(index)); + } + + /** + * @brief Return whether initialization is complete (cur_samples() equals + * n_samples()). + */ + bool ready() const { return m_ready; } + + /** + * @brief Return the stored number of accumulated initialization frames. + * @note The value is in [0, n_samples] and does not change during + * steady-state pushes. + */ + uint32_t cur_samples() const { return m_cur_samples; } + + /** + * @brief Zero the moments and cached mean, and mark the pedestal not ready. + */ + void clear() { + m_sum = Entry{0., 0.}; + m_mean = PEDESTAL_TYPE(0.); + m_cur_samples = 0; + m_ready = false; + } + + /** + * @brief Update every pixel using the steady-state exponential estimator, + * giving the new value weight 1 / n_samples. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs or ready() is false. + */ + template void push_ema(NDView frame) { + if (frame.shape() != std::array{m_rows, m_cols}) { + throw std::runtime_error( + "Frame shape does not match pedestal shape"); + } + + if (!ready()) { + throw std::runtime_error("Pedestal is not ready, cannot push"); + } + + const auto size = static_cast(m_rows) * m_cols; + const auto *data = frame.data(); + for (std::size_t index = 0; index < size; ++index) { + push_ema_unchecked(index, data[index]); + } + } + + /** + * @brief Update every pixel from a Frame using the steady-state estimator. + * @tparam T Actual pixel type stored in frame; this is not runtime-checked. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs or ready() is false. + */ + template void push_ema(Frame &frame) { + push_ema(frame.view()); + } + + /** + * @brief Update the exponential moving average with smoothing factor + * 1/n_samples + * @param row Pixel row. + * @param col Pixel column. + * @param val New pixel value. + * @pre row and col are valid; indices are not checked. + * @throws std::runtime_error if ready() is false. + */ + template + void push_ema(const uint32_t row, const uint32_t col, const T val) { + if (!ready()) { + throw std::runtime_error("Pedestal is not ready, cannot push"); + } + + push_ema_unchecked(rc_to_index(row, col), val); + } + + /** + * @brief Update one pixel and its cached mean without runtime checks in + * release builds. + * @param index Flat row-major pixel index. + * @param value New pixel value, with weight 1 / n_samples. + * @pre ready() is true and index is a valid flat row-major index. These + * preconditions are asserted only in debug builds. + */ + template + void push_ema_unchecked(const std::size_t index, const T value) noexcept { + assert(m_ready); + assert(index < static_cast(m_sum.size())); + + const auto val = static_cast(value); + auto &entry = m_sum[index]; + entry.sum += val - entry.sum * m_inv_samples; + entry.sum2 += val * val - entry.sum2 * m_inv_samples; + m_mean[index] = static_cast(entry.sum * m_inv_samples); + } + + /** + * @brief Accumulate one initialization frame. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs or n_samples frames have + * already been accumulated. + * @note The statistics can be accessed and ready() becomes true only after + * the n_samples frames have been added. + */ + template void add_init_frame(NDView frame) { + if (frame.shape() != std::array{m_rows, m_cols}) { + throw std::runtime_error( + "Frame shape does not match pedestal shape"); + } + + // if the pedestal is already initialized we cannot add more frames + if (ready()) { + throw std::runtime_error("Pedestal initialization is already done"); + } + + for (ssize_t i = 0; i < m_sum.size(); ++i) { + const auto val = static_cast(frame[i]); + auto &entry = m_sum[i]; + entry.sum += val; + entry.sum2 += val * val; + } + m_cur_samples += 1; + + if (m_cur_samples == m_samples) { + update_mean(); + m_ready = true; + } + } + + /** + * @brief Initialize from n_samples frames after skip_first, then apply all + * remaining file frames as steady-state updates. + * @tparam T Pixel representation stored in the file; this is not checked. + * @param filename Input image file. Its dimensions define the pedestal + * shape. + * @param n_samples Number of initialization frames. + * @param skip_first Number of leading frames to ignore. + * @throws std::runtime_error if fewer than n_samples frames remain after + * skip_first, or if any constructor argument is invalid. + */ + template + static FastPedestal from_file(const std::filesystem::path &filename, + uint32_t n_samples = 1000, + uint32_t skip_first = 0) { + File f(filename); + + const auto total_frames = f.total_frames(); + const auto first_frame = static_cast(skip_first); + const auto initialization_frames = static_cast(n_samples); + if (first_frame > total_frames || + initialization_frames > total_frames - first_frame) { + throw std::runtime_error( + "File has less frames than the number of samples needed to " + "initialize the pedestal"); + } + + if (skip_first > 0) { + f.seek(static_cast(skip_first)); + } + const auto rows = static_cast(f.rows()); + const auto cols = static_cast(f.cols()); + FastPedestal pedestal(rows, cols, n_samples); + NDArray frame({rows, cols}); + + auto frame_index = first_frame; + const auto initialization_end = first_frame + initialization_frames; + while (frame_index < initialization_end) { + f.read_into(frame.buffer()); + pedestal.template add_init_frame(frame.view()); + frame_index++; + } + + // read the rest of the file + while (frame_index < total_frames) { + f.read_into(frame.buffer()); + pedestal.template push_ema(frame.view()); + frame_index++; + } + return pedestal; + } + + /** @brief Return the number of image rows. */ + uint32_t rows() const { return m_rows; } + + /** @brief Return the number of image columns. */ + uint32_t cols() const { return m_cols; } + + /** + * @brief Return the initialization frame count and steady-state + * update-weight denominator. + */ + uint32_t n_samples() const { return m_samples; } + + private: + double variance_unchecked(ssize_t index) const { + const auto &entry = m_sum[index]; + const auto mean = entry.sum * m_inv_samples; + const auto variance = std::fma(-mean, mean, entry.sum2 * m_inv_samples); + // Roundoff in the moments can make a near-zero variance negative. + return std::max(variance, 0.0); + } + + /** + * @brief Write the cached mean after the final add_init_frame. All other + * (non initialization) pushes update the cached mean immediately. + */ + void update_mean() { + for (ssize_t i = 0; i < m_sum.size(); i++) { + auto &entry = m_sum[i]; + m_mean[i] = static_cast(entry.sum * m_inv_samples); + } + } +}; +} // namespace aare diff --git a/include/aare/FilePtr.hpp b/include/aare/FilePtr.hpp index 2cd08490..2e84b87c 100644 --- a/include/aare/FilePtr.hpp +++ b/include/aare/FilePtr.hpp @@ -18,6 +18,7 @@ class FilePtr { FilePtr &operator=(const FilePtr &) = delete; // since we handle a resource FilePtr(FilePtr &&other); FilePtr &operator=(FilePtr &&other); + explicit operator bool() const noexcept; FILE *get(); ssize_t tell(); void seek(ssize_t offset, int whence = SEEK_SET) { @@ -28,4 +29,4 @@ class FilePtr { ~FilePtr(); }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/Fit.hpp b/include/aare/Fit.hpp index 1a2fa55f..cc7edce1 100644 --- a/include/aare/Fit.hpp +++ b/include/aare/Fit.hpp @@ -7,17 +7,24 @@ namespace aare { // --------------------------------------------------------------------------- -// Minuit2-based pixel fitting. +// Pixel fitting with the minimizer selected by the FitModel. // Template bodies and explicit instantiations live in src/Fit.cpp. // --------------------------------------------------------------------------- /** - * @brief Fit a single pixel's data using Minuit2. + * @brief Fit a single pixel's data. + * + * The minimizer (Migrad, Fumili or LevenbergMarquardt) is selected by the + * model, see aare::Minimizer. Parameter errors, when requested, come from + * MnHesse with Migrad and from the linearised covariance otherwise; fixed + * parameters and parameters ending on a limit report 0. A failed fit + * returns zeros. * * User-precedence rules: - * - Fixed parameters: untouched (value and fixed flag preserved from model). - * - User-set start: value preserved, step size auto-filled. - * - Neither: both value and step size auto-filled from data. + * - Fixed parameters: kept at their value, excluded from the fit. + * - User-set start: value used as the starting point. + * - Neither: starting value estimated from the data and clamped + * into the parameter limits. * * @tparam Model Model struct (Gaussian, RisingScurve, …). * diff --git a/include/aare/FitModel.hpp b/include/aare/FitModel.hpp index a52ea9fa..f485849e 100644 --- a/include/aare/FitModel.hpp +++ b/include/aare/FitModel.hpp @@ -3,59 +3,102 @@ #include "aare/Models.hpp" #include -#include #include #include #include namespace aare { +/** + * @brief Minimizer used by FitModel. + * + * - Migrad: Minuit2's variable-metric minimizer, driven by the analytic + * gradient of the chi-squared. Parameter errors come from MnHesse. + * - Fumili: Minuit2's Gauss-Newton minimizer with a trust region. It takes + * the gradient and a linearised Hessian from the model's analytic + * derivatives and typically needs far fewer function evaluations than + * Migrad. Parameter errors come from the covariance of that linearised + * Hessian, without a Hesse step. Minuit2's Fumili cannot converge once a + * two-sided limit becomes active; a pixel without a valid Fumili minimum + * is refitted with Migrad. + * - LevenbergMarquardt: the built-in damped Gauss-Newton solver + * (src/LevenbergMarquardt.hpp). It uses the analytic Jacobian, reflects + * steps at parameter limits and reuses its buffers between pixels, so a + * data cube is fitted without per-pixel allocations. Parameter errors are + * sqrt(diag((J^T J)^-1)) over the free parameters. + * - VarPro: the built-in variable projection solver + * (src/VariableProjection.hpp) for models that declare their linear + * parameters (model::is_separable). Every trial point solves the linear + * parameters exactly and the Levenberg-Marquardt iteration runs over the + * nonlinear ones only, which needs fewer evaluations and avoids poor + * start values of the linear parameters. Start values of free linear + * parameters are ignored. A limit on a linear parameter cannot be + * enforced by the linear solve: a pixel whose solution violates one, a + * pixel that does not converge, and every pixel of a model without the + * separable structure are fitted with LevenbergMarquardt instead. + * Parameter errors are the same Gauss-Newton estimates as with + * LevenbergMarquardt. + * + * With every minimizer, fixed parameters and parameters that end on a limit + * report an error of 0 and a failed fit returns zeros. `tolerance` is the + * EDM tolerance in Minuit's convention (Migrad, LevenbergMarquardt and + * VarPro stop below 0.002 * tolerance, Fumili below + * 1e-4 * tolerance). The EDM is in units of chi2, so an unweighted fit of + * small values meets it far from the minimum; pass errors or a lower + * tolerance for such data. `max_calls` bounds the work per pixel in the + * minimizer's own units (Minuit2 function calls, or model evaluations for + * LevenbergMarquardt and VarPro, which spend one evaluation per + * iteration and two when a step crosses a limit). `strategy` only affects + * the Minuit2 minimizers. + */ +enum class Minimizer { Migrad, Fumili, LevenbergMarquardt, VarPro }; + +/** + * @brief Fit configuration for one model: parameter limits, fixed + * parameters, user start values and minimizer settings. + * + * The object holds plain data and can be shared read-only between threads. + * Method bodies live in src/Fit.cpp, which explicitly instantiates the class + * for every supported model. + */ template class FitModel { - - // Forward declaration only — full definition lives in src/FitModelImpl.hpp - // and is never installed. Downstream code cannot dereference this pointer. - struct FitModelImpl; // stores Minuit2 user-set parameters - - std::unique_ptr impl_; - unsigned int max_calls_; - double tolerance_; - bool compute_errors_; - - std::array user_fixed_{}; - std::array user_start_{}; - - /** @brief Safely resolve a parameter name to its index. */ - unsigned int checked_index(const std::string &name) const; - public: static constexpr std::size_t npar = Model::npar; /** * @brief Construct a fit model with sensible defaults. * - * @param strategy Minuit2 strategy level (0 = fast/gradient, 1 = - * default). - * @param max_calls Maximum FCN calls per pixel minimisation. - * @param tolerance Minuit2 EDM tolerance. - * @param compute_errors If true, run MnHesse after minimisation. + * @param strategy Minuit2 strategy level (0 = fast, 1 = Minuit2's + * default). Ignored by LevenbergMarquardt. + * @param max_calls Work budget per pixel: Minuit2 function calls, + * or model evaluations for LevenbergMarquardt. + * @param tolerance EDM tolerance in Minuit's convention, see + * aare::Minimizer. + * @param compute_errors If true, also report parameter errors: from + * MnHesse with Migrad, from the linearised + * covariance with Fumili and LevenbergMarquardt. + * Fixed parameters and parameters ending on a limit + * report 0. + * @param minimizer Minimizer, see aare::Minimizer. */ explicit FitModel(unsigned int strategy = 0, unsigned int max_calls = 100, - double tolerance = 0.5, bool compute_errors = false); + double tolerance = 0.5, bool compute_errors = false, + Minimizer minimizer = Minimizer::Migrad); - // Destructor must be defined in Fit.cpp where FitModelImpl is complete. - ~FitModel(); - FitModel(const FitModel &); - FitModel &operator=(const FitModel &); - FitModel(FitModel &&) noexcept = default; - FitModel &operator=(FitModel &&) noexcept = default; - - /** @brief Set lower and upper bounds for parameter idx.*/ + /** + * @brief Set lower and upper limits for parameter idx. + * + * Infinite values leave that side open (the Minuit2 minimizers turn an + * open side into a wide finite one, at least 1e6 away from the other). + * Throws std::runtime_error when lo >= hi and std::out_of_range for a + * bad index. + */ void SetParLimits(unsigned int idx, double lo, double hi); /** * @brief Fix parameter idx at value val. * - * Excluded from minimisation. Automatic estimates will not touch it. + * Excluded from minimisation. Automatic estimates will not touch it. */ void FixParameter(unsigned int idx, double val); @@ -76,16 +119,36 @@ template class FitModel { void SetMaxCalls(unsigned int n) { max_calls_ = n; } void SetTolerance(double t) { tolerance_ = t; } void SetComputeErrors(bool b) { compute_errors_ = b; } + void SetMinimizer(Minimizer m) { minimizer_ = m; } + unsigned int strategy() const { return strategy_; } unsigned int max_calls() const { return max_calls_; } double tolerance() const { return tolerance_; } bool compute_errors() const { return compute_errors_; } + Minimizer minimizer() const { return minimizer_; } bool is_user_fixed(unsigned int idx) const { return user_fixed_[idx]; } bool is_user_start(unsigned int idx) const { return user_start_[idx]; } + /** @brief Lower limit of parameter idx, -infinity when open. */ + double lower_limit(unsigned int idx) const { return lower_[idx]; } + /** @brief Upper limit of parameter idx, +infinity when open. */ + double upper_limit(unsigned int idx) const { return upper_[idx]; } + /** @brief User start or fixed value of parameter idx, 0 when unset. */ + double value(unsigned int idx) const { return value_[idx]; } - // Returns the internal Minuit2 state. FitModelImpl is an incomplete type - // here; only callers that include src/FitModelImpl.hpp can dereference it. - FitModelImpl *impl() const { return impl_.get(); } + private: + void check_index(unsigned int idx) const; + unsigned int checked_index(const std::string &name) const; + + unsigned int strategy_; + unsigned int max_calls_; + double tolerance_; + bool compute_errors_; + Minimizer minimizer_; + std::array lower_{}; + std::array upper_{}; + std::array value_{}; + std::array user_fixed_{}; + std::array user_start_{}; }; // Suppress implicit instantiation for all supported model types. diff --git a/include/aare/GainMap.hpp b/include/aare/GainMap.hpp index ff888915..3cc63077 100644 --- a/include/aare/GainMap.hpp +++ b/include/aare/GainMap.hpp @@ -34,29 +34,40 @@ class InvertedGainMap { template >> void apply_gain_map(ClusterVector &clustervec) { - // in principle we need to know the size of the image for this lookup - size_t ClusterSizeX = clustervec.cluster_size_x(); - size_t ClusterSizeY = clustervec.cluster_size_y(); - using T = typename ClusterVector::value_type; - int64_t index_cluster_center_x = ClusterSizeX / 2; - int64_t index_cluster_center_y = ClusterSizeY / 2; + constexpr size_t cluster_size_x = ClusterType::cluster_size_x; + constexpr size_t cluster_size_y = ClusterType::cluster_size_y; + constexpr ssize_t pixel_left_from_center = cluster_size_x / 2; + constexpr ssize_t pixels_right_from_center = + cluster_size_x - pixel_left_from_center - 1; + constexpr ssize_t pixels_top_from_center = cluster_size_y / 2; + constexpr ssize_t pixels_bottom_from_center = + cluster_size_y - pixels_top_from_center - 1; + for (size_t i = 0; i < clustervec.size(); i++) { auto &cl = clustervec[i]; + const auto center_x = static_cast(cl.x); + const auto center_y = static_cast(cl.y); - if (cl.x > 0 && cl.y > 0 && cl.x < m_gain_map.shape(1) - 1 && - cl.y < m_gain_map.shape(0) - 1) { - for (size_t j = 0; j < ClusterSizeX * ClusterSizeY; j++) { - size_t x = cl.x + j % ClusterSizeX - index_cluster_center_x; - size_t y = cl.y + j / ClusterSizeX - index_cluster_center_y; + if (center_x >= pixel_left_from_center && + center_y >= pixels_top_from_center && + center_x < m_gain_map.shape(1) - pixels_right_from_center && + center_y < m_gain_map.shape(0) - pixels_bottom_from_center) { + for (size_t j = 0; j < cluster_size_x * cluster_size_y; j++) { + const auto x = center_x + + static_cast(j % cluster_size_x) - + pixel_left_from_center; + const auto y = center_y + + static_cast(j / cluster_size_x) - + pixels_top_from_center; cl.data[j] = static_cast( static_cast(cl.data[j]) * m_gain_map( y, x)); // cast after conversion to keep precision } } else { - // clear edge clusters + // Clear clusters whose footprint extends beyond the gain map. cl.data.fill(0); } } @@ -66,4 +77,4 @@ class InvertedGainMap { NDArray m_gain_map{}; }; -} // end of namespace aare \ No newline at end of file +} // end of namespace aare diff --git a/include/aare/Models.hpp b/include/aare/Models.hpp index 04df428b..34b3d8d2 100644 --- a/include/aare/Models.hpp +++ b/include/aare/Models.hpp @@ -2,10 +2,15 @@ #pragma once #include "aare/NDView.hpp" +#include "aare/defs.hpp" #include #include #include +#include +#include +#include #include +#include #include namespace aare::model { @@ -13,10 +18,14 @@ namespace aare::model { inline constexpr double inv_sqrt2 = 0.70710678118654752440; inline constexpr double inv_sqrt_2pi = 0.39894228040143267794; -inline double fast_erf(double x) { - // Abramowitz–Stegun Handbook of Mathematical Functions - // erf approximation with max error ~1.5e-7, faster than std::erf. - +/** + * @brief erf(z) given exp(-z^2), from the Abramowitz-Stegun approximation + * 7.1.26 (Handbook of Mathematical Functions, max error about 1.5e-7). + * + * Models whose derivatives need exp(-z^2) anyway pass it in, so that a + * single exponential serves both the value and the gradient. + */ +ALWAYS_INLINE double fast_erf_from_exp(double z, double exp_minus_z2) { const double a1 = 0.254829592; const double a2 = -0.284496736; const double a3 = 1.421413741; @@ -24,22 +33,69 @@ inline double fast_erf(double x) { const double a5 = 1.061405429; const double p = 0.3275911; - const int sign = x < 0 ? -1 : 1; - x = std::abs(x); - - // 7.1.26 - const double t = 1.0 / (1.0 + p * x); + const double t = 1.0 / (1.0 + p * std::abs(z)); const double y = 1.0 - - (((((a5 * t + a4) * t + a3) * t + a2) * t + a1) * t) * std::exp(-x * x); + (((((a5 * t + a4) * t + a3) * t + a2) * t + a1) * t) * exp_minus_z2; - return sign * y; + return z < 0 ? -y : y; +} + +/** @brief erf approximation, faster than std::erf; see fast_erf_from_exp. */ +ALWAYS_INLINE double fast_erf(double x) { + return fast_erf_from_exp(x, std::exp(-x * x)); +} + +/** + * @brief exp(x) without a call into libm, for loops that should vectorise. + * + * Range reduction x = k ln2 + r with the integer k taken from the mantissa + * of x * log2(e) + 1.5 * 2^52, a degree-13 Taylor polynomial for exp(r) on + * |r| <= ln2 / 2, and 2^k assembled in the exponent bits, so that a loop + * calling it compiles to vector instructions on SSE2, AVX and NEON. The + * relative error is below 4e-16 for finite x; |x| is clamped to 700. The + * bulk basis functions of the models (basis_columns) use it; eval and + * eval_and_grad keep std::exp. It is ALWAYS_INLINE because gcc otherwise + * leaves it out of line in the large fitting translation unit, and a call + * per point defeats the vectorisation of the caller. + */ +ALWAYS_INLINE double fast_exp(double x) { + constexpr double log2e = 1.4426950408889634074; + constexpr double magic = 6755399441055744.0; // 1.5 * 2^52 + constexpr double ln2_hi = 0.693145751953125; + constexpr double ln2_lo = 1.42860682030941723212e-6; + x = std::clamp(x, -700.0, 700.0); + const double t = x * log2e + magic; + const double k = t - magic; + const double r = (x - k * ln2_hi) - k * ln2_lo; + double p = 1.0 / 6227020800.0; + p = p * r + 1.0 / 479001600.0; + p = p * r + 1.0 / 39916800.0; + p = p * r + 1.0 / 3628800.0; + p = p * r + 1.0 / 362880.0; + p = p * r + 1.0 / 40320.0; + p = p * r + 1.0 / 5040.0; + p = p * r + 1.0 / 720.0; + p = p * r + 1.0 / 120.0; + p = p * r + 1.0 / 24.0; + p = p * r + 1.0 / 6.0; + p = p * r + 0.5; + p = p * r + 1.0; + p = p * r + 1.0; + std::int64_t ti = 0; + std::int64_t mi = 0; + std::memcpy(&ti, &t, sizeof(ti)); + std::memcpy(&mi, &magic, sizeof(mi)); + const std::int64_t bits = ((ti - mi) + 1023) << 52; + double scale = 0.0; + std::memcpy(&scale, &bits, sizeof(scale)); + return p * scale; } /** * @brief Per-parameter metadata: name and optional default bounds. * - * Used by FitModel to generically initialise MnUserParameters. + * Used by FitModel for parameter names and default limits. * Unbounded directions use ±no_bound as sentinels. */ struct ParamInfo { @@ -51,22 +107,89 @@ struct ParamInfo { inline constexpr double no_bound = std::numeric_limits::infinity(); /** - * @brief Compute data-range statistics used by step-size estimators. + * @brief Detects a separable model, one that declares its linear parameters + * for Minimizer::VarPro. * - * Model-independent, called once per pixel. + * A model whose function is linear in some of its parameters lists them in + * ascending order in `linear_par` and provides + * + * static void basis_and_grad(double x, const std::vector &par, + * std::array &phi, + * std::array, nnl> &dphi) + * + * with nlin = linear_par.size() and nnl = npar - nlin, such that + * + * f(x; par) = sum_j par[linear_par[j]] * phi[j] + * + * and dphi[k][j] = d phi[j] / d par[nonlinear[k]], where nonlinear lists the + * remaining parameter indices in ascending order (see separable_traits). + * basis_and_grad ignores the linear entries of par. The [fit] tests check + * phi and dphi against eval and eval_and_grad for every model. + * + * A model may additionally provide the same quantities for all scan points + * at once, see has_basis_columns: + * + * static void basis_columns(const double *__restrict x, ssize_t n, + * const std::vector &par, + * double *__restrict phi, + * double *__restrict dphi) + * + * writes phi[j * n + i] = phi_j(x_i) and dphi[(k * nlin + j) * n + i] = + * d phi_j / d par[nonlinear[k]] (x_i), one contiguous column per function. + * Written as a loop free of reductions with the parameters hoisted and + * fast_exp instead of std::exp, it compiles to vector instructions, which + * is where VarPro spends most of its time. The __restrict qualifiers are + * needed: without them gcc has to prove that the output columns do not + * overlap, exceeds its budget of run-time alias checks once a loop writes + * more than three columns, and quietly emits scalar code. The tests check + * it against basis_and_grad. */ -inline void compute_ranges(NDView x, NDView y, - double &x_range, double &y_range, - double &slope_scale) { - const auto [x_min, x_max] = std::minmax_element(x.begin(), x.end()); - const auto [y_min, y_max] = std::minmax_element(y.begin(), y.end()); +template +struct is_separable : std::false_type {}; - x_range = std::max(*x_max - *x_min, 1e-9); - y_range = std::max(*y_max - *y_min, 1e-9); +template +struct is_separable> + : std::true_type {}; - slope_scale = std::max(y_range / x_range, 1e-9); +/** @brief Detects a separable model that provides basis_columns. */ +template +struct has_basis_columns : std::false_type {}; + +template +struct has_basis_columns> + : std::true_type {}; + +/** @brief The parameter indices not listed in linear, in ascending order. */ +template +constexpr std::array +nonlinear_indices(const std::array &linear) { + std::array out{}; + std::size_t m = 0; + for (std::size_t k = 0; k < Npar; ++k) { + bool is_linear = false; + for (std::size_t j = 0; j < Nlin; ++j) + is_linear = is_linear || linear[j] == k; + if (!is_linear) + out[m++] = k; + } + return out; } +/** @brief Index bookkeeping of a separable model, see is_separable. */ +template struct separable_traits { + static constexpr std::size_t npar = Model::npar; + static constexpr std::size_t nlin = Model::linear_par.size(); + static constexpr std::size_t nnl = npar - nlin; + static constexpr std::array nonlinear_par = + nonlinear_indices(Model::linear_par); + static constexpr std::size_t linear_index(int j) { + return Model::linear_par[static_cast(j)]; + } + using Basis = std::array; + using BasisGrad = std::array, nnl>; +}; + // _____________________________________________________________________ // // Pol1 @@ -105,6 +228,17 @@ struct Pol1 { g[1] = x; // df/dp1 } + /** @brief Both parameters are linear: f = p0 * 1 + p1 * x. */ + static constexpr std::array linear_par = {0, 1}; + + static void basis_and_grad( + double x, [[maybe_unused]] const std::vector &par, + std::array &phi, + [[maybe_unused]] std::array, 0> &dphi) { + phi[0] = 1.0; + phi[1] = x; + } + static bool is_valid([[maybe_unused]] const std::vector &par) { return true; // always valid } @@ -120,14 +254,6 @@ struct Pol1 { return {intercept, slope}; } - - static void compute_steps(const std::array &start, - [[maybe_unused]] double x_range, double y_range, - double slope_scale, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.1 * slope_scale; - } }; // _____________________________________________________________________ @@ -172,6 +298,18 @@ struct Pol2 { g[2] = x * x; // df/dp2 } + /** @brief All parameters are linear: f = p0 * 1 + p1 * x + p2 * x^2. */ + static constexpr std::array linear_par = {0, 1, 2}; + + static void basis_and_grad( + double x, [[maybe_unused]] const std::vector &par, + std::array &phi, + [[maybe_unused]] std::array, 0> &dphi) { + phi[0] = 1.0; + phi[1] = x; + phi[2] = x * x; + } + static bool is_valid([[maybe_unused]] const std::vector &par) { return true; // always valid } @@ -220,15 +358,6 @@ struct Pol2 { return {p0, p1, p2}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double slope_scale, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.1 * slope_scale; - steps[2] = 0.1 * slope_scale / std::max(x_range, 1e-12); - } }; // _____________________________________________________________________ @@ -288,6 +417,44 @@ struct Gaussian { sig; // df/dsigma = A*e*(x-mu)^2/sig^3 } + /** @brief A is linear: f = A * exp(-(x - mu)^2 / (2 sigma^2)). */ + static constexpr std::array linear_par = {0}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 2> &dphi) { + const double mu = par[1]; + const double sig = par[2]; + + const double dx = x - mu; + const double inv_sig = 1.0 / sig; + const double inv_sig2 = inv_sig * inv_sig; + const double e = std::exp(-0.5 * dx * dx * inv_sig2); + + phi[0] = e; + dphi[0][0] = e * dx * inv_sig2; // d/dmu + dphi[1][0] = e * dx * dx * inv_sig2 * inv_sig; // d/dsigma + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double mu = par[1]; + const double inv_sig = 1.0 / par[2]; + const double inv_sig2 = inv_sig * inv_sig; + double *e_col = phi; + double *dmu = dphi; + double *dsig = dphi + n; + for (ssize_t i = 0; i < n; ++i) { + const double dx = x[i] - mu; + const double e = fast_exp(-0.5 * dx * dx * inv_sig2); + e_col[i] = e; + dmu[i] = e * dx * inv_sig2; + dsig[i] = e * dx * dx * inv_sig2 * inv_sig; + } + } + /** @brief Reject degenerate sigma (zero width). */ static bool is_valid(const std::vector &par) { return par[2] != 0.0; @@ -326,18 +493,6 @@ struct Gaussian { return {A, mu, sig}; } - - /** - * @brief Data-driven Minuit step sizes. - */ - static void compute_steps(const std::array &start, - double x_range, double y_range, - double /*slope_scale*/, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.05 * x_range; - steps[2] = 0.05 * x_range; - } }; // _____________________________________________________________________ @@ -393,7 +548,7 @@ struct GaussianErfcPlateau { const double z = dx * inv_sqrt2 / sig; const double e = std::exp(-z * z); - const double step = 0.5 * (1.0 - fast_erf(z)); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); return A * e + S * step; } @@ -409,7 +564,7 @@ struct GaussianErfcPlateau { const double z = dx * inv_sqrt2 / sig; const double e = std::exp(-z * z); - const double step = 0.5 * (1.0 - fast_erf(z)); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); f = A * e + S * step; @@ -426,6 +581,63 @@ struct GaussianErfcPlateau { S * inv_sqrt_2pi * e * dx * inv_sig2; // df/dsigma } + /** @brief A and S are linear: f = A * e + S * step. */ + static constexpr std::array linear_par = {0, 1}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 2> &dphi) { + const double mu = par[2]; + const double sig = par[3]; + + const double dx = x - mu; + const double inv_sig = 1.0 / sig; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + const double z = dx * inv_sqrt2 * inv_sig; + + const double e = std::exp(-z * z); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); + + phi[0] = e; + phi[1] = step; + dphi[0][0] = e * dx * inv_sig2; // de/dmu + dphi[0][1] = inv_sqrt_2pi * e * inv_sig; // dstep/dmu + dphi[1][0] = e * dx * dx * inv_sig3; // de/dsigma + dphi[1][1] = inv_sqrt_2pi * e * dx * inv_sig2; // dstep/dsigma + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double mu = par[2]; + const double inv_sig = 1.0 / par[3]; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + const double cz = inv_sqrt2 * inv_sig; + const double ce = inv_sqrt_2pi * inv_sig; + const double ce2 = inv_sqrt_2pi * inv_sig2; + double *e_col = phi; + double *step_col = phi + n; + double *de_dmu = dphi; + double *dstep_dmu = dphi + n; + double *de_dsig = dphi + 2 * n; + double *dstep_dsig = dphi + 3 * n; + for (ssize_t i = 0; i < n; ++i) { + const double dx = x[i] - mu; + const double z = dx * cz; + const double e = fast_exp(-z * z); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); + e_col[i] = e; + step_col[i] = step; + de_dmu[i] = e * dx * inv_sig2; + dstep_dmu[i] = ce * e; + de_dsig[i] = e * dx * dx * inv_sig3; + dstep_dsig[i] = ce2 * e * dx; + } + } + /** @brief Reject degenerate or negative width. */ static bool is_valid(const std::vector &par) { return par[3] > 0.0; @@ -490,16 +702,6 @@ struct GaussianErfcPlateau { return {A, S, mu, sig}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double /*slope_scale*/, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = std::max(0.1 * std::abs(start[1]), 0.1 * y_range); - steps[2] = 0.05 * x_range; - steps[3] = 0.05 * x_range; - } }; // _____________________________________________________________________ @@ -556,7 +758,7 @@ struct GaussianChargeSharing { const double u = dx / sig; const double G = std::exp(-0.5 * u * u); - const double H = 0.5 * (1.0 - fast_erf(u * inv_sqrt2)); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); return p0 - p1 * x + N * (G + C * H); } @@ -574,7 +776,7 @@ struct GaussianChargeSharing { const double u = dx / sig; const double G = std::exp(-0.5 * u * u); - const double H = 0.5 * (1.0 - fast_erf(u * inv_sqrt2)); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); f = p0 - p1 * x + N * (G + C * H); @@ -596,6 +798,84 @@ struct GaussianChargeSharing { g[5] = N * H; } + /** + * @brief p0, p1 and N are linear: f = p0 * 1 + p1 * (-x) + N * (G + C H). + * C multiplies N, so it stays a nonlinear parameter. + */ + static constexpr std::array linear_par = {0, 1, 4}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 3> &dphi) { + const double mu = par[2]; + const double sig = par[3]; + const double C = par[5]; + + const double inv_sig = 1.0 / sig; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + + const double dx = x - mu; + const double u = dx * inv_sig; + + const double G = std::exp(-0.5 * u * u); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + + const double dG_dmu = G * dx * inv_sig2; + const double dG_dsig = G * dx * dx * inv_sig3; + const double dH_dmu = inv_sqrt_2pi * G * inv_sig; + const double dH_dsig = inv_sqrt_2pi * G * dx * inv_sig2; + + phi[0] = 1.0; + phi[1] = -x; + phi[2] = G + C * H; + for (int k = 0; k < 3; ++k) { + dphi[k][0] = 0.0; + dphi[k][1] = 0.0; + } + dphi[0][2] = dG_dmu + C * dH_dmu; // d/dmu + dphi[1][2] = dG_dsig + C * dH_dsig; // d/dsigma + dphi[2][2] = H; // d/dC + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double mu = par[2]; + const double inv_sig = 1.0 / par[3]; + const double C = par[5]; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + double *one = phi; + double *minus_x = phi + n; + double *shape = phi + 2 * n; + // d phi_0 and d phi_1 vanish for every nonlinear parameter. + std::fill(dphi, dphi + 2 * n, 0.0); + std::fill(dphi + 3 * n, dphi + 5 * n, 0.0); + std::fill(dphi + 6 * n, dphi + 8 * n, 0.0); + double *d_mu = dphi + 2 * n; + double *d_sig = dphi + 5 * n; + double *d_C = dphi + 8 * n; + for (ssize_t i = 0; i < n; ++i) { + const double xi = x[i]; + const double dx = xi - mu; + const double u = dx * inv_sig; + const double G = fast_exp(-0.5 * u * u); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + const double dG_dmu = G * dx * inv_sig2; + const double dG_dsig = G * dx * dx * inv_sig3; + const double dH_dmu = inv_sqrt_2pi * G * inv_sig; + const double dH_dsig = inv_sqrt_2pi * G * dx * inv_sig2; + one[i] = 1.0; + minus_x[i] = -xi; + shape[i] = G + C * H; + d_mu[i] = dG_dmu + C * dH_dmu; + d_sig[i] = dG_dsig + C * dH_dsig; + d_C[i] = H; + } + } + static bool is_valid(const std::vector &par) { return par[3] > 0.0; } @@ -662,18 +942,6 @@ struct GaussianChargeSharing { return {p0, p1, mu, sigma, N, C}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double slope_scale, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.1 * slope_scale; - steps[2] = 0.05 * x_range; - steps[3] = 0.05 * x_range; - steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); - steps[5] = std::max(0.1 * std::abs(start[5]), 0.01); - } }; // _____________________________________________________________________ @@ -745,8 +1013,8 @@ struct GaussianChargeSharingKb { const double G = std::exp(-0.5 * u * u); const double Gb = std::exp(-0.5 * ub * ub); - const double H = 0.5 * (1.0 - fast_erf(u * inv_sqrt2)); - const double Hb = 0.5 * (1.0 - fast_erf(ub * inv_sqrt2)); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + const double Hb = 0.5 * (1.0 - fast_erf_from_exp(ub * inv_sqrt2, Gb)); const double ka = G + C * H; const double kb = Gb + C * Hb; @@ -776,8 +1044,8 @@ struct GaussianChargeSharingKb { const double G = std::exp(-0.5 * u * u); const double Gb = std::exp(-0.5 * ub * ub); - const double H = 0.5 * (1.0 - fast_erf(u * inv_sqrt2)); - const double Hb = 0.5 * (1.0 - fast_erf(ub * inv_sqrt2)); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + const double Hb = 0.5 * (1.0 - fast_erf_from_exp(ub * inv_sqrt2, Gb)); const double ka = G + C * H; const double kb = Gb + C * Hb; @@ -822,6 +1090,133 @@ struct GaussianChargeSharingKb { g[7] = N * kb; } + /** + * @brief p0, p1 and N are linear: f = p0 * 1 + p1 * (-x) + N * (ka + q kb). + * C, r and q multiply N or enter the peak shapes, so they stay nonlinear. + */ + static constexpr std::array linear_par = {0, 1, 4}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 5> &dphi) { + const double mu = par[2]; + const double sig = par[3]; + const double C = par[5]; + const double r = par[6]; + const double q = par[7]; + + const double inv_sig = 1.0 / sig; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + + const double mu_b = r * mu; + + const double dx = x - mu; + const double dxb = x - mu_b; + + const double u = dx * inv_sig; + const double ub = dxb * inv_sig; + + const double G = std::exp(-0.5 * u * u); + const double Gb = std::exp(-0.5 * ub * ub); + + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + const double Hb = 0.5 * (1.0 - fast_erf_from_exp(ub * inv_sqrt2, Gb)); + + const double ka = G + C * H; + const double kb = Gb + C * Hb; + + const double dG_dmu = G * dx * inv_sig2; + const double dG_dsig = G * dx * dx * inv_sig3; + const double dH_dmu = inv_sqrt_2pi * G * inv_sig; + const double dH_dsig = inv_sqrt_2pi * G * dx * inv_sig2; + + const double dGb_dmu_b = Gb * dxb * inv_sig2; + const double dGb_dsig = Gb * dxb * dxb * inv_sig3; + const double dHb_dmu_b = inv_sqrt_2pi * Gb * inv_sig; + const double dHb_dsig = inv_sqrt_2pi * Gb * dxb * inv_sig2; + + const double dka_dmu = dG_dmu + C * dH_dmu; + const double dka_dsig = dG_dsig + C * dH_dsig; + + const double dkb_dmu_b = dGb_dmu_b + C * dHb_dmu_b; + const double dkb_dsig = dGb_dsig + C * dHb_dsig; + + phi[0] = 1.0; + phi[1] = -x; + phi[2] = ka + q * kb; + for (int k = 0; k < 5; ++k) { + dphi[k][0] = 0.0; + dphi[k][1] = 0.0; + } + dphi[0][2] = dka_dmu + q * r * dkb_dmu_b; // d/dmu, mu_b = r * mu + dphi[1][2] = dka_dsig + q * dkb_dsig; // d/dsigma + dphi[2][2] = H + q * Hb; // d/dC + dphi[3][2] = q * mu * dkb_dmu_b; // d/dr + dphi[4][2] = kb; // d/dq + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double mu = par[2]; + const double inv_sig = 1.0 / par[3]; + const double C = par[5]; + const double r = par[6]; + const double q = par[7]; + const double inv_sig2 = inv_sig * inv_sig; + const double inv_sig3 = inv_sig2 * inv_sig; + const double mu_b = r * mu; + double *one = phi; + double *minus_x = phi + n; + double *shape = phi + 2 * n; + // d phi_0 and d phi_1 vanish for every nonlinear parameter; the + // derivative of phi_2 with respect to nonlinear parameter k is + // column k * 3 + 2. + for (int k = 0; k < 5; ++k) + std::fill(dphi + (k * 3) * n, dphi + (k * 3 + 2) * n, 0.0); + double *d_mu = dphi + 2 * n; + double *d_sig = dphi + 5 * n; + double *d_C = dphi + 8 * n; + double *d_r = dphi + 11 * n; + double *d_q = dphi + 14 * n; + for (ssize_t i = 0; i < n; ++i) { + const double xi = x[i]; + const double dx = xi - mu; + const double dxb = xi - mu_b; + const double u = dx * inv_sig; + const double ub = dxb * inv_sig; + const double G = fast_exp(-0.5 * u * u); + const double Gb = fast_exp(-0.5 * ub * ub); + const double H = 0.5 * (1.0 - fast_erf_from_exp(u * inv_sqrt2, G)); + const double Hb = + 0.5 * (1.0 - fast_erf_from_exp(ub * inv_sqrt2, Gb)); + const double ka = G + C * H; + const double kb = Gb + C * Hb; + const double dG_dmu = G * dx * inv_sig2; + const double dG_dsig = G * dx * dx * inv_sig3; + const double dH_dmu = inv_sqrt_2pi * G * inv_sig; + const double dH_dsig = inv_sqrt_2pi * G * dx * inv_sig2; + const double dGb_dmu_b = Gb * dxb * inv_sig2; + const double dGb_dsig = Gb * dxb * dxb * inv_sig3; + const double dHb_dmu_b = inv_sqrt_2pi * Gb * inv_sig; + const double dHb_dsig = inv_sqrt_2pi * Gb * dxb * inv_sig2; + const double dka_dmu = dG_dmu + C * dH_dmu; + const double dka_dsig = dG_dsig + C * dH_dsig; + const double dkb_dmu_b = dGb_dmu_b + C * dHb_dmu_b; + const double dkb_dsig = dGb_dsig + C * dHb_dsig; + one[i] = 1.0; + minus_x[i] = -xi; + shape[i] = ka + q * kb; + d_mu[i] = dka_dmu + q * r * dkb_dmu_b; + d_sig[i] = dka_dsig + q * dkb_dsig; + d_C[i] = H + q * Hb; + d_r[i] = q * mu * dkb_dmu_b; + d_q[i] = kb; + } + } + static bool is_valid(const std::vector &par) { return par[3] > 0.0; } @@ -850,20 +1245,6 @@ struct GaussianChargeSharingKb { return {p0, p1, mu, sigma, N, C, kb_mean, kb_frac}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double slope_scale, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.1 * slope_scale; - steps[2] = 0.05 * x_range; - steps[3] = 0.05 * x_range; - steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); - steps[5] = std::max(0.1 * std::abs(start[5]), 0.01); - steps[6] = std::max(0.01 * std::abs(start[6]), 1e-3); - steps[7] = std::max(0.1 * std::abs(start[7]), 0.01); - } }; // _____________________________________________________________________ @@ -930,12 +1311,12 @@ struct RisingScurve { const double dx = x - p2; const double z = dx * inv_sqrt2 / p3; - const double step = 0.5 * (1.0 + fast_erf(z)); + const double e = std::exp(-z * z); + const double step = 0.5 * (1.0 + fast_erf_from_exp(z, e)); const double amp = p4 + p5 * dx; f = (p0 + p1 * x) + step * amp; - const double e = std::exp(-z * z); const double dSdp2 = -inv_sqrt_2pi * e / p3; const double dSdp3 = -inv_sqrt_2pi * e * dx / (p3 * p3); @@ -947,6 +1328,79 @@ struct RisingScurve { g[5] = step * dx; // df/dp5 } + /** + * @brief p0, p1, A and C are linear: + * f = p0 * 1 + p1 * x + A * S + C * S * (x - mu). + */ + static constexpr std::array linear_par = {0, 1, 4, 5}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 2> &dphi) { + const double p2 = par[2]; + const double p3 = par[3]; + + const double inv_p3 = 1.0 / p3; + const double dx = x - p2; + const double z = dx * inv_sqrt2 * inv_p3; + const double e = std::exp(-z * z); + const double step = 0.5 * (1.0 + fast_erf_from_exp(z, e)); + + const double dSdp2 = -inv_sqrt_2pi * e * inv_p3; + const double dSdp3 = dSdp2 * dx * inv_p3; + + phi[0] = 1.0; + phi[1] = x; + phi[2] = step; + phi[3] = step * dx; + dphi[0][0] = 0.0; + dphi[0][1] = 0.0; + dphi[0][2] = dSdp2; // dS/dmu + dphi[0][3] = dSdp2 * dx - step; // d(S dx)/dmu + dphi[1][0] = 0.0; + dphi[1][1] = 0.0; + dphi[1][2] = dSdp3; // dS/dsigma + dphi[1][3] = dSdp3 * dx; // d(S dx)/dsigma + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double p2 = par[2]; + const double inv_p3 = 1.0 / par[3]; + const double cz = inv_sqrt2 * inv_p3; + const double ce = -inv_sqrt_2pi * inv_p3; + double *one = phi; + double *x_col = phi + n; + double *step_col = phi + 2 * n; + double *step_dx = phi + 3 * n; + // d phi_0 and d phi_1 vanish for mu and sigma. + std::fill(dphi, dphi + 2 * n, 0.0); + std::fill(dphi + 4 * n, dphi + 6 * n, 0.0); + double *dmu_step = dphi + 2 * n; + double *dmu_step_dx = dphi + 3 * n; + double *dsig_step = dphi + 6 * n; + double *dsig_step_dx = dphi + 7 * n; + for (ssize_t i = 0; i < n; ++i) { + const double xi = x[i]; + const double dx = xi - p2; + const double z = dx * cz; + const double e = fast_exp(-z * z); + const double step = 0.5 * (1.0 + fast_erf_from_exp(z, e)); + const double dSdp2 = ce * e; + const double dSdp3 = dSdp2 * dx * inv_p3; + one[i] = 1.0; + x_col[i] = xi; + step_col[i] = step; + step_dx[i] = step * dx; + dmu_step[i] = dSdp2; + dmu_step_dx[i] = dSdp2 * dx - step; + dsig_step[i] = dSdp3; + dsig_step_dx[i] = dSdp3 * dx; + } + } + /** @brief Reject degenerate width (zero transition width). */ static bool is_valid(const std::vector &par) { return par[3] != 0.0; @@ -1017,18 +1471,6 @@ struct RisingScurve { return {p0, p1, p2, p3, p4, p5}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double slope_scale, - std::array &steps) { - steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); - steps[1] = 0.1 * slope_scale; - steps[2] = 0.05 * x_range; - steps[3] = 0.05 * x_range; - steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); - steps[5] = 0.1 * slope_scale; - } }; // _____________________________________________________________________ @@ -1083,12 +1525,12 @@ struct FallingScurve { const double dx = x - p2; const double z = dx * inv_sqrt2 / p3; - const double step = 0.5 * (1.0 - fast_erf(z)); + const double e = std::exp(-z * z); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); const double amp = p4 + p5 * dx; f = (p0 + p1 * x) + step * amp; - const double e = std::exp(-z * z); const double dSdp2 = +inv_sqrt_2pi * e / p3; // sign flipped vs rising const double dSdp3 = +inv_sqrt_2pi * e * dx / (p3 * p3); @@ -1100,6 +1542,80 @@ struct FallingScurve { g[5] = step * dx; } + /** + * @brief p0, p1, A and C are linear: + * f = p0 * 1 + p1 * x + A * S + C * S * (x - mu). + */ + static constexpr std::array linear_par = {0, 1, 4, 5}; + + static void basis_and_grad(double x, const std::vector &par, + std::array &phi, + std::array, 2> &dphi) { + const double p2 = par[2]; + const double p3 = par[3]; + + const double inv_p3 = 1.0 / p3; + const double dx = x - p2; + const double z = dx * inv_sqrt2 * inv_p3; + const double e = std::exp(-z * z); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); + + const double dSdp2 = + inv_sqrt_2pi * e * inv_p3; // sign flipped vs rising + const double dSdp3 = dSdp2 * dx * inv_p3; + + phi[0] = 1.0; + phi[1] = x; + phi[2] = step; + phi[3] = step * dx; + dphi[0][0] = 0.0; + dphi[0][1] = 0.0; + dphi[0][2] = dSdp2; // dS/dmu + dphi[0][3] = dSdp2 * dx - step; // d(S dx)/dmu + dphi[1][0] = 0.0; + dphi[1][1] = 0.0; + dphi[1][2] = dSdp3; // dS/dsigma + dphi[1][3] = dSdp3 * dx; // d(S dx)/dsigma + } + + /** @brief basis_and_grad for all points, one column per function. */ + static void basis_columns(const double *__restrict x, ssize_t n, + const std::vector &par, + double *__restrict phi, double *__restrict dphi) { + const double p2 = par[2]; + const double inv_p3 = 1.0 / par[3]; + const double cz = inv_sqrt2 * inv_p3; + const double ce = inv_sqrt_2pi * inv_p3; // sign flipped vs rising + double *one = phi; + double *x_col = phi + n; + double *step_col = phi + 2 * n; + double *step_dx = phi + 3 * n; + // d phi_0 and d phi_1 vanish for mu and sigma. + std::fill(dphi, dphi + 2 * n, 0.0); + std::fill(dphi + 4 * n, dphi + 6 * n, 0.0); + double *dmu_step = dphi + 2 * n; + double *dmu_step_dx = dphi + 3 * n; + double *dsig_step = dphi + 6 * n; + double *dsig_step_dx = dphi + 7 * n; + for (ssize_t i = 0; i < n; ++i) { + const double xi = x[i]; + const double dx = xi - p2; + const double z = dx * cz; + const double e = fast_exp(-z * z); + const double step = 0.5 * (1.0 - fast_erf_from_exp(z, e)); + const double dSdp2 = ce * e; + const double dSdp3 = dSdp2 * dx * inv_p3; + one[i] = 1.0; + x_col[i] = xi; + step_col[i] = step; + step_dx[i] = step * dx; + dmu_step[i] = dSdp2; + dmu_step_dx[i] = dSdp2 * dx - step; + dsig_step[i] = dSdp3; + dsig_step_dx[i] = dSdp3 * dx; + } + } + /** @brief Reject degenerate width (zero transition width). */ static bool is_valid(const std::vector &par) { return par[3] != 0.0; @@ -1170,14 +1686,6 @@ struct FallingScurve { return {p0, p1, p2, p3, p4, p5}; } - - static void compute_steps(const std::array &start, - double x_range, double y_range, - double slope_scale, - std::array &steps) { - RisingScurve::compute_steps(start, x_range, y_range, slope_scale, - steps); - } }; } // namespace aare::model diff --git a/include/aare/NDArray.hpp b/include/aare/NDArray.hpp index 43964028..3a2fbda3 100644 --- a/include/aare/NDArray.hpp +++ b/include/aare/NDArray.hpp @@ -27,6 +27,8 @@ class NDArray : public ArrayExpr, Ndim> { T *data_; public: + static constexpr bool is_leaf = true; + /////////////////////////////////////////////////////////////////////////////// // Constructors // @@ -64,14 +66,20 @@ class NDArray : public ArrayExpr, Ndim> { /** * @brief Construct a new NDArray object from a NDView. - * @note The data is copied from the view to the NDArray. + * @note The data is copied from the view to the NDArray. The view is + * only read, so both NDView and NDView are + * accepted. * * @param v view of data to initialize the NDArray with */ - explicit NDArray(const NDView v) : NDArray(v.shape()) { + explicit NDArray(NDView v) : NDArray(v.shape()) { std::copy(v.begin(), v.end(), begin()); } + // Keeps an exact match for mutable views so that they do not pick the + // element-wise ArrayExpr constructor. + explicit NDArray(NDView v) : NDArray(NDView(v)) {} + /** * @brief Construct a new NDArray object from an std::array. */ @@ -135,9 +143,7 @@ class NDArray : public ArrayExpr, Ndim> { */ template NDArray(ArrayExpr &&expr) : NDArray(expr.shape()) { - for (size_t i = 0; i < size_; ++i) { - data_[i] = expr[i]; - } + *this = expr; } /** @@ -146,6 +152,24 @@ class NDArray : public ArrayExpr, Ndim> { */ ~NDArray() { delete[] data_; } + /** + * @brief Copy data from a view of matching shape into this array without + * reallocating. + * @param v view to copy from, must have the same shape as this array. + * Accepts NDView and NDView. + * @throws std::runtime_error if the shapes differ + * + * Use this instead of assigning a new NDArray when the buffer needs to be + * kept, for example when the array is part of a preallocated pool. + */ + void copy_from(NDView v) { + if (v.shape() != shape_) { + throw std::runtime_error(LOCATION + + "Shape mismatch in NDArray::copy_from"); + } + std::copy(v.begin(), v.end(), begin()); + } + /////////////////////////////////////////////////////////////////////////////// // Iterators and indexing // @@ -307,6 +331,26 @@ class NDArray : public ArrayExpr, Ndim> { return *this; } + /** + * @brief Evaluate an ArrayExpr into the NDArray. Reuses the existing + * buffer if the shape matches. + */ + template NDArray &operator=(const ArrayExpr &expr) { + if (expr.shape() != shape_) { + // Evaluate before releasing the buffer, expr might refer to it + NDArray tmp(expr.shape()); + tmp = expr; + return *this = std::move(tmp); + } + // Local copies, see elementwise() + const auto e = operand(expr); + T *data = data_; + for (size_t i = 0, n = size_; i < n; ++i) { + data[i] = e[i]; + } + return *this; + } + /////////////////////////////////////////////////////////////////////////////// // Math operators // @@ -320,10 +364,7 @@ class NDArray : public ArrayExpr, Ndim> { throw(std::runtime_error( "Shape of NDArray must match for operator +=")); - for (size_t i = 0; i < size_; ++i) { - data_[i] += other.data_[i]; - } - return *this; + return elementwise(other.data_, std::plus()); } /** @@ -334,10 +375,7 @@ class NDArray : public ArrayExpr, Ndim> { throw(std::runtime_error( "Shape of NDArray must match for operator -=")); - for (size_t i = 0; i < size_; ++i) { - data_[i] -= other.data_[i]; - } - return *this; + return elementwise(other.data_, std::minus()); } /** @@ -348,10 +386,7 @@ class NDArray : public ArrayExpr, Ndim> { throw(std::runtime_error( "Shape of NDArray must match for operator *=")); - for (size_t i = 0; i < size_; ++i) { - data_[i] *= other.data_[i]; - } - return *this; + return elementwise(other.data_, std::multiplies()); } /** @@ -362,13 +397,9 @@ class NDArray : public ArrayExpr, Ndim> { */ template NDArray &operator/=(const NDArray &other) { // check shape - if (shape_ == other.shape()) { - for (size_t i = 0; i < size_; ++i) { - data_[i] /= other(i); - } - return *this; - } - throw(std::runtime_error("Shape of NDArray must match")); + if (shape_ != other.shape()) + throw(std::runtime_error("Shape of NDArray must match")); + return elementwise(other.data(), std::divides<>()); } /** @@ -383,36 +414,28 @@ class NDArray : public ArrayExpr, Ndim> { * @brief Add a scalar value to all elements in the NDArray. */ NDArray &operator+=(const T &value) { - for (size_t i = 0; i < size_; ++i) - data_[i] += value; - return *this; + return elementwise(value, std::plus()); } /** * @brief Subtract a scalar value to all elements in the NDArray. */ NDArray &operator-=(const T &value) { - for (size_t i = 0; i < size_; ++i) - data_[i] -= value; - return *this; + return elementwise(value, std::minus()); } /** * @brief Multiply all elements in the NDArray with a scalar value */ NDArray &operator*=(const T &value) { - for (size_t i = 0; i < size_; ++i) - data_[i] *= value; - return *this; + return elementwise(value, std::multiplies()); } /** * @brief Divide all elements in the NDArray with a scalar value */ NDArray &operator/=(const T &value) { - for (size_t i = 0; i < size_; ++i) - data_[i] /= value; - return *this; + return elementwise(value, std::divides()); } /** @@ -420,53 +443,7 @@ class NDArray : public ArrayExpr, Ndim> { * Used for example to mask out gain bits for Jungfrau detectors. */ NDArray &operator&=(const T &mask) { - for (auto it = begin(); it != end(); ++it) - *it &= mask; - return *this; - } - - /** - * @brief Operator + with a scalar value. Returns a new NDArray. - * - * TODO! Expression template version of this? - */ - NDArray operator+(const T &value) { - NDArray result = *this; - result += value; - return result; - } - - /** - * @brief Operator - with a scalar value. Returns a new NDArray. - * - * TODO! Expression template version of this? - */ - NDArray operator-(const T &value) { - NDArray result = *this; - result -= value; - return result; - } - - /** - * @brief Operator * with a scalar value. Returns a new NDArray. - * - * TODO! Expression template version of this? - */ - NDArray operator*(const T &value) { - NDArray result = *this; - result *= value; - return result; - } - - /** - * @brief Operator / with a scalar value. Returns a new NDArray. - * - * TODO! Expression template version of this? - */ - NDArray operator/(const T &value) { - NDArray result = *this; - result /= value; - return result; + return elementwise(mask, std::bit_and()); } /** @@ -492,19 +469,16 @@ class NDArray : public ArrayExpr, Ndim> { * @brief Compute the square root of all elements in the NDArray. */ void sqrt() { - for (size_t i = 0; i < size_; ++i) { - data_[i] = std::sqrt(data_[i]); + T *data = data_; + for (size_t i = 0, n = size_; i < n; ++i) { + data[i] = std::sqrt(data[i]); } } /* * @brief Prefix increment operator. Increments all elements by 1. */ - NDArray &operator++() { - for (size_t i = 0; i < size_; ++i) - data_[i] += T{1}; - return *this; - } + NDArray &operator++() { return elementwise(T{1}, std::plus()); } /** @brief Create a mutable view of the NDArray. */ NDView view() { return NDView{data_, shape_}; } @@ -515,6 +489,31 @@ class NDArray : public ArrayExpr, Ndim> { } private: + /** + * @brief Apply data_[i] = op(data_[i], value) to all elements. + * + * The members are copied to locals since a store to data_[i] may alias + * them for some T (uint8_t, int64_t, ...). The compiler then has to reload + * them every iteration, which blocks vectorization. + */ + template NDArray &elementwise(T value, Op op) { + T *data = data_; + for (size_t i = 0, n = size_; i < n; ++i) { + data[i] = op(data[i], value); + } + return *this; + } + + /** @brief Apply data_[i] = op(data_[i], other[i]) to all elements. */ + template + NDArray &elementwise(const V *other, Op op) { + T *data = data_; + for (size_t i = 0, n = size_; i < n; ++i) { + data[i] = op(data[i], other[i]); + } + return *this; + } + /** * @brief Reset the NDArray to an empty state. Dropping the ownership of * the data. Used internally for move operations to avoid double free or diff --git a/include/aare/NDView.hpp b/include/aare/NDView.hpp index c507b41f..715127c9 100644 --- a/include/aare/NDView.hpp +++ b/include/aare/NDView.hpp @@ -173,8 +173,7 @@ class NDView : public ArrayExpr, Ndim> { } NDView &operator=(const T val) { - for (auto it = begin(); it != end(); ++it) - *it = val; + std::fill(begin(), end(), val); return *this; } @@ -232,15 +231,19 @@ class NDView : public ArrayExpr, Ndim> { template NDView &elemenwise(T val, BinaryOperation op) { - for (uint64_t i = 0; i != size_; ++i) { - buffer_[i] = op(buffer_[i], val); + // Local copies since a store to buffer_[i] may alias the members + T *buffer = buffer_; + for (uint64_t i = 0, n = size_; i != n; ++i) { + buffer[i] = op(buffer[i], val); } return *this; } template NDView &elemenwise(const NDView &other, BinaryOperation op) { - for (uint64_t i = 0; i != size_; ++i) { - buffer_[i] = op(buffer_[i], other.buffer_[i]); + T *buffer = buffer_; + const T *other_buffer = other.buffer_; + for (uint64_t i = 0, n = size_; i != n; ++i) { + buffer[i] = op(buffer[i], other_buffer[i]); } return *this; } diff --git a/include/aare/Pedestal.hpp b/include/aare/Pedestal.hpp index 48bb6568..33ad77e9 100644 --- a/include/aare/Pedestal.hpp +++ b/include/aare/Pedestal.hpp @@ -3,15 +3,19 @@ #include "aare/Frame.hpp" #include "aare/NDArray.hpp" #include "aare/NDView.hpp" +#include #include namespace aare { /** - * @brief Calculate the pedestal of a series of frames. Can be used as - * standalone but mostly used in the ClusterFinder. + * @brief Maintain per-pixel mean and population standard deviation. * - * @tparam SUM_TYPE type of the sum + * Each pixel accumulates its first n_samples values. Subsequent pushes update + * the exponential moving average with a smoothing factor of 1 / n_samples. + * Statistics are available during initialization and are zero for empty pixels. + * Internal moments and the private variance intermediate use double precision. + * @tparam SUM_TYPE Type returned for the mean and standard deviation. */ template class Pedestal { uint32_t m_rows; @@ -20,9 +24,8 @@ template class Pedestal { uint32_t m_samples; NDArray m_cur_samples; - // TODO! in case of int needs to be changed to uint64_t - NDArray m_sum; - NDArray m_sum2; + NDArray m_sum; + NDArray m_sum2; // Cache mean since it is used over and over in the ClusterFinder // This optimization is related to the access pattern of the ClusterFinder @@ -30,58 +33,76 @@ template class Pedestal { NDArray m_mean; public: + /** + * @brief Construct an empty pedestal with zero mean and standard deviation. + * @param rows Number of image rows. + * @param cols Number of image columns. + * @param n_samples Number of initialization samples per pixel and + * reciprocal of the weight assigned to each subsequent value. + * @throws std::runtime_error if rows, cols, or n_samples is zero. + */ Pedestal(uint32_t rows, uint32_t cols, uint32_t n_samples = 1000) : m_rows(rows), m_cols(cols), m_samples(n_samples), m_cur_samples(NDArray({rows, cols}, 0)), - m_sum(NDArray({rows, cols})), - m_sum2(NDArray({rows, cols})), - m_mean(NDArray({rows, cols})) { - assert(rows > 0 && cols > 0 && n_samples > 0); - m_sum = 0; - m_sum2 = 0; - m_mean = 0; + m_sum(NDArray({rows, cols}, 0.0)), + m_sum2(NDArray({rows, cols}, 0.0)), + m_mean(NDArray({rows, cols}, SUM_TYPE(0))) { + if (!(rows > 0 && cols > 0 && n_samples > 0)) { + throw std::runtime_error( + fmt::format("Invalid parameters for Pedestal: rows={}, " + "cols={}, n_samples={} need to be positive", + rows, cols, n_samples)); + } } + ~Pedestal() = default; - NDArray mean() { return m_mean; } - + /** + * @brief Return a non-owning view of the cached mean. + * @note The caller must treat the data as read-only and must not retain the + * view after this object is destroyed, moved, or assigned. + */ NDView view() const { return m_mean.view(); } + /** + * @brief Return the cached mean at (row, col), or zero for an empty pixel. + * @pre row and col are valid pixel indices. + */ SUM_TYPE mean(const uint32_t row, const uint32_t col) const { return m_mean(row, col); } + /** @brief Return a copy of the cached mean. */ + NDArray mean() const { return m_mean; } + + /** + * @brief Calculate the population standard deviation at (row, col). + * @pre row and col are valid pixel indices. + * @note Variance is normalized by the pixel's current sample count. Empty + * pixels return zero, and negative variance from roundoff is clamped to + * zero. + */ SUM_TYPE std(const uint32_t row, const uint32_t col) const { return std::sqrt(variance(row, col)); } - SUM_TYPE variance(const uint32_t row, const uint32_t col) const { - if (m_cur_samples(row, col) == 0) { - return 0.0; + /** + * @brief Calculate and return the population standard deviation of every + * pixel. Empty pixels return zero. + */ + NDArray std() const { + NDArray res({m_rows, m_cols}); + for (uint32_t row = 0; row < m_rows; ++row) { + for (uint32_t col = 0; col < m_cols; ++col) { + res(row, col) = std(row, col); + } } - return m_sum2(row, col) / m_cur_samples(row, col) - - mean(row, col) * mean(row, col); - } - - NDArray variance() { - NDArray variance_array({m_rows, m_cols}); - for (uint32_t i = 0; i < m_rows * m_cols; i++) { - variance_array(i / m_cols, i % m_cols) = - variance(i / m_cols, i % m_cols); - } - return variance_array; - } - - NDArray std() { - NDArray standard_deviation_array({m_rows, m_cols}); - for (uint32_t i = 0; i < m_rows * m_cols; i++) { - standard_deviation_array(i / m_cols, i % m_cols) = - std(i / m_cols, i % m_cols); - } - - return standard_deviation_array; + return res; } + /** + * @brief Zero the moments, cached mean, and sample counts of every pixel. + */ void clear() { m_sum = 0; m_sum2 = 0; @@ -89,6 +110,10 @@ template class Pedestal { m_mean = 0; } + /** + * @brief Zero the moments, cached mean, and sample count at (row, col). + * @pre row and col are valid pixel indices. + */ void clear(const uint32_t row, const uint32_t col) { m_sum(row, col) = 0; m_sum2(row, col) = 0; @@ -96,9 +121,13 @@ template class Pedestal { m_mean(row, col) = 0; } + /** + * @brief Accumulate or exponentially update every pixel and its cached + * mean. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs. + */ template void push(NDView frame) { - assert(frame.size() == m_rows * m_cols); - // TODO! move away from m_rows, m_cols if (frame.shape() != std::array{m_rows, m_cols}) { throw std::runtime_error( @@ -112,11 +141,18 @@ template class Pedestal { } } + /** + * @brief Push only pixels whose absolute difference from the cached mean is + * strictly less than their threshold. + * @param frame Frame whose shape must exactly match the pedestal. + * @param threshold Per-pixel thresholds with the same shape as the + * pedestal. + * @throws std::runtime_error if either shape differs. + * @note Rejected pixels keep their statistics and sample counts unchanged. + */ template void push_with_threshold(const NDView frame, const NDView threshold) { - assert(frame.size() == m_rows * m_cols); - // TODO! move away from m_rows, m_cols if (frame.shape() != std::array{m_rows, m_cols}) { throw std::runtime_error( @@ -139,45 +175,43 @@ template class Pedestal { } /** - * Push but don't update the cached mean. Speeds up the process - * when initializing the pedestal. - * + * @brief Accumulate or exponentially update every pixel from a Frame. + * @tparam T Actual pixel type stored in frame; this is not runtime-checked. + * @param frame Frame whose shape must exactly match the pedestal. + * @throws std::runtime_error if the shape differs. */ - template void push_no_update(NDView frame) { - assert(frame.size() == m_rows * m_cols); + template void push(Frame &frame) { push(frame.view()); } - // TODO! move away from m_rows, m_cols - if (frame.shape() != std::array{m_rows, m_cols}) { - throw std::runtime_error( - "Frame shape does not match pedestal shape"); - } - - for (size_t row = 0; row < m_rows; row++) { - for (size_t col = 0; col < m_cols; col++) { - push_no_update(row, col, frame(row, col)); - } - } - } - - template void push(Frame &frame) { - assert(frame.rows() == static_cast(m_rows) && - frame.cols() == static_cast(m_cols)); - push(frame.view()); - } - - // getter functions + /** @brief Return the number of image rows. */ uint32_t rows() const { return m_rows; } - uint32_t cols() const { return m_cols; } - uint32_t n_samples() const { return m_samples; } - NDArray cur_samples() const { return m_cur_samples; } - NDArray get_sum() const { return m_sum; } - NDArray get_sum2() const { return m_sum2; } - // pixel level operations (should be refactored to allow users to implement - // their own pixel level operations) + /** @brief Return the number of image columns. */ + uint32_t cols() const { return m_cols; } + + /** + * @brief Return the initialization sample count per pixel and steady-state + * update-weight denominator. + */ + uint32_t n_samples() const { return m_samples; } + + /** + * @brief Return a copy of the per-pixel initialization sample counts. + * @note Each count is in [0, n_samples] and does not change during + * steady-state pushes. Thresholded pushes can leave counts unequal. + */ + NDArray cur_samples() const { return m_cur_samples; } + + /** + * @brief Accumulate or exponentially update one pixel and its cached mean. + * @param row Pixel row. + * @param col Pixel column. + * @param val_ New pixel value, with weight 1 / n_samples after + * initialization. + * @pre row and col are valid pixel indices. + */ template void push(const uint32_t row, const uint32_t col, const T val_) { - SUM_TYPE val = static_cast(val_); + const auto val = static_cast(val_); if (m_cur_samples(row, col) < m_samples) { m_sum(row, col) += val; m_sum2(row, col) += val * val; @@ -191,34 +225,16 @@ template class Pedestal { m_mean(row, col) = m_sum(row, col) / m_cur_samples(row, col); } - template - void push_no_update(const uint32_t row, const uint32_t col, const T val_) { - SUM_TYPE val = static_cast(val_); - if (m_cur_samples(row, col) < m_samples) { - m_sum(row, col) += val; - m_sum2(row, col) += val * val; - m_cur_samples(row, col)++; - } else { - m_sum(row, col) += val - m_sum(row, col) / m_cur_samples(row, col); - m_sum2(row, col) += - val * val - m_sum2(row, col) / m_cur_samples(row, col); + private: + double variance(const uint32_t row, const uint32_t col) const { + if (m_cur_samples(row, col) == 0) { + return 0.0; } - } - - /** - * @brief Update the mean of the pedestal. This is used after having done - * push_no_update. It is not necessary to call this function after push. - */ - void update_mean() { m_mean = m_sum / m_cur_samples; } - - template - void push_fast(const uint32_t row, const uint32_t col, const T val_) { - // Assume we reached the steady state where all pixels have - // m_samples samples - SUM_TYPE val = static_cast(val_); - m_sum(row, col) += val - m_sum(row, col) / m_samples; - m_sum2(row, col) += val * val - m_sum2(row, col) / m_samples; - m_mean(row, col) = m_sum(row, col) / m_samples; + const auto mean = m_sum(row, col) / m_cur_samples(row, col); + const auto var = + m_sum2(row, col) / m_cur_samples(row, col) - mean * mean; + // Roundoff in the moments can make a near-zero variance negative. + return std::max(var, 0.0); } }; } // namespace aare diff --git a/include/aare/ProducerConsumerQueue.hpp b/include/aare/ProducerConsumerQueue.hpp index b9ec9d02..d630490f 100644 --- a/include/aare/ProducerConsumerQueue.hpp +++ b/include/aare/ProducerConsumerQueue.hpp @@ -1,4 +1,4 @@ -// SPDX-License-Identifier: MPL-2.0 +// SPDX-License-Identifier: Apache-2.0 /* * Copyright (c) Meta Platforms, Inc. and affiliates. * @@ -18,10 +18,18 @@ // @author Bo Hu (bhu@fb.com) // @author Jordan DeLong (delong.j@fb.com) -// Changes made by PSD Detector Group: -// Copied: Line 34 constexpr std::size_t hardware_destructive_interference_size -// = 128; from folly/lang/Align.h Changed extension to .hpp Changed namespace to -// aare +// Changes made by the CPS Detector Group at PSI relative to +// folly/ProducerConsumerQueue.h in folly v2024.03.18.00 (commit 8aceb9bf727c; +// the file is identical in v2022.01.03.00 through v2024.03.18.00): +// - Replaced the include of folly/concurrency/CacheLocality.h with a local +// constexpr hardware_destructive_interference_size = 128, the value that +// folly/lang/Align.h uses except on ARM and s390x. +// - Changed the file extension to .hpp and the namespace to aare. +// - Added a default constructor equivalent to ProducerConsumerQueue(2). +// - Declared the move constructor and move assignment deleted. The queue is +// not safe to move and an earlier attempt at move support was removed. +// - The constructor throws std::invalid_argument for size < 2 instead of +// asserting, so the check is also active in release builds. #pragma once @@ -31,12 +39,14 @@ #include #include #include +#include #include #include -constexpr std::size_t hardware_destructive_interference_size = 128; namespace aare { +constexpr std::size_t hardware_destructive_interference_size = 128; + /* * ProducerConsumerQueue is a one producer and one consumer queue * without locks. @@ -46,38 +56,18 @@ template struct ProducerConsumerQueue { ProducerConsumerQueue(const ProducerConsumerQueue &) = delete; ProducerConsumerQueue &operator=(const ProducerConsumerQueue &) = delete; - - ProducerConsumerQueue(ProducerConsumerQueue &&other) { - size_ = other.size_; - records_ = other.records_; - other.records_ = nullptr; - readIndex_ = other.readIndex_.load(std::memory_order_acquire); - writeIndex_ = other.writeIndex_.load(std::memory_order_acquire); - } - ProducerConsumerQueue &operator=(ProducerConsumerQueue &&other) { - size_ = other.size_; - records_ = other.records_; - other.records_ = nullptr; - readIndex_ = other.readIndex_.load(std::memory_order_acquire); - writeIndex_ = other.writeIndex_.load(std::memory_order_acquire); - return *this; - } + ProducerConsumerQueue(ProducerConsumerQueue &&) = delete; + ProducerConsumerQueue &operator=(ProducerConsumerQueue &&) = delete; ProducerConsumerQueue() : ProducerConsumerQueue(2) {}; - // size must be >= 2. + // size must be >= 2, otherwise std::invalid_argument is thrown. // // Also, note that the number of usable slots in the queue at any // given time is actually (size-1), so if you start with an empty queue, // isFull() will return true after size-1 insertions. explicit ProducerConsumerQueue(uint32_t size) - : size_(size), - records_(static_cast(std::malloc(sizeof(T) * size))), - readIndex_(0), writeIndex_(0) { - assert(size >= 2); - if (!records_) { - throw std::bad_alloc(); - } - } + : size_(size), records_(allocate_records(size)), readIndex_(0), + writeIndex_(0) {} ~ProducerConsumerQueue() { // We need to destruct anything that may still exist in our queue. @@ -192,11 +182,23 @@ template struct ProducerConsumerQueue { private: using AtomicIndex = std::atomic; + // Validates size before allocating, so a throw cannot leak the buffer. + static T *allocate_records(uint32_t size) { + if (size < 2) { + throw std::invalid_argument( + "ProducerConsumerQueue size must be at least 2, got " + + std::to_string(size)); + } + auto *records = static_cast(std::malloc(sizeof(T) * size)); + if (!records) { + throw std::bad_alloc(); + } + return records; + } + char pad0_[hardware_destructive_interference_size]; - // const uint32_t size_; - uint32_t size_; - // T *const records_; - T *records_; + const uint32_t size_; + T *const records_; alignas(hardware_destructive_interference_size) AtomicIndex readIndex_; alignas(hardware_destructive_interference_size) AtomicIndex writeIndex_; diff --git a/include/aare/ROI.hpp b/include/aare/ROI.hpp new file mode 100644 index 00000000..36838f43 --- /dev/null +++ b/include/aare/ROI.hpp @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include "aare/defs.hpp" +#include +#include +#include +#include +#include +#include + +namespace aare { + +class ROIGeometry; // forward declaration to avoid circular dependency +class DetectorGeometry; // forward declaration to avoid circular dependency + +struct ROI { + ssize_t xmin{}; + ssize_t xmax{}; + ssize_t ymin{}; + ssize_t ymax{}; + + ssize_t height() const { return ymax - ymin; } + ssize_t width() const { return xmax - xmin; } + bool contains(ssize_t x, ssize_t y) const { + return x >= xmin && x < xmax && y >= ymin && y < ymax; + } + + bool operator==(const ROI &other) const { + return xmin == other.xmin && xmax == other.xmax && ymin == other.ymin && + ymax == other.ymax; + } +}; + +/** + * @brief Merge all consecutive ROIs in a vector into a single ROI. + * @param rois vector of ROIs to merge + * @return vector of merged ROIs + * @tparam horizontally_aligned true if the ROIs are horizontally aligned, false + * otherwise + * @tparam vertically_aligned true if the ROIs are vertically aligned, false + * otherwise + */ +template +std::vector merge_consecutive_rois(std::vector &rois) { + + if constexpr (horizontally_aligned && vertically_aligned) { + throw std::runtime_error( + LOCATION + "Vector of the same ROI? Cannot merge ROIs both " + "horizontally and vertically at the same time."); + } + + if (rois.empty()) { + return {}; + } + if (rois.size() == 1) { + return rois; + } + + auto merge_along_x = [](std::vector in_rois) { + std::sort(in_rois.begin(), in_rois.end(), + [](const ROI &a, const ROI &b) { + return (a.ymin != b.ymin) ? (a.ymin < b.ymin) + : (a.xmin < b.xmin); + }); // N log (N) + + std::vector merged_rois; + merged_rois.reserve(in_rois.size()); + merged_rois.push_back(in_rois[0]); + + for (size_t i = 1; i < in_rois.size(); ++i) { + auto &last = merged_rois.back(); + const auto ¤t = in_rois[i]; + if (last.ymin == current.ymin && last.ymax == current.ymax && + last.xmax == current.xmin) { + // merge + last.xmax = current.xmax; + } else { + merged_rois.push_back(current); + } + } + return merged_rois; + }; + + auto merge_along_y = [](std::vector in_rois) { + std::sort(in_rois.begin(), in_rois.end(), + [](const ROI &a, const ROI &b) { + return (a.xmin != b.xmin) ? (a.xmin < b.xmin) + : (a.ymin < b.ymin); + }); + + std::vector merged_rois; + merged_rois.reserve(in_rois.size()); + merged_rois.push_back(in_rois[0]); + + for (size_t i = 1; i < in_rois.size(); ++i) { + auto &last = merged_rois.back(); + const auto ¤t = in_rois[i]; + if (last.xmin == current.xmin && last.xmax == current.xmax && + last.ymax == current.ymin) { + last.ymax = current.ymax; + } else { + merged_rois.push_back(current); + } + } + return merged_rois; + }; + + if constexpr (horizontally_aligned) { + return merge_along_y(rois); // one sort + one pass + } else if constexpr (vertically_aligned) { + return merge_along_x(rois); // one sort + one pass + } else { + return merge_along_y(merge_along_x(rois)); // generic case: two passes + } +} + +/** + * @brief Check if the ROI covers the entire detector geometry + * @param roi Region of interest + * @param geometry Detector geometry + * @return true if the ROI covers the entire detector geometry, false otherwise + */ +bool complete_ROI(const ROI &roi, const DetectorGeometry &geometry); + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry); + +bool complete_ROI(const ROIGeometry &roi, const DetectorGeometry &geometry); + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry); + +} // namespace aare \ No newline at end of file diff --git a/include/aare/ROIGeometry.hpp b/include/aare/ROIGeometry.hpp index 05450df5..a8a90c37 100644 --- a/include/aare/ROIGeometry.hpp +++ b/include/aare/ROIGeometry.hpp @@ -1,6 +1,9 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once #include "aare/DetectorGeometry.hpp" -#include "aare/defs.hpp" +#include "aare/ROI.hpp" +#include +#include namespace aare { @@ -17,11 +20,6 @@ class ROIGeometry { */ ROIGeometry(const ROI &roi, DetectorGeometry &geometry); - /** @brief Constructor for ROI geometry expanding over full detector - * @param geometry general detector geometry - */ - ROIGeometry(DetectorGeometry &geometry); - /// @brief Get number of modules in the ROI size_t num_modules_in_roi() const; diff --git a/include/aare/RawFile.hpp b/include/aare/RawFile.hpp index 999b737f..4ef92962 100644 --- a/include/aare/RawFile.hpp +++ b/include/aare/RawFile.hpp @@ -1,6 +1,5 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once -#include "aare/DetectorGeometry.hpp" #include "aare/FileInterface.hpp" #include "aare/Frame.hpp" #include "aare/NDArray.hpp" //for pixel map @@ -30,17 +29,18 @@ class RawFile : public FileInterface { RawMasterFile m_master; size_t m_current_frame{}; - DetectorGeometry m_geometry; - - /// @brief Geometries e.g. number of modules, size etc. for each ROI std::vector m_ROI_geometries; + /// @brief Minimum frame count across the selected raw subfile series. + size_t m_frames_in_file{}; + public: /** * @brief RawFile constructor * @param fname path to the master file (.json) * @param mode file mode (only "r" is supported at the moment) - + * @throws std::runtime_error if frame padding is disabled and the frame + * discard policy is not DiscardPartial. */ RawFile(const std::filesystem::path &fname, const std::string &mode = "r"); virtual ~RawFile() override = default; @@ -104,6 +104,7 @@ class RawFile : public FileInterface { size_t bytes_per_pixel() const; void seek(size_t frame_index) override; size_t tell() override; + /// @brief Minimum actual frame count across all subfiles and ROIs. size_t total_frames() const override; size_t rows() const override; /** @@ -122,7 +123,7 @@ class RawFile : public FileInterface { size_t n_modules() const; /** - * @brief number of ROIs defined + * @brief number of ROIs defined (always 1 for complete ROI) */ size_t num_rois() const; @@ -150,6 +151,9 @@ class RawFile : public FileInterface { static DetectorHeader read_header(const std::filesystem::path &fname); private: + std::runtime_error frame_error(size_t frame_index, + const std::string &message) const; + /** * @brief read the frame at the given frame index into the image buffer * @param frame_index frame number to read diff --git a/include/aare/RawMasterFile.hpp b/include/aare/RawMasterFile.hpp index d7b13a42..26ea3b41 100644 --- a/include/aare/RawMasterFile.hpp +++ b/include/aare/RawMasterFile.hpp @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MPL-2.0 #pragma once -#include "aare/defs.hpp" +#include "aare/DetectorGeometry.hpp" +#include "aare/ROI.hpp" #include #include #include @@ -88,7 +89,8 @@ class RawMasterFile { std::optional m_exptime; std::chrono::nanoseconds m_period{0}; - xy m_geometry{}; + /// @brief modules in x and y direction + xy m_detector_layout{}; xy m_udp_interfaces_per_module{1, 1}; size_t m_max_frames_per_file{}; @@ -109,12 +111,28 @@ class RawMasterFile { std::optional m_number_of_rows; std::optional m_counter_mask; - std::optional> m_rois; + /// @brief index of disabled UDP ports - index relative to UDP_port_types + std::vector m_disabled_udp_ports{}; + + /// @brief udp port types + std::optional> m_udp_port_types{}; + + /// @brief ROIs defined in master file or derived from disabled UDP ports + std::vector m_rois; + + /// @brief Detector geometry - geometry for each module + DetectorGeometry m_geometry{}; + + /// @brief ROI geometries + std::vector m_ROI_geometries; public: RawMasterFile(const std::filesystem::path &fpath); RawMasterFile(std::istream &is, const std::string &fname); // for testing + /// @brief Get the filename including path of the master file. + std::filesystem::path master_fname() const { return m_fnc.master_fname(); } + std::filesystem::path data_fname(size_t mod_id, size_t file_id) const; const std::string &version() const; //!< For example "7.2" @@ -131,10 +149,14 @@ class RawMasterFile { const FrameDiscardPolicy &frame_discard_policy() const; size_t total_frames_expected() const; - xy geometry() const; + xy detector_layout() const; size_t n_modules() const; uint8_t quad() const; + const DetectorGeometry &geometry() const; + + const std::vector &roi_geometries() const; + ReadoutMode get_reading_mode() const; std::optional analog_samples() const; @@ -143,9 +165,21 @@ class RawMasterFile { std::optional number_of_rows() const; std::optional counter_mask() const; - std::optional> rois() const; + /// @brief Get the types of UDP ports + /// @return Optional vector of UDP port types as strings (only present for + /// masterfile version >= 8.1) + std::optional> udp_port_types() const; - std::optional roi() const; + /// @brief Get the indices of disabled UDP ports + /// @return vector of indices of disabled UDP ports (empty if none are + /// disabled) + std::vector disabled_udp_ports() const; + + std::vector rois() const; + + /// @brief get roi for the case of a single ROI + /// @return ROI object (complete ROI if no roi present in master file) + ROI roi() const; ScanParameters scan_parameters() const; @@ -157,7 +191,8 @@ class RawMasterFile { private: void parse_json(std::istream &is); void parse_raw(std::istream &is); + void update_rois_from_disabled_udp_ports(); void retrieve_geometry(); }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/RawSubFile.hpp b/include/aare/RawSubFile.hpp index 50db61fa..deff6fb7 100644 --- a/include/aare/RawSubFile.hpp +++ b/include/aare/RawSubFile.hpp @@ -84,7 +84,13 @@ class RawSubFile { size_t frames_in_file() const { return m_total_frames; } + /// @brief Path of the current data file in the raw subfile series. + std::filesystem::path current_path() const; + private: + std::runtime_error frame_error(size_t frame_index, + const std::string &message) const; + template void read_with_map(std::byte *image_buf); void parse_fname(const std::filesystem::path &fname); @@ -93,4 +99,4 @@ class RawSubFile { std::filesystem::path fpath(size_t file_index) const; }; -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/include/aare/calibration.hpp b/include/aare/calibration.hpp index 28550b3a..aac09cba 100644 --- a/include/aare/calibration.hpp +++ b/include/aare/calibration.hpp @@ -11,20 +11,6 @@ namespace aare { -// Really try to convince the compile to inline this function -// TODO! Clang? -#if (defined(_MSC_VER) || defined(__INTEL_COMPILER)) -#define STRONG_INLINE __forceinline -#else -#define STRONG_INLINE inline -#endif - -#if defined(__GNUC__) -#define ALWAYS_INLINE __attribute__((always_inline)) inline -#else -#define ALWAYS_INLINE STRONG_INLINE -#endif - /** * @brief Get the gain from the raw ADC value. In Jungfrau the gain is * encoded in the left most 2 bits of the raw value. diff --git a/include/aare/decode.hpp b/include/aare/decode.hpp index a6efd187..00433109 100644 --- a/include/aare/decode.hpp +++ b/include/aare/decode.hpp @@ -16,6 +16,23 @@ void adc_sar_05_decode64to16(NDView input, void adc_sar_04_decode64to16(NDView input, NDView output); +/** + * @brief Decode packed 64-bit ADC SAR samples stored as raw bytes. + * + * Every 8 bytes of a row form one 64-bit word in native byte order. Words + * are assembled with memcpy, so the buffer does not need to be 8-byte + * aligned. The output must have the same number of rows as the input and + * input.shape(1) / 8 columns. + * @throws std::invalid_argument if the row length is not a multiple of 8 + * bytes or the output shape does not match. + */ +void adc_sar_05_06_07_08decode64to16(NDView input, + NDView output); +void adc_sar_05_decode64to16(NDView input, + NDView output); +void adc_sar_04_decode64to16(NDView input, + NDView output); + /** * @brief Called with a 32 bit unsigned integer, shift by offset * and then return the lower 24 bits as an 32 bit integer diff --git a/include/aare/defs.hpp b/include/aare/defs.hpp index 68ee37a0..39c3fee9 100644 --- a/include/aare/defs.hpp +++ b/include/aare/defs.hpp @@ -32,6 +32,26 @@ } while (0) #endif +/** + * ALWAYS_INLINE forces a function to be inlined at every call site. gcc + * refuses to inline a plain `inline` function once its translation-unit + * growth budget (--param inline-unit-growth) is used up, which happens in + * the fitting code where every model is instantiated for every minimizer; + * a per-point helper that is then called out of line also stops the + * surrounding loop from vectorising. + */ +#if (defined(_MSC_VER) || defined(__INTEL_COMPILER)) +#define STRONG_INLINE __forceinline +#else +#define STRONG_INLINE inline +#endif + +#if defined(__GNUC__) +#define ALWAYS_INLINE __attribute__((always_inline)) inline +#else +#define ALWAYS_INLINE STRONG_INLINE +#endif + namespace aare { inline constexpr size_t bits_per_byte = 8; @@ -94,19 +114,6 @@ template struct t_xy { }; using xy = t_xy; -struct ROI { - ssize_t xmin{}; - ssize_t xmax{}; - ssize_t ymin{}; - ssize_t ymax{}; - - ssize_t height() const { return ymax - ymin; } - ssize_t width() const { return xmax - xmin; } - bool contains(ssize_t x, ssize_t y) const { - return x >= xmin && x < xmax && y >= ymin && y < ymax; - } -}; - /// @brief Chip specifications for Matterhorn1 struct Matterhorn10 { constexpr static size_t nRows = 256; @@ -320,7 +327,21 @@ enum class corner : int { cBottomRight = 3 }; -enum class TimingMode { Auto, Trigger }; +enum class UDPPortPosition : uint8_t { + LEFT = 0, + RIGHT = 1, + TOP = 2, + BOTTOM = 3 +}; + +enum class TimingMode : uint8_t { + AUTO_TIMING = 0, + TRIGGER_EXPOSURE = 1, + GATED = 2, + BURST_TRIGGER = 3, + TRIGGER_GATED = 4 +}; + enum class FrameDiscardPolicy { NoDiscard, Discard, DiscardPartial }; using DataTypeVariants = std::variant; diff --git a/include/aare/hist/PedestalTrackingPixelHistogram.hpp b/include/aare/hist/PedestalTrackingPixelHistogram.hpp index c42b5972..fe489cfe 100644 --- a/include/aare/hist/PedestalTrackingPixelHistogram.hpp +++ b/include/aare/hist/PedestalTrackingPixelHistogram.hpp @@ -1,4 +1,6 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once +#include "aare/FastPedestal.hpp" #include "aare/NDArray.hpp" #include "aare/NDView.hpp" #include "aare/Pedestal.hpp" @@ -45,7 +47,7 @@ class PedestalTrackingPixelHistogram { // worker using the LOCAL row index (i.e. 0..row_count(t)-1), NOT the // global row index. Owned exclusively by worker `t` during a // dispatched fan-out. - std::vector> partial_pedestals_; + std::vector> partial_pedestals_; std::vector> partial_std_; // cached for pedestal // tracking @@ -66,6 +68,11 @@ class PedestalTrackingPixelHistogram { // Always the outermost lock; work_mutex_ is taken briefly inside it. mutable std::mutex fill_mutex_; + // Serialises producers of async_queue_ (which is SPSC) and gives + // fill_from_file exclusive ownership of the ingestion path while its + // direct, double-buffered dispatch is active. + std::mutex ingestion_mutex_; + // Async producer/consumer pipeline. SPSC queue feeds the coordinator // thread, which batches queued images before dispatching them. std::unique_ptr async_queue_; @@ -111,8 +118,23 @@ class PedestalTrackingPixelHistogram { void fill_async(NDArray &&image); + /** + * @brief Fill from ordered, parallel-read batches. + * + * After the first batch, reading batch N+1 overlaps histogram processing + * for batch N. Two fixed-capacity read buffers are allocated once and + * reused for the duration of the call. + * + * @param fname input file accepted by File + * @param max_frames maximum frames to process, or all frames for -1 + * @param verbose print periodic progress information + * @param reader_threads number of MultiThreadedFileReader workers + * @param reader_chunk_size frames claimed per reader worker and batch + */ void fill_from_file(const std::filesystem::path &fname, - ssize_t max_frames = -1, bool verbose = false); + ssize_t max_frames = -1, bool verbose = false, + std::size_t reader_threads = 2, + std::size_t reader_chunk_size = 4); void process_pedestal_file(const std::filesystem::path &fname, ssize_t max_frames = -1, bool verbose = false); diff --git a/include/aare/hist/PixelHistogram.hpp b/include/aare/hist/PixelHistogram.hpp index 05432d65..c9b199b9 100644 --- a/include/aare/hist/PixelHistogram.hpp +++ b/include/aare/hist/PixelHistogram.hpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once #include "aare/NDArray.hpp" #include "aare/NDView.hpp" diff --git a/include/aare/hist/PixelHistogramImpl.hpp b/include/aare/hist/PixelHistogramImpl.hpp index 30b468b9..9af42fe1 100644 --- a/include/aare/hist/PixelHistogramImpl.hpp +++ b/include/aare/hist/PixelHistogramImpl.hpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once /* Basic pixel histogram class with templated axis and storage type. @@ -10,6 +11,7 @@ silently dropped. #include "aare/NDArray.hpp" #include "aare/NDView.hpp" +#include #include #include #include @@ -37,6 +39,9 @@ template class PixelHistogramImpl { void fill(const NDView &frame); void fill(int row, int col, T value); void fill_unchecked(int row, int col, T value); + // Fill using a row-major pixel index, avoiding repeated 3D stride + // calculation in tiled worker loops. The index is intentionally unchecked. + void fill_flat_unchecked(std::size_t pixel, T value); NDArray values() const; // Zero-copy view of the underlying [rows x cols x n_bins] storage. @@ -98,22 +103,29 @@ void PixelHistogramImpl::fill(int row, int col, T value) { template void PixelHistogramImpl::fill_unchecked(int row, int col, T value) { + const auto pixel = + static_cast(row) * static_cast(m_cols) + + static_cast(col); + fill_flat_unchecked(pixel, value); +} + +template +void PixelHistogramImpl::fill_flat_unchecked(std::size_t pixel, + T value) { if (value < m_xmin || value >= m_xmax) { return; } int bin = static_cast((value - m_xmin) * m_scale); // Guard against floating-point rounding pushing val just below - // xmax to bin == n_bins. - if (bin >= m_n_bins) { - bin = m_n_bins - 1; - } + bin = std::clamp(bin, 0, m_n_bins - 1); + auto &cell = m_values.data()[pixel * static_cast(m_n_bins) + + static_cast(bin)]; if constexpr (std::is_integral_v) { - if (m_values(row, col, bin) >= - std::numeric_limits::max()) { + if (cell >= std::numeric_limits::max()) { return; } } - ++m_values(row, col, bin); + ++cell; } template diff --git a/src/to_string.hpp b/include/aare/to_string.hpp similarity index 83% rename from src/to_string.hpp rename to include/aare/to_string.hpp index 0bb10ada..dbb0c09f 100644 --- a/src/to_string.hpp +++ b/include/aare/to_string.hpp @@ -55,6 +55,14 @@ template T string_to(const std::string &arg) { */ template <> DetectorType string_to(const std::string &arg); +/** + * @brief Convert a string to UDPPortPosition + * @param name string representation of the UDPPortPosition + * @return UDPPortPosition + * @throw runtime_error if the string does not match any UDPPortPosition + */ +template <> UDPPortPosition string_to(const std::string &arg); + /** * @brief Convert a string to TimingMode * @param mode string representation of the TimingMode @@ -79,4 +87,12 @@ template <> FrameDiscardPolicy string_to(const std::string &arg); */ template <> DACIndex string_to(const std::string &arg); +/** + * @brief Convert a corner enum to string + * @param arg corner enum value + * @return string representation of the corner enum + * @throw runtime_error if the corner value is invalid + */ +std::string to_string(const corner &arg); + } // namespace aare \ No newline at end of file diff --git a/include/aare/utils/math_helpers.hpp b/include/aare/utils/math_helpers.hpp index 9481bd8a..6d9b3c80 100644 --- a/include/aare/utils/math_helpers.hpp +++ b/include/aare/utils/math_helpers.hpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once #include diff --git a/pyproject.toml b/pyproject.toml index 8fa52bde..6054356c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ regex = '^(?P\d+(?:\.\d+)*(?:[\.\+\w]+)?)$' result = "{version}" [build-system] -requires = ["scikit-build-core>=0.10", "pybind11", "numpy"] +requires = ["scikit-build-core>=0.11", "pybind11", "numpy"] build-backend = "scikit_build_core.build" [project] @@ -16,7 +16,8 @@ dependencies = [ "numpy", "matplotlib", ] -license = { file = "LICENSE" } +license = "MPL-2.0 AND Apache-2.0 AND MIT AND BSD-3-Clause AND LGPL-2.1-or-later" +license-files = ["LICENSES/*", "THIRD-PARTY-NOTICES.txt"] diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt index 401bf26b..7f4db4c0 100644 --- a/python/CMakeLists.txt +++ b/python/CMakeLists.txt @@ -39,7 +39,10 @@ set(PYTHON_FILES aare/ClusterFinder.py aare/ClusterVector.py aare/Cluster.py + aare/FastPedestal.py + aare/Pedestal.py aare/calibration.py + aare/factory.py aare/experimental.py aare/RawFile.py aare/transform.py diff --git a/python/aare/Cluster.py b/python/aare/Cluster.py index f2ea07eb..2da800b9 100644 --- a/python/aare/Cluster.py +++ b/python/aare/Cluster.py @@ -1,6 +1,7 @@ +# SPDX-License-Identifier: MPL-2.0 from . import _aare import numpy as np -from .ClusterFinder import _type_to_char +from .factory import _type_to_char def Cluster(x : int, y : int, data, cluster_size=(3,3), dtype = np.int32): @@ -21,4 +22,4 @@ def Cluster(x : int, y : int, data, cluster_size=(3,3), dtype = np.int32): except AttributeError: raise ValueError(f"Unsupported combination of type and cluster size: {dtype}/{cluster_size} when requesting {class_name}") - return cls(x, y, data) \ No newline at end of file + return cls(x, y, data) diff --git a/python/aare/ClusterFinder.py b/python/aare/ClusterFinder.py index eae7ad7d..d612fcb9 100644 --- a/python/aare/ClusterFinder.py +++ b/python/aare/ClusterFinder.py @@ -1,21 +1,10 @@ # SPDX-License-Identifier: MPL-2.0 from . import _aare import numpy as np +from .factory import _type_to_char _supported_cluster_sizes = [(2,2), (3,3), (5,5), (7,7), (9,9),] -def _type_to_char(dtype): - if dtype == np.int32: - return 'i' - elif dtype == np.float32: - return 'f' - elif dtype == np.float64: - return 'd' - elif dtype == np.int16: - return 'i16' - else: - raise ValueError(f"Unsupported dtype: {dtype}. Only np.int32, np.float32, and np.float64 are supported.") - def _get_class(name, cluster_size, dtype): """ Helper function to get the class based on the name, cluster size, and dtype. @@ -29,24 +18,55 @@ def _get_class(name, cluster_size, dtype): -def ClusterFinder(image_size, cluster_size=(3,3), n_sigma=5, dtype = np.int32, capacity = 1024): +def ClusterFinder(image_size, *, cluster_size=(3,3), n_sigma=5, dtype = np.int32, capacity = 1024, min_pedestal_samples = 1000): """ Factory function to create a ClusterFinder object. Provides a cleaner syntax for the templated ClusterFinder in C++. + + Parameters + ---------- + image_size : tuple + The size of the image as a tuple (height, width). + cluster_size : tuple, optional + The size of the cluster to find as a tuple (height, width). Default is (3,3). + n_sigma : int, optional + Multiplier of the standard deviation used as a threshold to identify potential photon pixels. Default is 5. + dtype : data-type, optional + The data type of the image. Default is np.int32. + capacity : int, optional + The maximum number of clusters than can be stored before reallocating. Default is 1024. + min_pedestal_samples : int, optional + The minimum number of pedestal samples to accumulate before using the pedestal. Default is 1000. """ cls = _get_class("ClusterFinder", cluster_size, dtype) - return cls(image_size, n_sigma=n_sigma, capacity=capacity) + return cls(image_size, n_sigma=n_sigma, capacity=capacity, min_pedestal_samples=min_pedestal_samples) - -def ClusterFinderMT(image_size, cluster_size = (3,3), dtype=np.int32, n_sigma=5, capacity = 1024, n_threads = 3): +def ClusterFinderMT(image_size, *, cluster_size = (3,3), dtype=np.int32, n_sigma=5, capacity = 1024, n_threads = 3, min_pedestal_samples = 1000): """ Factory function to create a ClusterFinderMT object. Provides a cleaner syntax for the templated ClusterFinderMT in C++. + + Parameters + ---------- + image_size : tuple + The size of the image as a tuple (height, width). + cluster_size : tuple, optional + The size of the cluster to find as a tuple (height, width). Default is (3,3). + n_sigma : int, optional + Multiplier of the standard deviation used as a threshold to identify potential photon pixels. Default is 5. + dtype : data-type, optional + The data type of the image. Default is np.int32. + capacity : int, optional + The maximum number of clusters than can be stored before reallocating. Default is 1024. + n_threads : int, optional + The number of threads to use for processing. Default is 3. + min_pedestal_samples : int, optional + The minimum number of pedestal samples to accumulate before using the pedestal. Default is 1000. """ cls = _get_class("ClusterFinderMT", cluster_size, dtype) - return cls(image_size, n_sigma=n_sigma, capacity=capacity, n_threads=n_threads) + return cls(image_size, n_sigma=n_sigma, capacity=capacity, min_pedestal_samples=min_pedestal_samples, n_threads=n_threads) def ClusterCollector(clusterfindermt, dtype=np.int32): @@ -69,19 +89,55 @@ def ClusterFileSink(clusterfindermt, cluster_file, dtype=np.int32): def ClusterFile(fname, cluster_size=(3,3), dtype=np.int32, chunk_size = 1000, mode = "r"): - """ - Factory function to create a ClusterFile object. Provides a cleaner syntax for - the templated ClusterFile in C++. + """Create a reader or writer for a binary cluster file. + + Parameters + ---------- + fname : path-like + Cluster file to open. + cluster_size : tuple[int, int], default=(3, 3) + Cluster dimensions stored in the file. + dtype : numpy dtype, default=numpy.int32 + Data type of the cluster values stored in the file. + chunk_size : int, default=1000 + Maximum number of selected clusters returned by ``chunks()`` and + default iteration. Must be positive when iterating over chunks. + mode : {"r", "w", "a"}, default="r" + Open for reading, truncate and write, or append, respectively. + + Returns + ------- + ClusterFile + The compiled ClusterFile specialization matching ``cluster_size`` and + ``dtype``. + + Notes + ----- + The file format contains no cluster shape or data-type metadata. Supplying + values that do not match the file causes its bytes to be interpreted + incorrectly. Use ``frames()`` to iterate over complete frames, including + empty or fully filtered frames with their stored frame numbers. Use + ``chunks()`` or ``chunks(chunk_size)`` to iterate over selected clusters + in batches. Chunks may combine frames, so their frame number is not + reliable per-cluster metadata. + + Iterators consume the current file position without rewinding; use one + traversal at a time. Each result owns its storage and remains valid after + advancing the iterator or closing the file. + + Examples + -------- .. code-block:: python from aare import ClusterFile - - with ClusterFile("clusters.clust", cluster_size=(3,3), dtype=np.int32) as cf: - # cf is now a ClusterFile_Cluster3x3i object but you don't need to know that. - for clusters in cf: - # Loop over clusters in chunks of 1000 - # The type of clusters will be a ClusterVector_Cluster3x3i in this case + + with ClusterFile( + "clusters.clust", cluster_size=(3, 3), dtype=np.int32 + ) as cf: + for clusters in cf.chunks(): + # Process clusters in chunks of at most 1000. + ... """ diff --git a/python/aare/ClusterVector.py b/python/aare/ClusterVector.py index a294bb55..6c4a65f9 100644 --- a/python/aare/ClusterVector.py +++ b/python/aare/ClusterVector.py @@ -7,16 +7,33 @@ from .ClusterFinder import _get_class def ClusterVector(cluster_size=(3,3), dtype = np.int32): """ - Factory function to create a ClusterVector object. Provides a cleaner syntax for - the templated ClusterVector in C++. + Create an empty ClusterVector for a supported cluster size and pixel dtype. - .. code-block:: python + Parameters + ---------- + cluster_size : tuple[int, int], default=(3, 3) + Cluster dimensions in the x and y directions. + dtype : numpy.dtype, default=numpy.int32 + Pixel storage type. The cluster size and dtype combination must have a + compiled binding. - from aare import ClusterVector - - ClusterVector(cluster_size=(3,3), dtype=np.float64) + Returns + ------- + ClusterVector + An empty vector with frame number 0 and space reserved for at least + 1024 clusters. + + Raises + ------ + ValueError + If the requested size and dtype combination is unavailable. + + Examples + -------- + >>> import numpy as np + >>> from aare import ClusterVector + >>> clusters = ClusterVector(cluster_size=(3, 3), dtype=np.float64) """ cls = _get_class("ClusterVector", cluster_size, dtype) return cls() - diff --git a/python/aare/FastPedestal.py b/python/aare/FastPedestal.py new file mode 100644 index 00000000..3e28a205 --- /dev/null +++ b/python/aare/FastPedestal.py @@ -0,0 +1,56 @@ +# SPDX-License-Identifier: MPL-2.0 + +import numpy as np + +from .factory import _get_typed_class + + +def _get_fast_pedestal_class(dtype): + return _get_typed_class("FastPedestal", dtype) + + +def FastPedestal(rows, cols, n_samples=1000, dtype=np.float64): + """Create an empty per-pixel running pedestal. + + This factory hides the dtype suffix used by the templated C++ bindings. + Call ``add_init_frame()`` exactly ``n_samples`` times before using the + statistics or calling ``push_ema()``. Subsequent frames have weight + ``1 / n_samples`` in the running mean and population variance. + + Args: + rows: Number of image rows. + cols: Number of image columns. + n_samples: Initialization frame count and steady-state update-weight + denominator. + dtype: Output dtype for the mean and standard deviation. + Supported values are ``np.float64``, ``np.float32``, and + ``np.int16``. + """ + cls = _get_fast_pedestal_class(dtype) + return cls(rows, cols, n_samples) + + +def from_file(filename, n_samples=1000, skip_first=0, dtype=np.float64): + """Create a FastPedestal from frames in a file. + + After ignoring ``skip_first`` frames, the next ``n_samples`` frames + initialize the pedestal. Every remaining frame is then applied as a + steady-state update. Input frames are read as uint16 data. + + Args: + filename: Input image file. + n_samples: Number of frames used for initialization. + skip_first: Number of leading frames to ignore. + dtype: Output dtype for the mean and standard deviation. + + Raises: + RuntimeError: If fewer than ``n_samples`` frames remain after + ``skip_first`` or ``n_samples`` is zero. + """ + cls = _get_fast_pedestal_class(dtype) + return cls.from_file( + filename, n_samples=n_samples, skip_first=skip_first + ) + + +FastPedestal.from_file = from_file diff --git a/python/aare/Pedestal.py b/python/aare/Pedestal.py new file mode 100644 index 00000000..d89d6012 --- /dev/null +++ b/python/aare/Pedestal.py @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: MPL-2.0 + +import numpy as np + +from .factory import _get_typed_class + + +def Pedestal(rows, cols, n_samples=1000, dtype=np.float64): + """Create an empty per-pixel running pedestal. + + This factory hides the dtype suffix used by the templated C++ bindings. + Call ``push()`` to update the statistics and cached mean for each frame. + Statistics are available during initialization and are zero for empty pixels. + Internal moments and variance always use double precision. + + Args: + rows: Number of image rows. + cols: Number of image columns. + n_samples: Number of samples accumulated before switching to + steady-state updates with weight ``1 / n_samples``. + dtype: Output dtype for the mean and standard deviation. + Supported values are ``np.float64``, ``np.float32``, and + ``np.int16``. + """ + cls = _get_typed_class("Pedestal", dtype) + return cls(rows, cols, n_samples) diff --git a/python/aare/RawFile.py b/python/aare/RawFile.py index 988ff3d5..253fcc53 100644 --- a/python/aare/RawFile.py +++ b/python/aare/RawFile.py @@ -43,7 +43,7 @@ class RawFile(_aare.RawFile): Returns: int: Number of frames in file. """ - return super().frames_in_file + return self.total_frames def __enter__(self): return self diff --git a/python/aare/__init__.py b/python/aare/__init__.py index 84224613..570176f9 100644 --- a/python/aare/__init__.py +++ b/python/aare/__init__.py @@ -5,6 +5,16 @@ from . import _aare from . import transform from . import experimental +from ._aare import ( + FastPedestal_d, + FastPedestal_f, + FastPedestal_i16, + Pedestal_d, + Pedestal_f, + Pedestal_i16, + ClusterFinder_Cluster3x3i, + VarClusterFinder, +) from ._aare import ( File, JungfrauDataFile, @@ -12,20 +22,25 @@ from ._aare import ( RawSubFile, ) from ._aare import Pedestal_d, Pedestal_f, ClusterFinder_Cluster3x3i, VarClusterFinder -from ._aare import DetectorType, ReadoutMode +from ._aare import DetectorType, FrameDiscardPolicy, ReadoutMode from ._aare import hitmap from ._aare import ROI from ._aare import corner +from ._aare import UDPPortPosition + # from ._aare import ClusterFinderMT, ClusterCollector, ClusterFileSink, ClusterVector_i from ._version import __version__ +from .FastPedestal import FastPedestal +from .Pedestal import Pedestal from .ClusterFinder import ClusterFinder, ClusterCollector, ClusterFinderMT, ClusterFileSink, ClusterFile from .ClusterVector import ClusterVector from .Cluster import Cluster from ._aare import Gaussian, RisingScurve, FallingScurve, Pol1, Pol2, GaussianErfcPlateau, GaussianChargeSharing, GaussianChargeSharingKb from ._aare import fit +from ._aare import Minimizer from ._aare import Interpolator from ._aare import calculate_eta2, calculate_eta3, calculate_cross_eta3, calculate_full_eta2 from ._aare import reduce_to_2x2, reduce_to_3x3 diff --git a/python/aare/_version.py b/python/aare/_version.py index f3530c4c..c2bab73f 100644 --- a/python/aare/_version.py +++ b/python/aare/_version.py @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 # This file is used to get the version of the package from the VERSION file from importlib.metadata import PackageNotFoundError, version from pathlib import Path diff --git a/python/aare/calibration.py b/python/aare/calibration.py index b7a23e74..20f8c77b 100644 --- a/python/aare/calibration.py +++ b/python/aare/calibration.py @@ -5,9 +5,12 @@ def load_calibration(fname, hg0=False): """ Load calibration data from a file. - Parameters: - fname (str): Path to the calibration file. - hg0 (bool): If True, load HG0 calibration data instead of G0. + Parameters + ---------- + fname : str + Path to the calibration file. + hg0 : bool + If True, load HG0 calibration data instead of G0. """ gains = 3 diff --git a/python/aare/factory.py b/python/aare/factory.py new file mode 100644 index 00000000..4da3fdbe --- /dev/null +++ b/python/aare/factory.py @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MPL-2.0 + +import numpy as np + +from . import _aare + + +_TYPE_TO_CHAR = { + np.dtype(np.int32): "i", + np.dtype(np.float32): "f", + np.dtype(np.float64): "d", + np.dtype(np.int16): "i16", +} + + +def _type_to_char(dtype): + """Return the suffix used by bindings instantiated for ``dtype``.""" + try: + return _TYPE_TO_CHAR[np.dtype(dtype)] + except (KeyError, TypeError): + supported = ", ".join(str(dtype) for dtype in _TYPE_TO_CHAR) + raise ValueError( + f"Unsupported dtype: {dtype}. Supported dtypes are {supported}." + ) from None + + +def _get_typed_class(name, dtype): + """Return a bound class named ``_``.""" + class_name = f"{name}_{_type_to_char(dtype)}" + try: + return getattr(_aare, class_name) + except AttributeError: + raise ValueError( + f"Unsupported dtype for {name}: {dtype} " + f"(binding {class_name} is not available)." + ) from None diff --git a/python/src/bind_Cluster.hpp b/python/src/bind_Cluster.hpp index eca0881b..afa9bfd1 100644 --- a/python/src/bind_Cluster.hpp +++ b/python/src/bind_Cluster.hpp @@ -10,7 +10,6 @@ #include namespace py = pybind11; -using pd_type = double; using namespace aare; @@ -25,20 +24,26 @@ void define_Cluster(py::module &m, const std::string &typestr) { py::class_>( m, class_name.c_str(), py::buffer_protocol()) + // conversion ok: small input, accept lists and any numeric dtype .def(py::init([](CoordType x, CoordType y, py::array_t data) { - py::buffer_info buf_info = data.request(); + constexpr py::ssize_t n_pixels = ClusterSizeX * ClusterSizeY; + if (data.ndim() != 1 || data.size() != n_pixels) { + throw py::value_error(fmt::format( + "Cluster data must be a 1D array of {} values, got {}D " + "array with {} values", + n_pixels, data.ndim(), data.size())); + } Cluster cluster; cluster.x = x; cluster.y = y; auto r = data.template unchecked<1>(); // no bounds checks - for (py::ssize_t i = 0; i < data.size(); ++i) { + for (py::ssize_t i = 0; i < n_pixels; ++i) { cluster.data[i] = r(i); } return cluster; })) - // TODO! Review if to keep or not .def_property_readonly( "data", [](Cluster &c) @@ -61,6 +66,26 @@ void define_Cluster(py::module &m, const std::string &typestr) { .def_readonly("y", &Cluster::y) + .def("__repr__", + [class_name](const Cluster &self) { + fmt::memory_buffer data; + fmt::format_to(std::back_inserter(data), "[["); + for (size_t i = 0; i < self.data.size(); ++i) { + if (i != 0) { + fmt::format_to(std::back_inserter(data), + i % ClusterSizeX == 0 ? "], [" : ", "); + } + fmt::format_to(std::back_inserter(data), "{}", + self.data[i]); + } + fmt::format_to(std::back_inserter(data), "]]"); + + return fmt::format("{}(x={}, y={}, data={})", class_name, + self.x, self.y, + fmt::string_view(data.data(), data.size())); + }) + .def( "max_sum_2x2", [](Cluster &self) { @@ -68,8 +93,16 @@ void define_Cluster(py::module &m, const std::string &typestr) { return py::make_tuple(max_sum.sum, static_cast(max_sum.index)); }, - R"(calculates sum of 2x2 subcluster with highest energy and index relative to cluster center 0: top_left, 1: top_right, 2: bottom_left, 3: bottom_right - )"); + R"doc( + Return the highest-sum center-adjacent 2x2 subcluster. + + Returns + ------- + tuple + ``(sum, index)``, where index is 0 for top-left, 1 for + top-right, 2 for bottom-left, or 3 for bottom-right relative to + the cluster center. + )doc"); } template &cl) { return reduce_to_3x3(cl); }, - py::return_value_policy::move, R"(Reduce cluster to 3x3 subcluster)"); + py::return_value_policy::move, py::arg("cluster"), R"doc( + Return the 3x3 block around the cluster's center index. + + Both input dimensions must be at least 3, and at least one must be + greater than 3. + The input coordinates are preserved and output data is stored in + row-major order. + )doc"); } template &cl) { return reduce_to_2x2(cl); }, - py::return_value_policy::move, - R"( - Reduce cluster to 2x2 subcluster by taking the 2x2 subcluster with - the highest photon energy. + py::return_value_policy::move, py::arg("cluster"), R"doc( + Return the highest-sum center-adjacent 2x2 block. - RETURN: - - reduced cluster (cluster is filled in row major ordering starting at the top left. Thus for a max subcluster in the top left corner the photon hit is at the fourth position.) - - )"); + The input coordinates are preserved and output data is stored in + row-major order. + )doc"); } -#pragma GCC diagnostic pop \ No newline at end of file +#pragma GCC diagnostic pop diff --git a/python/src/bind_ClusterCollector.hpp b/python/src/bind_ClusterCollector.hpp index 2e621eda..01dd03c2 100644 --- a/python/src/bind_ClusterCollector.hpp +++ b/python/src/bind_ClusterCollector.hpp @@ -6,6 +6,8 @@ #include "aare/ClusterVector.hpp" #include "aare/NDView.hpp" #include "aare/Pedestal.hpp" + +#include "module_config.hpp" #include "np_helper.hpp" #include @@ -15,7 +17,6 @@ #include namespace py = pybind11; -using pd_type = double; using namespace aare; @@ -30,7 +31,7 @@ void define_ClusterCollector(py::module &m, const std::string &typestr) { using ClusterType = Cluster; py::class_>(m, class_name.c_str()) - .def(py::init *>()) + .def(py::init *>()) .def("stop", &ClusterCollector::stop) .def( "steal_clusters", diff --git a/python/src/bind_ClusterFile.hpp b/python/src/bind_ClusterFile.hpp index 0831d64f..c8544e9d 100644 --- a/python/src/bind_ClusterFile.hpp +++ b/python/src/bind_ClusterFile.hpp @@ -11,6 +11,7 @@ #include #include #include +#include // Disable warnings for unused parameters, as we ignore some // in the __exit__ method @@ -20,18 +21,90 @@ namespace py = pybind11; using namespace ::aare; +template class ClusterFileIterator { + Range m_range; + std::optional m_iterator; + bool m_done{false}; + + public: + explicit ClusterFileIterator(Range range) : m_range(std::move(range)) {} + + typename Range::Iterator::value_type next() { + if (m_done) { + throw py::stop_iteration(); + } + // Advance before returning the next result, never after yielding it. + if (m_iterator) { + ++*m_iterator; + } else { + m_iterator.emplace(m_range.begin()); + } + if (*m_iterator == m_range.end()) { + m_done = true; + throw py::stop_iteration(); + } + return std::move(**m_iterator); + } +}; + +template +void define_cluster_file_iterator(py::module &m, const std::string &name) { + py::class_(m, name.c_str()) + .def( + "__iter__", [](Iterator &self) -> Iterator & { return self; }, + py::return_value_policy::reference_internal) + .def("__next__", &Iterator::next); +} + template void define_ClusterFile(py::module &m, const std::string &typestr) { using ClusterType = Cluster; + using File = ClusterFile; + using FrameIterator = ClusterFileIterator; + using ChunkIterator = ClusterFileIterator; auto class_name = fmt::format("ClusterFile_{}", typestr); + define_cluster_file_iterator(m, + class_name + "_FrameIterator"); + define_cluster_file_iterator(m, + class_name + "_ChunkIterator"); - py::class_>(m, class_name.c_str()) + py::class_>( + m, class_name.c_str(), + "Read and write legacy binary cluster files. The format contains no " + "cluster type or shape metadata, so this class must match the file.") .def(py::init(), - py::arg(), py::arg("chunk_size") = 1000, py::arg("mode") = "r") + py::arg("fname"), py::arg("chunk_size") = 1000, + py::arg("mode") = "r", + "Open a cluster file. Mode must be 'r' to read, 'w' to truncate " + "and write, or 'a' to append.") + .def( + "frames", [](File &self) { return FrameIterator(self.frames()); }, + py::keep_alive<0, 1>(), + "Iterate over complete frames from the current position, including " + "empty and fully filtered frames with their stored frame numbers. " + "Results own their storage. The iterator keeps the file alive but " + "shares its cursor; use only one traversal at a time.") + .def( + "chunks", [](File &self) { return ChunkIterator(self.chunks()); }, + py::keep_alive<0, 1>(), + "Iterate from the current position using the constructor's chunk " + "size, which must be positive. Results own their storage. The " + "iterator keeps the file alive but shares its cursor; use only " + "one traversal at a time.") + .def( + "chunks", + [](File &self, size_t chunk_size) { + return ChunkIterator(self.chunks(chunk_size)); + }, + py::arg("chunk_size"), py::keep_alive<0, 1>(), + "Iterate over chunks of up to chunk_size selected clusters. The " + "size must be positive and does not change the constructor's " + "default. Chunks may span frames; their frame numbers are not " + "per-cluster metadata. Only the final chunk can be short.") .def( "read_clusters", [](ClusterFile &self, size_t n_clusters) { @@ -39,32 +112,62 @@ void define_ClusterFile(py::module &m, const std::string &typestr) { self.read_clusters(n_clusters)); return v; }, - py::return_value_policy::take_ownership, py::arg("n_clusters")) - .def("read_frame", - [](ClusterFile &self) { - auto v = new ClusterVector(self.read_frame()); - return v; - }) - .def("set_roi", &ClusterFile::set_roi, py::arg("roi")) - .def("tell", &ClusterFile::tell) - .def("estimate_n_clusters", - &ClusterFile::estimate_n_clusters) + py::return_value_policy::take_ownership, py::arg("n_clusters"), + "Read up to n_clusters without preserving frame boundaries. The " + "result may combine frames, so its frame number is not reliable " + "per-cluster metadata.") .def( - "set_noise_map", - [](ClusterFile &self, py::array_t noise_map) { + "read_frame", + [](ClusterFile &self) -> py::object { + auto clusters = self.read_frame(); + if (!clusters) { + return py::none(); + } + return py::cast( + new ClusterVector(std::move(*clusters)), + py::return_value_policy::take_ownership); + }, + "Read and return the next complete frame with its frame number, " + "or None at end of file.") + .def("set_roi", &ClusterFile::set_roi, py::arg("roi"), + "Select clusters whose centers lie within the half-open ROI.") + .def("tell", &ClusterFile::tell, + "Return the current byte position in the file.") + .def("estimate_n_clusters", + &ClusterFile::estimate_n_clusters, + "Estimate the number of clusters from the file size. Frame " + "headers can make this larger than the actual count.") + .def( + "set_noise_map", // conversions ok + [](ClusterFile &self, + py::array_t + noise_map) { auto view = make_view_2d(noise_map); self.set_noise_map(view); }, - py::arg("noise_map")) + py::arg("noise_map"), + "Set a two-dimensional, C-contiguous int32 noise map indexed as " + "[y, x]. The map must cover every cluster center coordinate.") - .def("set_gain_map", - [](ClusterFile &self, py::array_t gain_map) { - auto view = make_view_2d(gain_map); - self.set_gain_map(view); - }) + .def( + "set_gain_map", // conversions ok + [](ClusterFile &self, + py::array_t + gain_map) { + auto view = make_view_2d(gain_map); + self.set_gain_map(view); + }, + py::arg("gain_map"), + "Set a two-dimensional, C-contiguous float64 gain map in " + "ADU/energy, indexed as [y, x]. Clusters whose complete footprint " + "extends beyond the map are retained with all data values set to " + "zero.") - .def("close", &ClusterFile::close) - .def("write_frame", &ClusterFile::write_frame) + .def("close", &ClusterFile::close, + "Close the file. Calling close more than once is safe.") + .def("write_frame", &ClusterFile::write_frame, + py::arg("clusters"), + "Write one ClusterVector, including its frame number.") .def("__enter__", [](ClusterFile &self) { return &self; }) .def("__exit__", [](ClusterFile &self, @@ -74,14 +177,8 @@ void define_ClusterFile(py::module &m, const std::string &typestr) { self.close(); }) .def("__iter__", [](ClusterFile &self) { return &self; }) - .def("__next__", [](ClusterFile &self) { - auto v = new ClusterVector( - self.read_clusters(self.chunk_size())); - if (v->size() == 0) { - throw py::stop_iteration(); - } - return v; - }); + .def("__next__", + [](File &self) { return ChunkIterator(self.chunks()).next(); }); } #pragma GCC diagnostic pop diff --git a/python/src/bind_ClusterFileSink.hpp b/python/src/bind_ClusterFileSink.hpp index ade574d4..b366886a 100644 --- a/python/src/bind_ClusterFileSink.hpp +++ b/python/src/bind_ClusterFileSink.hpp @@ -15,8 +15,6 @@ #include namespace py = pybind11; -using pd_type = double; - using namespace aare; #pragma GCC diagnostic push @@ -29,6 +27,8 @@ void define_ClusterFileSink(py::module &m, const std::string &typestr) { using ClusterType = Cluster; + // TODO! adapt to set pedestal type (needs templating of ClusterFileSink) + // or maybe access through base class? py::class_>(m, class_name.c_str()) .def(py::init *, const std::filesystem::path &>()) diff --git a/python/src/bind_ClusterFinder.hpp b/python/src/bind_ClusterFinder.hpp index da479dc5..034b1b88 100644 --- a/python/src/bind_ClusterFinder.hpp +++ b/python/src/bind_ClusterFinder.hpp @@ -6,6 +6,8 @@ #include "aare/ClusterVector.hpp" #include "aare/NDView.hpp" #include "aare/Pedestal.hpp" + +#include "module_config.hpp" #include "np_helper.hpp" #include @@ -15,7 +17,6 @@ #include namespace py = pybind11; -using pd_type = double; using namespace aare; @@ -31,8 +32,10 @@ void define_ClusterFinder(py::module &m, const std::string &typestr) { py::class_>( m, class_name.c_str()) - .def(py::init, pd_type, size_t>(), py::arg("image_size"), - py::arg("n_sigma") = 5.0, py::arg("capacity") = 1'000'000) + .def(py::init, pd_type, size_t, size_t>(), + py::arg("image_size"), py::arg("n_sigma") = 5.0, + py::arg("capacity") = 1'000'000, + py::arg("min_pedestal_samples") = 1000) .def_property( "nSigma", @@ -40,14 +43,22 @@ void define_ClusterFinder(py::module &m, const std::string &typestr) { &ClusterFinder::set_nSigma, R"(number of sigma above the pedestal to consider a photon during cluster finding.)") - .def("push_pedestal_frame", - [](ClusterFinder &self, - py::array_t frame) { - auto view = make_view_2d(frame); - self.push_pedestal_frame(view); - }) + .def( + "push_pedestal_frame", + [](ClusterFinder &self, + py::array_t frame) { + auto view = make_view_2d(frame); + self.push_pedestal_frame(view); + }, + py::arg("frame").noconvert()) .def("clear_pedestal", &ClusterFinder::clear_pedestal) + .def("update_threshold", + &ClusterFinder::update_threshold) + .def_property_readonly( + "pedestal_ready", + &ClusterFinder::pedestal_ready, + R"(True once enough pedestal frames have been pushed to find clusters.)") .def_property_readonly( "pedestal", [](ClusterFinder &self) { @@ -79,7 +90,7 @@ void define_ClusterFinder(py::module &m, const std::string &typestr) { self.find_clusters(view, frame_number); return; }, - py::arg(), py::arg("frame_number") = 0); + py::arg("frame").noconvert(), py::arg("frame_number") = 0); } #pragma GCC diagnostic pop \ No newline at end of file diff --git a/python/src/bind_ClusterFinderMT.hpp b/python/src/bind_ClusterFinderMT.hpp index 4bb81012..fdd4b580 100644 --- a/python/src/bind_ClusterFinderMT.hpp +++ b/python/src/bind_ClusterFinderMT.hpp @@ -6,6 +6,8 @@ #include "aare/ClusterVector.hpp" #include "aare/NDView.hpp" #include "aare/Pedestal.hpp" + +#include "module_config.hpp" #include "np_helper.hpp" #include @@ -15,7 +17,6 @@ #include namespace py = pybind11; -using pd_type = double; using namespace aare; @@ -31,24 +32,27 @@ void define_ClusterFinderMT(py::module &m, const std::string &typestr) { py::class_>( m, class_name.c_str()) - .def(py::init, pd_type, size_t, size_t>(), + .def(py::init, pd_type, size_t, size_t, size_t, size_t>(), py::arg("image_size"), py::arg("n_sigma") = 5.0, - py::arg("capacity") = 2048, py::arg("n_threads") = 3) - .def("push_pedestal_frame", - [](ClusterFinderMT &self, - py::array_t frame) { - auto view = make_view_2d(frame); - self.push_pedestal_frame(view); - }) + py::arg("capacity") = 2048, py::arg("n_threads") = 3, + py::arg("queue_depth") = 16, + py::arg("min_pedestal_samples") = 1000) + .def( + "push_pedestal_frame", + [](ClusterFinderMT &self, + py::array_t frame) { + auto view = make_view_2d(frame); + self.push_pedestal_frame(view); + }, + py::arg("frame").noconvert()) .def( "find_clusters", [](ClusterFinderMT &self, py::array_t frame, uint64_t frame_number) { auto view = make_view_2d(frame); self.find_clusters(view, frame_number); - return; }, - py::arg(), py::arg("frame_number") = 0) + py::arg("frame").noconvert(), py::arg("frame_number") = 0) .def_property_readonly( "cluster_size", [](ClusterFinderMT &self) { @@ -56,6 +60,12 @@ void define_ClusterFinderMT(py::module &m, const std::string &typestr) { }) .def("clear_pedestal", &ClusterFinderMT::clear_pedestal) + .def("update_threshold", + &ClusterFinderMT::update_threshold) + .def( + "pedestal_ready", + &ClusterFinderMT::pedestal_ready, + R"(waits for the queued frames, then returns True if all cluster finders have enough pedestal frames to find clusters.)") .def("sync", &ClusterFinderMT::sync) .def("stop", &ClusterFinderMT::stop) .def("start", &ClusterFinderMT::start) diff --git a/python/src/bind_ClusterVector.hpp b/python/src/bind_ClusterVector.hpp index a3d59f57..9b031e5d 100644 --- a/python/src/bind_ClusterVector.hpp +++ b/python/src/bind_ClusterVector.hpp @@ -15,7 +15,6 @@ #include namespace py = pybind11; -using pd_type = double; using namespace aare; @@ -30,37 +29,72 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { py::class_, void>>( - m, class_name.c_str(), + m, class_name.c_str(), R"doc( + A contiguous, move-only container of fixed-size clusters. + + The class supports the Python buffer protocol, so ``numpy.array`` can + either copy its data or create a zero-copy view. A zero-copy view is + valid only while the ClusterVector's underlying allocation and size + remain unchanged. + )doc", py::buffer_protocol()) - .def(py::init()) // TODO change!!! + .def(py::init(), R"doc( + Create an empty ClusterVector with frame number 0 and space reserved + for at least 1024 clusters. + )doc") .def( "__call__", [](ClusterVector &self, py::array_t mask) { return self(make_view_1d(mask)); }, - py::arg("mask"), R"( - Create a copy of the clustervector and apply a boolean mask to the ClusterVector. + py::arg("mask").noconvert(), R"doc( + Return a filtered copy of this ClusterVector. Parameters ---------- + mask : numpy.ndarray + One-dimensional, writable, C-contiguous array with dtype + ``numpy.bool_`` and one element per cluster. - mask : 1d boolean numpy array - Mask to apply to the ClusterVector. Must be the same length as the number of clusters in the ClusterVector. + Returns + ------- + ClusterVector + Selected clusters in their original order. The frame number is + preserved. + )doc") - )") + .def( + "push_back", + [](ClusterVector &self, const ClusterType &cluster) { + self.push_back(cluster); + }, + py::arg("cluster"), R"doc( + Append one cluster. - .def("push_back", - [](ClusterVector &self, const ClusterType &cluster) { - self.push_back(cluster); - }) + Notes + ----- + Do not call this method while a zero-copy NumPy view of the + ClusterVector exists. Reallocation invalidates the view, while an + append without reallocation leaves its shape unchanged. + )doc") - .def("sum", - [](ClusterVector &self) { - auto *vec = new std::vector(self.sum()); - return return_vector(vec); - }) + .def( + "sum", + [](ClusterVector &self) { + auto *vec = new std::vector(self.sum()); + return return_vector(vec); + }, + R"doc( + Return the sum of all pixels in each cluster. + + Returns + ------- + numpy.ndarray + One value per cluster, in container order and with the cluster + pixel dtype. + )doc") .def( "sum_2x2", [](ClusterVector &self) { @@ -69,24 +103,45 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { return return_vector(vec); }, - R"(calculates sum of 2x2 subcluster with highest energy and index relative to cluster center 0: top_left, 1: top_right, 2: bottom_left, 3: bottom_right - )") - .def_property_readonly("size", &ClusterVector::size) - .def("empty", &ClusterVector::empty) - .def("item_size", &ClusterVector::item_size) - .def_property_readonly("fmt", - [typestr](ClusterVector &self) { - return fmt_format; - }) + R"doc( + Return the highest-sum center-adjacent 2x2 subcluster for each + cluster. + + Returns + ------- + numpy.ndarray + Structured array with ``sum`` and ``index`` fields. Indices are + 0 for top-left, 1 for top-right, 2 for bottom-left, and 3 for + bottom-right, relative to the cluster center. + )doc") + .def_property_readonly("size", &ClusterVector::size, + "Number of stored clusters.") + .def("empty", &ClusterVector::empty, + "Return True when no clusters are stored.") + .def("item_size", &ClusterVector::item_size, + "Return the size in bytes of one stored cluster, including " + "padding.") + .def_property_readonly( + "fmt", + [typestr](ClusterVector &self) { + return fmt_format; + }, + "PEP 3118 format string for one stored cluster.") .def_property_readonly("cluster_size_x", - &ClusterVector::cluster_size_x) + &ClusterVector::cluster_size_x, + "Cluster size in the x dimension.") .def_property_readonly("cluster_size_y", - &ClusterVector::cluster_size_y) + &ClusterVector::cluster_size_y, + "Cluster size in the y dimension.") .def_property_readonly("capacity", - &ClusterVector::capacity) + &ClusterVector::capacity, + "Number of clusters that fit without " + "reallocation.") .def_property("frame_number", &ClusterVector::frame_number, - &ClusterVector::set_frame_number) + &ClusterVector::set_frame_number, + "Signed 32-bit frame number; 0 can indicate clusters " + "from multiple frames.") .def_buffer( [typestr](ClusterVector &self) -> py::buffer_info { return py::buffer_info( @@ -100,31 +155,38 @@ void define_ClusterVector(py::module &m, const std::string &typestr) { }); // Free functions using ClusterVector - m.def("hitmap", - [](std::array image_size, ClusterVector &cv) { - // Create a numpy array to hold the hitmap - // The shape of the array is (image_size[0], image_size[1]) - // note that the python array is passed as [row, col] which - // is the opposite of the clusters [x,y] - py::array_t hitmap(image_size); - auto r = hitmap.mutable_unchecked<2>(); + m.def( + "hitmap", + [](std::array image_size, ClusterVector &cv) { + // Create a numpy array to hold the hitmap + // The shape of the array is (image_size[0], image_size[1]) + // note that the python array is passed as [row, col] which + // is the opposite of the clusters [x,y] + py::array_t hitmap(image_size); + auto r = hitmap.mutable_unchecked<2>(); - // Initialize hitmap to 0 - for (py::ssize_t i = 0; i < r.shape(0); i++) - for (py::ssize_t j = 0; j < r.shape(1); j++) - r(i, j) = 0; + // Initialize hitmap to 0 + for (py::ssize_t i = 0; i < r.shape(0); i++) + for (py::ssize_t j = 0; j < r.shape(1); j++) + r(i, j) = 0; - // Loop over the clusters and increment the hitmap - // Skip out of bound clusters - for (const auto &cluster : cv) { - auto x = cluster.x; - auto y = cluster.y; - if (x < image_size[1] && y < image_size[0]) - r(cluster.y, cluster.x) += 1; - } + // Loop over the clusters and increment the hitmap + // Skip out of bound clusters + for (const auto &cluster : cv) { + auto x = cluster.x; + auto y = cluster.y; + if (x < image_size[1] && y < image_size[0]) + r(cluster.y, cluster.x) += 1; + } - return hitmap; - }); + return hitmap; + }, + py::arg("image_size"), py::arg("clusters"), R"doc( + Count cluster centers into an ``int32`` image whose shape is given by + ``image_size`` as ``(rows, columns)``. Element ``[y, x]`` contains the + number of cluster centers at that coordinate. Out-of-bounds centers are + ignored. + )doc"); } template >( reduce_to_2x2(cv)); }, - R"( - Reduce cluster to 2x2 subcluster by taking the 2x2 subcluster with - the highest photon energy. - - Parameters - - cv : ClusterVector (clusters are filled in row-major ordering starting at the top left. Thus for a max subcluster in the top left corner the photon hit is at the fourth position.) - - )", + R"doc( + Reduce every cluster to its highest-sum center-adjacent 2x2 block. + Returns a new ClusterVector; cluster order, coordinates, and frame + number are preserved. Input pixel data is interpreted in row-major + order. + )doc", py::arg("clustervector")); } @@ -160,13 +219,13 @@ void define_3x3_reduction(py::module &m) { return new ClusterVector>( reduce_to_3x3(cv)); }, - R"( - Reduce cluster to 3x3 subcluster - - Parameters - - cv : ClusterVector - )", + R"doc( + Reduce every cluster to the 3x3 block around its center index. + Both input dimensions must be at least 3, and at least one must be + greater than 3. + Returns a new ClusterVector; cluster order, coordinates, and frame + number are preserved. + )doc", py::arg("clustervector")); } diff --git a/python/src/bind_Defs.hpp b/python/src/bind_Defs.hpp index 84a15cf3..9850f407 100644 --- a/python/src/bind_Defs.hpp +++ b/python/src/bind_Defs.hpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #include #include @@ -31,4 +32,76 @@ void define_defs_bindings(py::module &m) { auto jungfrau = py::class_(m, "Jungfrau"); jungfrau.attr("nRows") = Jungfrau::nRows; jungfrau.attr("nCols") = Jungfrau::nCols; + py::class_(m, "ROI") + .def(py::init<>()) + .def(py::init(), py::arg("xmin"), + py::arg("xmax"), py::arg("ymin"), py::arg("ymax")) + .def_readwrite("xmin", &ROI::xmin) + .def_readwrite("xmax", &ROI::xmax) + .def_readwrite("ymin", &ROI::ymin) + .def_readwrite("ymax", &ROI::ymax) + + .def( + "slice", + [](const ROI &self) { + return std::make_tuple(py::slice(self.ymin, self.ymax, 1), + py::slice(self.xmin, self.xmax, 1)); + }, + R"doc( + The slice can be used to index into a 2D array to extract the region of interest. + + Returns + ------- + tuple of slice + A tuple containing two slice objects for the x and y dimensions. + )doc") + + .def("__str__", + [](const ROI &self) { + return fmt::format("ROI: xmin: {} xmax: {} ymin: {} ymax: {}", + self.xmin, self.xmax, self.ymin, self.ymax); + }) + .def("__repr__", + [](const ROI &self) { + return fmt::format( + "", self.xmin, + self.xmax, self.ymin, self.ymax); + }) + .def("__iter__", + [](const ROI &self) { + return py::make_iterator(&self.xmin, &self.ymax + 1); // NOLINT + }) + + .def("__eq__", [](const ROI &self, const ROI &other) { + return self.xmin == other.xmin && self.xmax == other.xmax && + self.ymin == other.ymin && self.ymax == other.ymax; + }); + + py::enum_(m, "DetectorType") + .value("Jungfrau", DetectorType::Jungfrau) + .value("Eiger", DetectorType::Eiger) + .value("Mythen3", DetectorType::Mythen3) + .value("Moench", DetectorType::Moench) + .value("Moench03", DetectorType::Moench03) + .value("Moench03_old", DetectorType::Moench03_old) + .value("ChipTestBoard", DetectorType::ChipTestBoard) + .value("Unknown", DetectorType::Unknown); + + py::enum_(m, "UDPPortPosition") + .value("LEFT", UDPPortPosition::LEFT) + .value("RIGHT", UDPPortPosition::RIGHT) + .value("TOP", UDPPortPosition::TOP) + .value("BOTTOM", UDPPortPosition::BOTTOM); + + py::enum_(m, "TimingMode") + .value("AUTO_TIMING", TimingMode::AUTO_TIMING) + .value("TRIGGER_EXPOSURE", TimingMode::TRIGGER_EXPOSURE) + .value("GATED", TimingMode::GATED) + .value("BURST_TRIGGER", TimingMode::BURST_TRIGGER) + .value("TRIGGER_GATED", TimingMode::TRIGGER_GATED); + + py::enum_(m, "FrameDiscardPolicy") + .value("NoDiscard", FrameDiscardPolicy::NoDiscard) + .value("Discard", FrameDiscardPolicy::Discard) + .value("DiscardPartial", FrameDiscardPolicy::DiscardPartial); } diff --git a/python/src/bind_Eta.hpp b/python/src/bind_Eta.hpp index c27012f7..e2b9302e 100644 --- a/python/src/bind_Eta.hpp +++ b/python/src/bind_Eta.hpp @@ -1,6 +1,9 @@ +// SPDX-License-Identifier: MPL-2.0 #include "aare/CalculateEta.hpp" +#include "aare/to_string.hpp" #include +#include // #include only for version 3 #include @@ -18,7 +21,11 @@ void define_eta(py::module &m, const std::string &typestr) { .def_readwrite("c", &Eta2::c, "eta corner value cTopLeft, cTopRight, " "cBottomLeft, cBottomRight") - .def_readwrite("sum", &Eta2::sum, "photon energy of cluster"); + .def_readwrite("sum", &Eta2::sum, "photon energy of cluster") + .def("__repr__", [class_name](const Eta2 &self) { + return fmt::format("{}(x={}, y={}, c={}, sum={})", class_name, + self.x, self.y, to_string(self.c), self.sum); + }); } void define_corner_enum(py::module &m) { @@ -101,4 +108,4 @@ void register_calculate_3x3eta(py::module &m) { }, R"(calculates eta3x3 taking into account cross pixels in cluster)", py::arg("cluster")); -} \ No newline at end of file +} diff --git a/python/src/bind_FastPedestal.hpp b/python/src/bind_FastPedestal.hpp new file mode 100644 index 00000000..5c278490 --- /dev/null +++ b/python/src/bind_FastPedestal.hpp @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/FastPedestal.hpp" +#include "np_helper.hpp" + +#include +#include +#include +#include + +namespace py = pybind11; + +template +void define_fast_pedestal_bindings(py::module &m, const std::string &name) { + + py::class_>( + m, name.c_str(), + "Maintain a per-pixel running mean and population standard deviation.", + py::buffer_protocol()) + .def(py::init(), py::arg("rows"), + py::arg("cols"), py::arg("n_samples"), + "Construct an empty pedestal. It becomes ready after n_samples " + "calls to add_init_frame().") + .def(py::init(), py::arg("rows"), py::arg("cols"), + "Construct an empty pedestal with n_samples=1000.") + + .def( + "mean", + [](FastPedestal &self) { + auto mean = new NDArray{}; + *mean = self.mean(); + return return_image_data(mean); + }, + "Return a copy of the cached mean. The pedestal must be ready.") + .def( + "std", + [](FastPedestal &self) { + auto standard_deviation = new NDArray{}; + *standard_deviation = self.std(); + return return_image_data(standard_deviation); + }, + "Return the population standard deviation as a NumPy array. The " + "pedestal must be ready.") + .def( + "view", + [](py::object self_py) { + return py::module_::import("numpy").attr("asarray")(self_py); + }, + "Return a non-owning, non-writable NumPy view of the cached mean. " + "The pedestal must be ready.") + + // We need to buffer protocol to allow for numpy operations using the + // pedestal mean + .def_buffer([](FastPedestal &self) { + auto mean = self.view(); + return py::buffer_info( + const_cast(mean.data()), sizeof(SUM_TYPE), + py::format_descriptor::format(), 2, + {static_cast(mean.shape(0)), + static_cast(mean.shape(1))}, + {static_cast(mean.strides()[0] * sizeof(SUM_TYPE)), + static_cast(mean.strides()[1] * + sizeof(SUM_TYPE))}, + true); + }) + // Subtracting a FastPedestal from a NumPy array + .def( + "__array_ufunc__", + [](py::object self, py::object ufunc, const std::string &method, + py::args inputs, py::kwargs kwargs) -> py::object { + if (method != "__call__" || inputs.size() != 2 || + inputs[1].ptr() != self.ptr() || + py::cast(ufunc.attr("__name__")) != + "subtract") { + return py::reinterpret_borrow( + Py_NotImplemented); + } + + auto mean = + py::module_::import("builtins").attr("memoryview")(self); + return ufunc(inputs[0], mean, **kwargs); + }, + "Support subtracting a FastPedestal from a NumPy array.") + .def("clear", py::overload_cast<>(&FastPedestal::clear), + "Reset all statistics and initialization state to zero.") + .def_property_readonly("rows", &FastPedestal::rows, + "Number of image rows.") + .def_property_readonly("cols", &FastPedestal::cols, + "Number of image columns.") + .def_property_readonly("cur_samples", + &FastPedestal::cur_samples, + "Number of initialization frames accumulated. " + "Steady-state pushes do not change it.") + .def_property_readonly( + "ready", &FastPedestal::ready, + "Whether n_samples initialization frames have been accumulated.") + .def_property_readonly( + "n_samples", &FastPedestal::n_samples, + "Initialization frame count and steady-state update-weight " + "denominator.") + .def( + "clone", + [](FastPedestal &pedestal) { + return FastPedestal(pedestal); + }, + "Return an independent copy of the pedestal and its state.") + .def( + "push_ema", + [](FastPedestal &pedestal, py::array_t frame) { + pedestal.push_ema(make_view_2d(frame)); + }, + py::arg("frame").noconvert(), + "Update exponential moving average. The pedstal must " + "already be ready for this update.") + .def( + "add_init_frame", + [](FastPedestal &pedestal, py::array_t frame) { + pedestal.add_init_frame(make_view_2d(frame)); + }, + py::arg("frame").noconvert(), + "Accumulate one uint16 initialization frame. Call exactly " + "n_samples times to make the pedestal ready.") + .def_static( + "from_file", + [](const std::filesystem::path &filename, uint32_t n_samples, + uint32_t skip_first) { + return FastPedestal::template from_file( + filename, n_samples, skip_first); + }, + py::arg("filename"), py::arg("n_samples") = 1000, + py::arg("skip_first") = 0, + "Create a pedestal from a uint16 file. Skip skip_first frames, use " + "the next n_samples for initialization, then apply every remaining " + "frame as a steady-state update."); +} diff --git a/python/src/bind_Interpolator.hpp b/python/src/bind_Interpolator.hpp index a98e539d..3b7a015b 100644 --- a/python/src/bind_Interpolator.hpp +++ b/python/src/bind_Interpolator.hpp @@ -101,9 +101,7 @@ void define_interpolation_bindings(py::module &m) { auto interpolator = py::class_(m, "Interpolator") .def(py::init( - [](py::array_t - etacube, + [](py::array_t etacube, py::array_t xbins, py::array_t ybins, py::array_t ebins) { return Interpolator( @@ -124,9 +122,9 @@ void define_interpolation_bindings(py::module &m) { ebins: bin edges of photon energy )doc", - py::arg("etacube"), - py::arg("xbins"), py::arg("ybins"), - py::arg("ebins")) + py::arg("etacube").noconvert(), + py::arg("xbins").noconvert(), py::arg("ybins").noconvert(), + py::arg("ebins").noconvert()) .def(py::init( [](py::array_t xbins, py::array_t ybins, @@ -146,13 +144,12 @@ void define_interpolation_bindings(py::module &m) { bin edges of etay ebins: bin edges of photon energy - )", py::arg("xbins"), - py::arg("ybins"), py::arg("ebins")) + )", py::arg("xbins").noconvert(), + py::arg("ybins").noconvert(), py::arg("ebins").noconvert()) .def( "rosenblatttransform", [](Interpolator &self, - py::array_t + py::array_t etacube) { return self.rosenblatttransform(make_view_3d(etacube)); }, @@ -162,7 +159,7 @@ void define_interpolation_bindings(py::module &m) { etacube: joint distribution of eta_x, eta_y and photon energy (**Note:** for the joint distribution first dimension is eta_x, second: eta_y, third: energy bins.) )", - py::arg("etacube")) + py::arg("etacube").noconvert()) .def("get_ietax", [](Interpolator &self) { auto *ptr = new NDArray{}; diff --git a/python/src/bind_Pedestal.hpp b/python/src/bind_Pedestal.hpp new file mode 100644 index 00000000..9e8af455 --- /dev/null +++ b/python/src/bind_Pedestal.hpp @@ -0,0 +1,121 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/Pedestal.hpp" +#include "np_helper.hpp" + +#include +#include +#include +#include +#include + +namespace py = pybind11; + +template +void define_pedestal_bindings(py::module &m, const std::string &name) { + + py::class_>( + m, name.c_str(), + "Maintain a per-pixel running mean and population standard deviation. " + "Statistics are available during initialization and are zero for empty " + "pixels.", + py::buffer_protocol()) + .def(py::init(), py::arg("rows"), + py::arg("cols"), py::arg("n_samples"), + "Construct an empty pedestal. Each pixel accumulates n_samples " + "values before switching to exponential updates.") + .def(py::init(), py::arg("rows"), py::arg("cols"), + "Construct an empty pedestal with n_samples=1000.") + .def( + "mean", + [](Pedestal &self) { + auto mea = new NDArray{}; + *mea = self.mean(); + return return_image_data(mea); + }, + "Return a copy of the cached mean. Empty pixels return zero.") + .def( + "view", + [](py::object self_py) { + return py::module_::import("numpy").attr("asarray")(self_py); + }, + "Return a non-owning, non-writable NumPy view of the cached mean.") + .def( + "std", + [](Pedestal &self) { + auto std = new NDArray{}; + *std = self.std(); + return return_image_data(std); + }, + "Return the population standard deviation as a NumPy array. " + "Empty pixels return zero.") + .def( + "__array_ufunc__", + [](py::object self, py::object ufunc, const std::string &method, + py::args inputs, py::kwargs kwargs) -> py::object { + if (method != "__call__" || inputs.size() != 2 || + inputs[1].ptr() != self.ptr() || + py::cast(ufunc.attr("__name__")) != + "subtract") { + return py::reinterpret_borrow( + Py_NotImplemented); + } + + auto mean = + py::module_::import("builtins").attr("memoryview")(self); + return ufunc(inputs[0], mean, **kwargs); + }, + "Support subtracting a Pedestal from a NumPy array.") + .def("clear", py::overload_cast<>(&Pedestal::clear), + "Reset all statistics and per-pixel sample counts to zero.") + .def_property_readonly("rows", &Pedestal::rows, + "Number of image rows.") + .def_property_readonly("cols", &Pedestal::cols, + "Number of image columns.") + .def_property_readonly( + "n_samples", &Pedestal::n_samples, + "Initialization sample count per pixel and steady-state " + "update-weight denominator.") + .def( + "clone", + [&](Pedestal &pedestal) { + return Pedestal(pedestal); + }, + "Return an independent copy of the pedestal and its state.") + // TODO! add push for other data types + .def( + "push", + [](Pedestal &pedestal, py::array_t f) { + auto v = make_view_2d(f); + pedestal.push(v); + }, + py::arg("frame").noconvert(), + "Accumulate or exponentially update every pixel from a " + "C-contiguous uint16 frame matching the pedestal shape. After " + "n_samples values per pixel, new values have weight 1 / n_samples.") + .def( + "push_with_threshold", + [](Pedestal &pedestal, py::array_t f, + py::array_t threshold) { + auto frame_view = make_view_2d(f); + auto threshold_view = make_view_2d(threshold); + pedestal.push_with_threshold(frame_view, threshold_view); + }, + py::arg("frame").noconvert(), py::arg("threshold").noconvert(), + "Push only pixels where abs(frame - mean) is strictly less than " + "threshold. Both arrays must be C-contiguous with the pedestal " + "shape; frame must be uint16 and threshold must use the output " + "dtype. Rejected pixels keep their statistics and sample counts.") + .def_buffer([](Pedestal &self) { + auto mean = self.view(); + return py::buffer_info( + const_cast(mean.data()), sizeof(SUM_TYPE), + py::format_descriptor::format(), 2, + {static_cast(mean.shape(0)), + static_cast(mean.shape(1))}, + {static_cast(mean.strides()[0] * sizeof(SUM_TYPE)), + static_cast(mean.strides()[1] * + sizeof(SUM_TYPE))}, + true); + }); +} diff --git a/python/src/bind_PedestalTrackingPixelHistogram.hpp b/python/src/bind_PedestalTrackingPixelHistogram.hpp index eedcbf34..06b16147 100644 --- a/python/src/bind_PedestalTrackingPixelHistogram.hpp +++ b/python/src/bind_PedestalTrackingPixelHistogram.hpp @@ -68,8 +68,7 @@ void define_pedestal_tracking_pixel_histogram_bindings(py::module &m) { .def( "push_pedestal_no_update", [](PedestalTrackingPixelHistogram &self, - py::array_t - frame) { + py::array_t frame) { auto view = make_view_2d(frame); self.push_pedestal_no_update(view); }, @@ -146,8 +145,7 @@ void define_pedestal_tracking_pixel_histogram_bindings(py::module &m) { .def( "fill_async", [](PedestalTrackingPixelHistogram &self, - py::array_t - image) { + py::array_t image) { // Copy the numpy buffer into an owned NDArray while we // still hold the GIL so we don't depend on the array's // backing storage outliving this call. @@ -195,9 +193,13 @@ void define_pedestal_tracking_pixel_histogram_bindings(py::module &m) { Args: file_path: Path to the file to fill from max_frames: Maximum number of frames to fill from the file (default: -1) + reader_threads: Number of parallel file reader workers (default: 2) + reader_chunk_size: Frames claimed by each reader worker per batch (default: 4) )", py::call_guard(), py::arg("fname"), - py::arg("max_frames") = -1, py::arg("verbose") = false) + py::arg("max_frames") = -1, py::arg("verbose") = false, + py::arg("reader_threads") = std::size_t{2}, + py::arg("reader_chunk_size") = std::size_t{4}) .def("process_pedestal_file", &PedestalTrackingPixelHistogram::process_pedestal_file, R"( diff --git a/python/src/bind_PixelHistogram.hpp b/python/src/bind_PixelHistogram.hpp index d205bce2..d3a1b462 100644 --- a/python/src/bind_PixelHistogram.hpp +++ b/python/src/bind_PixelHistogram.hpp @@ -46,7 +46,7 @@ void define_pixel_histogram_binding(py::module &m, const char *class_name, .def( "fill_async", - [](Hist &self, py::array_t image) { + [](Hist &self, py::array_t image) { // Copy the numpy buffer into an owned NDArray while we // still hold the GIL so we don't depend on the array's // backing storage outliving this call. diff --git a/python/src/bind_RawFile.hpp b/python/src/bind_RawFile.hpp index 85ef155f..2c4e20c8 100644 --- a/python/src/bind_RawFile.hpp +++ b/python/src/bind_RawFile.hpp @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -21,14 +22,23 @@ namespace py = pybind11; using namespace ::aare; +namespace { +std::runtime_error raw_file_read_error(RawFile &file, + const std::string &message) { + return std::runtime_error(fmt::format( + "Error reading frame index {} from file '{}': {}", file.tell(), + file.master().master_fname().string(), message)); +} +} // namespace + void define_raw_file_io_bindings(py::module &m) { py::class_(m, "RawFile") .def(py::init()) .def("read_frame", [](RawFile &self) { if (self.n_modules_in_roi().size() > 1) { - throw std::runtime_error( - "File contains multiple ROIs - use read_ROIs()"); + throw raw_file_read_error( + self, "File contains multiple ROIs - use read_ROIs()"); } std::vector shape; @@ -51,16 +61,19 @@ void define_raw_file_io_bindings(py::module &m) { "read_n", [](RawFile &self, size_t n_frames) { if (self.n_modules_in_roi().size() > 1) { - throw std::runtime_error( + throw raw_file_read_error( + self, "File contains multiple ROIs - use read_n_ROIs() to " "read a specific ROI or use read_ROIs and " "read one frame at a time."); } // adjust for actual frames left in the file n_frames = - std::min(n_frames, self.total_frames() - self.tell()); + std::min(n_frames, self.tell() < self.total_frames() + ? self.total_frames() - self.tell() + : size_t{0}); if (n_frames == 0) { - throw std::runtime_error("No frames left in file"); + throw raw_file_read_error(self, "No frames left in file"); } std::vector shape{n_frames, self.rows(), self.cols()}; @@ -89,12 +102,13 @@ void define_raw_file_io_bindings(py::module &m) { "read_roi", [](RawFile &self, const size_t roi_index) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } if (roi_index >= self.num_rois()) { - throw std::runtime_error(LOCATION + - "ROI index out of range."); + throw raw_file_read_error( + self, LOCATION + "ROI index out of range."); } // return headers from all subfiles @@ -141,7 +155,8 @@ void define_raw_file_io_bindings(py::module &m) { "read_rois", [](RawFile &self) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } size_t number_of_ROIs = self.num_rois(); @@ -196,19 +211,22 @@ void define_raw_file_io_bindings(py::module &m) { "read_n_with_roi", [](RawFile &self, const size_t num_frames, const size_t roi_index) { if (self.num_rois() == 0) { - throw std::runtime_error(LOCATION + "No ROIs defined."); + throw raw_file_read_error(self, + LOCATION + "No ROIs defined."); } if (roi_index >= self.num_rois()) { - throw std::runtime_error(LOCATION + - "ROI index out of range."); + throw raw_file_read_error( + self, LOCATION + "ROI index out of range."); } // adjust for actual frames left in the file size_t n_frames = - std::min(num_frames, self.total_frames() - self.tell()); + std::min(num_frames, self.tell() < self.total_frames() + ? self.total_frames() - self.tell() + : size_t{0}); if (n_frames == 0) { - throw std::runtime_error("No frames left in file"); + throw raw_file_read_error(self, "No frames left in file"); } std::vector shape{ n_frames, self.roi_geometries(roi_index).pixels_y(), @@ -306,4 +324,4 @@ void define_raw_file_io_bindings(py::module &m) { .def_property_readonly("n_modules", &RawFile::n_modules) .def_property_readonly("n_modules_in_roi", &RawFile::n_modules_in_roi) .def_property_readonly("num_rois", &RawFile::num_rois); -} \ No newline at end of file +} diff --git a/python/src/bind_calibration.hpp b/python/src/bind_calibration.hpp index a4a3f160..6efa87bc 100644 --- a/python/src/bind_calibration.hpp +++ b/python/src/bind_calibration.hpp @@ -11,38 +11,34 @@ namespace py = pybind11; template -py::array_t pybind_apply_calibration( - py::array_t data, - py::array_t pedestal, - py::array_t - calibration, - int n_threads = 4) { +py::array_t +pybind_apply_calibration(py::array_t data, + py::array_t pedestal, + py::array_t calibration, int n_threads = 4) { auto data_span = make_view_3d(data); // data is always 3D /* No pointer is passed, so NumPy will allocate the buffer */ auto result = py::array_t(data_span.shape()); auto res = make_view_3d(result); - if (data.ndim() == 3 && pedestal.ndim() == 3 && calibration.ndim() == 3) { + if (pedestal.ndim() == 3 && calibration.ndim() == 3) { auto ped = make_view_3d(pedestal); auto cal = make_view_3d(calibration); aare::apply_calibration(res, data_span, ped, cal, n_threads); - } else if (data.ndim() == 3 && pedestal.ndim() == 2 && - calibration.ndim() == 2) { + } else if (pedestal.ndim() == 2 && calibration.ndim() == 2) { auto ped = make_view_2d(pedestal); auto cal = make_view_2d(calibration); aare::apply_calibration(res, data_span, ped, cal, n_threads); } else { throw std::runtime_error( - "Invalid number of dimensions for data, pedestal or calibration"); + "Invalid number of dimensions for pedestal or calibration"); } return result; } -py::array_t pybind_count_switching_pixels( - py::array_t data, - ssize_t n_threads = 4) { +py::array_t pybind_count_switching_pixels(py::array_t data, + ssize_t n_threads = 4) { auto data_span = make_view_3d(data); auto arr = new NDArray{}; @@ -51,9 +47,8 @@ py::array_t pybind_count_switching_pixels( } template -py::array_t pybind_calculate_pedestal( - py::array_t data, - ssize_t n_threads) { +py::array_t pybind_calculate_pedestal(py::array_t data, + ssize_t n_threads) { auto data_span = make_view_3d(data); auto arr = new NDArray{}; @@ -62,9 +57,8 @@ py::array_t pybind_calculate_pedestal( } template -py::array_t pybind_calculate_pedestal_g0( - py::array_t data, - ssize_t n_threads) { +py::array_t pybind_calculate_pedestal_g0(py::array_t data, + ssize_t n_threads) { auto data_span = make_view_3d(data); auto arr = new NDArray{}; @@ -73,10 +67,27 @@ py::array_t pybind_calculate_pedestal_g0( } void bind_calibration(py::module &m) { + py::options options; + options.disable_function_signatures(); + m.def("apply_calibration", &pybind_apply_calibration, py::arg("raw_data").noconvert(), py::kw_only(), py::arg("pd").noconvert(), py::arg("cal").noconvert(), - py::arg("n_threads") = 4); + py::arg("n_threads") = 4, + R"( + Apply the calibration to the raw data and return the result as a 3D array of doubles. + + Parameters + ---------- + raw_data : array_like + 3D array of shape ``(frames, rows, cols)``. + pd : array_like + Pedestal array. + cal : array_like + Calibration array. + n_threads : int, optional + Number of threads to use. Defaults to 4. + )"); m.def("apply_calibration", &pybind_apply_calibration, py::arg("raw_data").noconvert(), py::kw_only(), diff --git a/python/src/bind_testing.hpp b/python/src/bind_testing.hpp new file mode 100644 index 00000000..2a4e8760 --- /dev/null +++ b/python/src/bind_testing.hpp @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: MPL-2.0 +// Probes used by the python test-suite to exercise the helpers in +// np_helper.hpp. Internal, no API guarantees. +#pragma once + +#include "aare/NDArray.hpp" +#include "aare/NDView.hpp" +#include "np_helper.hpp" + +#include +#include +#include +#include +#include +#include + +namespace py = pybind11; +using namespace ::aare; + +// Arrays are taken with the default flags (no c_style) so that nothing is +// made contiguous by pybind11 before it reaches make_view +template +void define_make_view_probes(py::module &m, const std::string &suffix) { + + // Copy what C++ sees through the view back to python + m.def(("view_roundtrip_" + suffix).c_str(), + [](py::array_t arr) { + auto view = make_view(arr); + return return_image_data(new NDArray(view)); + }, + py::arg("arr").noconvert()); + + // Same as above but through a read only view + m.def(("const_view_roundtrip_" + suffix).c_str(), + [](py::array_t arr) { + auto view = make_const_view(arr); + static_assert( + std::is_same_v>); + auto out = new NDArray(view.shape()); + std::copy(view.begin(), view.end(), out->begin()); + return return_image_data(out); + }, + py::arg("arr").noconvert()); + + // Write through the view + m.def(("view_fill_" + suffix).c_str(), + [](py::array_t arr, T value) { + auto view = make_view(arr); + view = value; + }, + py::arg("arr").noconvert(), py::arg("value")); +} + +void define_testing_bindings(py::module &m) { + auto testing = m.def_submodule( + "testing", "Internal helpers for the test-suite, no API guarantees"); + + // noconvert + overloads gives dispatch on dtype + define_make_view_probes(testing, "1d"); + define_make_view_probes(testing, "2d"); + define_make_view_probes(testing, "3d"); + define_make_view_probes(testing, "1d"); + define_make_view_probes(testing, "2d"); + define_make_view_probes(testing, "3d"); +} diff --git a/python/src/ctb_raw_file.hpp b/python/src/ctb_raw_file.hpp index 5a6953b7..b6338bdd 100644 --- a/python/src/ctb_raw_file.hpp +++ b/python/src/ctb_raw_file.hpp @@ -24,177 +24,119 @@ namespace py = pybind11; using namespace ::aare; +// One binding for the three ADC SAR decoders. Each is registered under +// its original Python name so existing transforms keep working. +template , NDView)> +py::array_t decode64to16(py::array_t input) { + auto bytes = make_const_view_2d(input); + constexpr ssize_t bytes_per_word = sizeof(uint64_t); + auto output = py::array_t( + {bytes.shape(0), bytes.shape(1) / bytes_per_word}); + decode(bytes, make_view_2d(output)); + return output; +} + +constexpr auto decode64to16_doc = + "Decode packed 64-bit ADC samples. Takes a two-dimensional, " + "C-contiguous uint8 array whose row length is a multiple of 8 bytes " + "and returns a uint16 array with one value per 64-bit word."; + void define_ctb_raw_file_io_bindings(py::module &m) { - m.def("adc_sar_05_06_07_08decode64to16", [](py::array_t input) { - if (input.ndim() != 2) { - throw std::runtime_error( - "Only 2D arrays are supported at this moment"); - } + m.def("adc_sar_05_06_07_08decode64to16", + &decode64to16, + py::arg("input").noconvert(), decode64to16_doc); - // Create a 2D output array with the same shape as the input - std::vector shape{input.shape(0), - input.shape(1) / - static_cast(bits_per_byte)}; - py::array_t output(shape); + m.def("adc_sar_05_decode64to16", &decode64to16, + py::arg("input").noconvert(), decode64to16_doc); - // Create a view of the input and output arrays - NDView input_view( - reinterpret_cast(input.mutable_data()), - {output.shape(0), output.shape(1)}); - NDView output_view(output.mutable_data(), - {output.shape(0), output.shape(1)}); + m.def("adc_sar_04_decode64to16", &decode64to16, + py::arg("input").noconvert(), decode64to16_doc); - adc_sar_05_06_07_08decode64to16(input_view, output_view); + m.def( + "apply_custom_weights", + [](py::array_t input, py::array_t weights) { + auto input_view = make_view_1d(input); + auto weights_view = make_view_1d(weights); + auto output = py::array_t(input_view.size()); + apply_custom_weights(input_view, make_view_1d(output), + weights_view); + return output; + }, + py::arg("input").noconvert(), py::arg("weights").noconvert(), + "Apply per-bit weights to every value of a one-dimensional, " + "C-contiguous uint16 array and return a float64 array."); - return output; - }); + m.def( + "expand24to32bit", + [](py::array_t input, uint32_t offset) { + constexpr ssize_t bytes_per_channel = 3; // 24 bit + auto input_view = make_const_view_1d(input); + auto output = + py::array_t(input_view.size() / bytes_per_channel); + aare::expand24to32bit(input_view, make_view_1d(output), + aare::BitOffset(offset)); + return output; + }, + py::arg("input").noconvert(), py::arg("offset"), + "Expand packed 24-bit values from a one-dimensional, C-contiguous " + "uint8 array into a uint32 array."); - m.def("adc_sar_05_decode64to16", [](py::array_t input) { - if (input.ndim() != 2) { - throw std::runtime_error( - "Only 2D arrays are supported at this moment"); - } + m.def( + "expand4to8bit", + [](py::array_t input) { + auto input_view = make_const_view_1d(input); + auto output = py::array_t(input_view.size() * 2); + aare::expand4to8bit(input_view, make_view_1d(output)); + return output; + }, + py::arg("input").noconvert(), + "Expand two 4-bit values per byte of a one-dimensional, C-contiguous " + "uint8 array into a uint8 array of twice the length."); - // Create a 2D output array with the same shape as the input - std::vector shape{input.shape(0), - input.shape(1) / - static_cast(bits_per_byte)}; - py::array_t output(shape); + m.def( + "decode_my302", + [](py::array_t input, uint32_t offset) { + // Physical layout of the chip + constexpr ssize_t channels = 64; + constexpr ssize_t counters = 3; + constexpr ssize_t bytes_per_channel = 3; // 24 bit + constexpr ssize_t n_outputs = 2; - // Create a view of the input and output arrays - NDView input_view( - reinterpret_cast(input.mutable_data()), - {output.shape(0), output.shape(1)}); - NDView output_view(output.mutable_data(), - {output.shape(0), output.shape(1)}); + auto input_view = make_const_view_1d(input); - adc_sar_05_decode64to16(input_view, output_view); + ssize_t expected_size = channels * counters * bytes_per_channel; - return output; - }); + // If we have an offset we need one extra byte per output + aare::BitOffset bitoff(offset); + if (bitoff.value()) + expected_size += n_outputs; - m.def("adc_sar_04_decode64to16", [](py::array_t input) { - if (input.ndim() != 2) { - throw std::runtime_error( - "Only 2D arrays are supported at this moment"); - } + if (input_view.size() != expected_size) { + throw py::value_error( + fmt::format("{} Expected an input size of {} bytes. Called " + "with input size of {}", + LOCATION, expected_size, input_view.size())); + } - // Create a 2D output array with the same shape as the input - std::vector shape{input.shape(0), - input.shape(1) / - static_cast(bits_per_byte)}; - py::array_t output(shape); + auto output = py::array_t(channels * counters); + auto output_view = make_view_1d(output); - // Create a view of the input and output arrays - NDView input_view( - reinterpret_cast(input.mutable_data()), - {output.shape(0), output.shape(1)}); - NDView output_view(output.mutable_data(), - {output.shape(0), output.shape(1)}); + const ssize_t in_step = input_view.size() / n_outputs; + const ssize_t out_step = output_view.size() / n_outputs; + for (ssize_t i = 0; i != n_outputs; ++i) { + NDView in_part( + input_view.data() + in_step * i, {in_step}); + NDView out_part(output_view.data() + out_step * i, + {out_step}); + aare::expand24to32bit(in_part, out_part, bitoff); + } - adc_sar_04_decode64to16(input_view, output_view); - - return output; - }); - - m.def("apply_custom_weights", - [](py::array_t - &input, - py::array_t - &weights) { - // Create new array with same shape as the input array - // (uninitialized values) - py::buffer_info buf = input.request(); - py::array_t output(buf.shape); - - // Use NDViews to call into the C++ library - auto weights_view = make_view_1d(weights); - NDView input_view(input.mutable_data(), - {input.size()}); - NDView output_view(output.mutable_data(), - {output.size()}); - - apply_custom_weights(input_view, output_view, weights_view); - return output; - }); - - m.def("expand24to32bit", - [](py::array_t - &input, - uint32_t offset) { - aare::BitOffset bitoff(offset); - py::buffer_info buf = input.request(); - - constexpr uint32_t bytes_per_channel = 3; // 24 bit - py::array_t output(buf.size / bytes_per_channel); - - NDView input_view(input.mutable_data(), - {input.size()}); - NDView output_view(output.mutable_data(), - {output.size()}); - - aare::expand24to32bit(input_view, output_view, bitoff); - return output; - }); - - m.def("expand4to8bit", - [](py::array_t - &input) { - py::buffer_info buf = input.request(); - - py::array_t output(buf.size * 2); - - NDView input_view(input.mutable_data(), - {input.size()}); - NDView output_view(output.mutable_data(), - {output.size()}); - - aare::expand4to8bit(input_view, output_view); - return output; - }); - - m.def("decode_my302", - [](py::array_t - &input, - uint32_t offset) { - // Physical layout of the chip - constexpr size_t channels = 64; - constexpr size_t counters = 3; - constexpr size_t bytes_per_channel = 3; // 24 bit - constexpr int n_outputs = 2; - - ssize_t expected_size = channels * counters * bytes_per_channel; - - // If whe have an offset we need one extra byte per output - aare::BitOffset bitoff(offset); - if (bitoff.value()) - expected_size += n_outputs; - - if (input.size() != expected_size) { - throw std::runtime_error(fmt::format( - "{} Expected an input size of {} bytes. Called " - "with input size of {}", - LOCATION, expected_size, input.size())); - } - - py::buffer_info buf = input.request(); - py::array_t output(channels * counters); - - for (int i = 0; i != n_outputs; ++i) { - auto step = input.size() / n_outputs; - auto out_step = output.size() / n_outputs; - NDView input_view(input.mutable_data() + step * i, - {input.size() / n_outputs}); - NDView output_view(output.mutable_data() + - out_step * i, - {output.size() / n_outputs}); - - aare::expand24to32bit(input_view, output_view, bitoff); - } - - return output; - }); + return output; + }, + py::arg("input").noconvert(), py::arg("offset"), + "Decode a Mythen 302 readout from a one-dimensional, C-contiguous " + "uint8 array into a uint32 array of 64 channels times 3 counters."); py::class_(m, "CtbRawFile") .def(py::init()) diff --git a/python/src/file.hpp b/python/src/file.hpp index ccb62d1f..a493c910 100644 --- a/python/src/file.hpp +++ b/python/src/file.hpp @@ -2,6 +2,7 @@ #include "aare/CtbRawFile.hpp" #include "aare/File.hpp" #include "aare/Frame.hpp" +#include "aare/ROI.hpp" #include "aare/RawFile.hpp" #include "aare/RawMasterFile.hpp" #include "aare/RawSubFile.hpp" @@ -27,16 +28,6 @@ using namespace ::aare; void define_file_io_bindings(py::module &m) { - py::enum_(m, "DetectorType") - .value("Jungfrau", DetectorType::Jungfrau) - .value("Eiger", DetectorType::Eiger) - .value("Mythen3", DetectorType::Mythen3) - .value("Moench", DetectorType::Moench) - .value("Moench03", DetectorType::Moench03) - .value("Moench03_old", DetectorType::Moench03_old) - .value("ChipTestBoard", DetectorType::ChipTestBoard) - .value("Unknown", DetectorType::Unknown); - PYBIND11_NUMPY_DTYPE(DetectorHeader, frameNumber, expLength, packetNumber, bunchId, timestamp, modId, row, column, reserved, debug, roundRNumber, detType, version, packetMask); @@ -170,28 +161,5 @@ void define_file_io_bindings(py::module &m) { .def_property_readonly("stop", &ScanParameters::stop) .def_property_readonly("step", &ScanParameters::step); - py::class_(m, "ROI") - .def(py::init<>()) - .def(py::init(), py::arg("xmin"), - py::arg("xmax"), py::arg("ymin"), py::arg("ymax")) - .def_readwrite("xmin", &ROI::xmin) - .def_readwrite("xmax", &ROI::xmax) - .def_readwrite("ymin", &ROI::ymin) - .def_readwrite("ymax", &ROI::ymax) - .def("__str__", - [](const ROI &self) { - return fmt::format("ROI: xmin: {} xmax: {} ymin: {} ymax: {}", - self.xmin, self.xmax, self.ymin, self.ymax); - }) - .def("__repr__", - [](const ROI &self) { - return fmt::format( - "", self.xmin, - self.xmax, self.ymin, self.ymax); - }) - .def("__iter__", [](const ROI &self) { - return py::make_iterator(&self.xmin, &self.ymax + 1); // NOLINT - }); - #pragma GCC diagnostic pop } \ No newline at end of file diff --git a/python/src/fit.hpp b/python/src/fit.hpp index 84f677b8..efe3024d 100644 --- a/python/src/fit.hpp +++ b/python/src/fit.hpp @@ -21,10 +21,43 @@ fit_dispatch(const aare::FitModel &model, template void bind_fit_model(py::module &m, const char *name) { using FM = aare::FitModel; + auto cls = py::class_(m, name); - cls.def(py::init(), + + cls.def(py::init(), py::arg("strategy") = 0, py::arg("max_calls") = 100, - py::arg("tolerance") = 0.5, py::arg("compute_errors") = false) + py::arg("tolerance") = 0.5, py::arg("compute_errors") = false, + py::arg("minimizer") = aare::Minimizer::Migrad, + R"doc( + Fit configuration for this model. + + Parameters + ---------- + strategy : int + Minuit2 strategy (0 = fast, 1 = Minuit2's default). Ignored + by ``Minimizer.LevenbergMarquardt`` and + ``Minimizer.VarPro``. + max_calls : int + Work budget per pixel: Minuit2 function calls, or model + evaluations for ``Minimizer.LevenbergMarquardt`` and + ``Minimizer.VarPro`` (one per iteration, two + when a step crosses a limit). 0 selects Minuit's default + budget. + tolerance : float + EDM tolerance in Minuit's convention. Migrad, + LevenbergMarquardt and VarPro stop below + ``0.002 * tolerance``, Fumili below ``1e-4 * tolerance``. + compute_errors : bool + Also return parameter errors: from Hesse with Migrad, from + the linearised covariance with Fumili, LevenbergMarquardt + and VarPro. Fixed parameters and parameters + ending on a limit report 0. + minimizer : Minimizer + ``Minimizer.Migrad`` (default), ``Minimizer.Fumili``, + ``Minimizer.LevenbergMarquardt`` or + ``Minimizer.VarPro``. + )doc") .def("SetParLimits", py::overload_cast(&FM::SetParLimits), py::arg("idx"), py::arg("lo"), py::arg("hi")) @@ -59,8 +92,10 @@ template void bind_fit_model(py::module &m, const char *name) { .def_property("tolerance", &FM::tolerance, &FM::SetTolerance) .def_property("compute_errors", &FM::compute_errors, &FM::SetComputeErrors) + .def_property("minimizer", &FM::minimizer, &FM::SetMinimizer) .def( - "__call__", + "__call__", // conversion ok, we want to be able to call with any + // dtype [](const FM & /*self*/, py::array_t x, py::array_t @@ -78,7 +113,7 @@ template void bind_fit_model(py::module &m, const char *name) { }, py::arg("x"), py::arg("par")) .def( - "fit", + "fit", // conversion ok [](const FM &self, py::array_t x, py::array_t y, @@ -88,6 +123,11 @@ template void bind_fit_model(py::module &m, const char *name) { R"doc( Fit this model to 1D or 3D data using Minuit2. + The minimizer is selected by the ``minimizer`` property + (``Minimizer.Migrad`` by default, ``Minimizer.Fumili``, + ``Minimizer.LevenbergMarquardt`` or + ``Minimizer.VarPro``). + Parameters ---------- x : array_like, shape (n_scan,) @@ -202,11 +242,6 @@ fit_dispatch(const aare::FitModel &model, py::array_t>( y_err_obj); - if (y_err.ndim() != 3) { - throw std::runtime_error( - "For 3D input y, y_err must also be 3D."); - } - auto err_out = new NDArray({y.shape(0), y.shape(1), npar}, 0.0); auto y_view_err = make_view_3d(y_err); @@ -228,8 +263,23 @@ fit_dispatch(const aare::FitModel &model, "chi2"_a = return_image_data(chi2_out)); } } else { - + // Unweighted fit; parameter errors are still available when + // requested, as in the 1D path. NDView dummy_err{}; + + if (model.compute_errors()) { + auto err_out = + new NDArray({y.shape(0), y.shape(1), npar}, 0.0); + + aare::fit_3d(model, x_view, y_view, dummy_err, + par_out->view(), err_out->view(), + chi2_out->view(), n_threads); + + return py::dict("par"_a = return_image_data(par_out), + "par_err"_a = return_image_data(err_out), + "chi2"_a = return_image_data(chi2_out)); + } + NDView dummy_err_out{}; { @@ -253,11 +303,6 @@ fit_dispatch(const aare::FitModel &model, py::array_t>( y_err_obj); - if (y_err.ndim() != 1) { - throw std::runtime_error( - "For 1D input y, y_err must also be 1D."); - } - auto y_view_err = make_view_1d(y_err); { py::gil_scoped_release release; // release GIL for parallel loop @@ -278,7 +323,82 @@ fit_dispatch(const aare::FitModel &model, } } +// Resolve the model type behind a Python object and run the fit. +py::object dispatch_any_model( + py::object model_obj, + py::array_t x, + py::array_t y, + py::object y_err_obj, int n_threads) { + using namespace aare::model; + +#define AARE_DISPATCH_MODEL(Model) \ + if (py::isinstance>(model_obj)) { \ + const auto &mdl = model_obj.cast &>(); \ + return fit_dispatch(mdl, x, y, y_err_obj, n_threads); \ + } + + AARE_DISPATCH_MODEL(Pol1) + AARE_DISPATCH_MODEL(Pol2) + AARE_DISPATCH_MODEL(Gaussian) + AARE_DISPATCH_MODEL(GaussianErfcPlateau) + AARE_DISPATCH_MODEL(GaussianChargeSharing) + AARE_DISPATCH_MODEL(GaussianChargeSharingKb) + AARE_DISPATCH_MODEL(RisingScurve) + AARE_DISPATCH_MODEL(FallingScurve) + +#undef AARE_DISPATCH_MODEL + + throw std::runtime_error( + "Unknown model type. Expected Pol1, Pol2, Gaussian, " + "GaussianErfcPlateau, GaussianChargeSharing, GaussianChargeSharingKb, " + "RisingScurve or FallingScurve."); +} + void define_fit_bindings(py::module &m) { + // The enum must exist before it is used as a constructor default below. + py::enum_(m, "Minimizer", R"doc( + Minimizer used by the fit models. + + ``Migrad`` (default) is Minuit2's variable-metric minimizer. With + ``compute_errors`` its parameter errors come from Hesse. + + ``Fumili`` is Minuit2's Gauss-Newton minimizer with a trust region. + It takes the gradient and a linearised Hessian from the analytic + derivatives of the model and usually needs far fewer function + evaluations than Migrad. Its parameter errors come from the + covariance of the linearised Hessian, without an extra Hesse step. + Minuit2's Fumili cannot converge once a two-sided limit becomes + active; a pixel without a valid Fumili minimum is refitted with + Migrad. + + ``LevenbergMarquardt`` is the built-in damped Gauss-Newton solver. It + uses the analytic Jacobian, reflects steps at parameter limits and + reuses its buffers between pixels, so data cubes are fitted without + per-pixel allocations. Its parameter errors are + ``sqrt(diag((J^T J)^-1))``; ``max_calls`` counts model evaluations + and ``strategy`` is ignored. + + ``VarPro`` is the built-in variable projection solver. + Every bundled model is linear in some of its parameters (the + amplitude of a Gaussian, the baseline and amplitude of an S-curve, + every parameter of a polynomial); each trial point solves those + exactly and the Levenberg-Marquardt iteration runs over the + remaining nonlinear parameters only. It needs fewer evaluations than + ``LevenbergMarquardt`` and cannot be misled by poor start values of + the linear parameters, which it ignores. A pixel whose linear + solution violates a limit on a linear parameter, or that does not + converge, is refitted with ``LevenbergMarquardt``. Its parameter + errors are the same Gauss-Newton estimates; ``max_calls`` counts + evaluations and ``strategy`` is ignored. + + With every minimizer, fixed parameters and parameters that end on a + limit report an error of 0 and a failed fit returns zeros. + )doc") + .value("Migrad", aare::Minimizer::Migrad) + .value("Fumili", aare::Minimizer::Fumili) + .value("LevenbergMarquardt", aare::Minimizer::LevenbergMarquardt) + .value("VarPro", aare::Minimizer::VarPro); + // ── Bind model classes ────────────────────────────────────────── bind_fit_model(m, "Gaussian"); bind_fit_model(m, "GaussianErfcPlateau"); @@ -293,91 +413,24 @@ void define_fit_bindings(py::module &m) { m.def( "fit", - [](py::object model_obj, + [](py::object model_obj, // conversion ok, we want to be able to call + // with any dtype py::array_t x, py::array_t y, py::object y_err_obj, int n_threads) -> py::object { - using namespace aare::model; - - // ── Polynomial of degree 1 ─────── - if (py::isinstance>(model_obj)) { - - const auto &mdl = - model_obj.cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, n_threads); - } - - // ── Polynomial of degree 2 ─────── - if (py::isinstance>(model_obj)) { - const auto &mdl = - model_obj.cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, n_threads); - } - - // ── Gaussian ─────── - if (py::isinstance>(model_obj)) { - const auto &mdl = - model_obj.cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, n_threads); - } - - // ── GaussianErfcPlateau ─────── - if (py::isinstance>( - model_obj)) { - const auto &mdl = - model_obj - .cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, - n_threads); - } - - // ── GaussianChargeSharing ─────── - if (py::isinstance>( - model_obj)) { - const auto &mdl = - model_obj - .cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, - n_threads); - } - - // ── GaussianChargeSharingKb ─────── - if (py::isinstance>( - model_obj)) { - const auto &mdl = model_obj.cast< - const aare::FitModel &>(); - return fit_dispatch( - mdl, x, y, y_err_obj, n_threads); - } - - // ── Rising Scurve ─────── - if (py::isinstance>(model_obj)) { - const auto &mdl = - model_obj.cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, - n_threads); - } - - // ── Falling Scurve ─────── - if (py::isinstance>(model_obj)) { - const auto &mdl = - model_obj.cast &>(); - return fit_dispatch(mdl, x, y, y_err_obj, - n_threads); - } - - throw std::runtime_error( - "Unknown model type. Expected Pol1, Pol2, Gaussian, " - "RisingScurve or FallingScurve."); + return dispatch_any_model(model_obj, x, y, y_err_obj, n_threads); }, R"( - Fit a model to 1D or 3D data using Minuit2. + Fit a model to 1D or 3D data with the minimizer selected by the + model's ``minimizer`` property. Parameters ---------- - model : Pol1, Pol2, Gaussian, RisingScurve, or FallingScurve - Configured model object. User-set limits, fixed parameters, - and start values take precedence over automatic estimates. + model : object + Configured model object: Pol1, Pol2, Gaussian, GaussianErfcPlateau, + GaussianChargeSharing, GaussianChargeSharingKb, RisingScurve or + FallingScurve. User-set limits, fixed parameters, and start values + take precedence over automatic estimates. x : array_like, shape (n_scan,) Scan points (e.g. energy or threshold values). y : array_like, shape (n_scan,) or (rows, cols, n_scan) @@ -389,15 +442,12 @@ void define_fit_bindings(py::module &m) { Returns ------- - For 1D input: - numpy array of shape (2*npar+1,) if compute_errors else (npar+1,). - Layout: [params..., (errors...,) chi2]. - - For 3D input: - dict with keys: - "par" : (rows, cols, npar) fitted parameters. - "par_err" : (rows, cols, npar) parameter errors (if compute_errors). - "chi2" : (rows, cols) chi-squared per pixel. + dict + ``par`` holds the fitted parameters, shape (npar,) for 1D input or + (rows, cols, npar) for 3D input. ``chi2`` holds the chi-squared, + shape (1,) or (rows, cols). ``par_err`` holds the parameter + errors when ``compute_errors`` is enabled; fixed parameters and + parameters on a limit report 0. A failed fit returns zeros. )", py::arg("model"), py::arg("x"), py::arg("y"), py::arg("y_err") = py::none(), py::arg("n_threads") = 4); diff --git a/python/src/module.cpp b/python/src/module.cpp index 52c830b9..b83deaca 100644 --- a/python/src/module.cpp +++ b/python/src/module.cpp @@ -1,6 +1,8 @@ // SPDX-License-Identifier: MPL-2.0 // Files with bindings to the different classes +#include "module_config.hpp" + // New style file naming #include "bind_Cluster.hpp" #include "bind_ClusterCollector.hpp" @@ -11,25 +13,28 @@ #include "bind_ClusterVector.hpp" #include "bind_Defs.hpp" #include "bind_Eta.hpp" +#include "bind_FastPedestal.hpp" #include "bind_Interpolator.hpp" #include "bind_MultiThreadedFileReader.hpp" +#include "bind_Pedestal.hpp" #include "bind_PedestalTrackingPixelHistogram.hpp" #include "bind_PixelHistogram.hpp" #include "bind_PixelMap.hpp" #include "bind_RawFile.hpp" #include "bind_calibration.hpp" +#include "bind_testing.hpp" // TODO! migrate the other names #include "ctb_raw_file.hpp" #include "file.hpp" #include "fit.hpp" #include "jungfrau_data_file.hpp" -#include "pedestal.hpp" #include "raw_master_file.hpp" #include "raw_sub_file.hpp" #include "var_cluster.hpp" // Pybind stuff +#include #include #include @@ -75,11 +80,16 @@ PYBIND11_MODULE(_aare, m) { define_pedestal_tracking_pixel_histogram_bindings(m); define_pedestal_bindings(m, "Pedestal_d"); define_pedestal_bindings(m, "Pedestal_f"); + define_pedestal_bindings(m, "Pedestal_i16"); + define_fast_pedestal_bindings(m, "FastPedestal_d"); + define_fast_pedestal_bindings(m, "FastPedestal_f"); + define_fast_pedestal_bindings(m, "FastPedestal_i16"); define_fit_bindings(m); define_interpolation_bindings(m); define_jungfrau_data_file_io_bindings(m); bind_calibration(m); + define_testing_bindings(m); DEFINE_CLUSTER_BINDINGS(int, 3, 3, uint16_t, i); DEFINE_CLUSTER_BINDINGS(double, 3, 3, uint16_t, d); @@ -106,6 +116,7 @@ PYBIND11_MODULE(_aare, m) { DEFINE_BINDINGS_CLUSTERFINDER(int, 3, 3, uint16_t, i); DEFINE_BINDINGS_CLUSTERFINDER(double, 3, 3, uint16_t, d); DEFINE_BINDINGS_CLUSTERFINDER(float, 3, 3, uint16_t, f); + DEFINE_BINDINGS_CLUSTERFINDER(int16_t, 3, 3, uint16_t, i16); DEFINE_BINDINGS_CLUSTERFINDER(int, 5, 5, uint16_t, i); DEFINE_BINDINGS_CLUSTERFINDER(double, 5, 5, uint16_t, d); @@ -119,9 +130,6 @@ PYBIND11_MODULE(_aare, m) { DEFINE_BINDINGS_CLUSTERFINDER(double, 9, 9, uint16_t, d); DEFINE_BINDINGS_CLUSTERFINDER(float, 9, 9, uint16_t, f); - define_3x3_reduction(m); - define_3x3_reduction(m); - define_3x3_reduction(m); define_3x3_reduction(m); define_3x3_reduction(m); define_3x3_reduction(m); @@ -132,9 +140,6 @@ PYBIND11_MODULE(_aare, m) { define_3x3_reduction(m); define_3x3_reduction(m); - reduce_to_3x3(m); - reduce_to_3x3(m); - reduce_to_3x3(m); reduce_to_3x3(m); reduce_to_3x3(m); reduce_to_3x3(m); diff --git a/python/src/module_config.hpp b/python/src/module_config.hpp new file mode 100644 index 00000000..ad5d7ca1 --- /dev/null +++ b/python/src/module_config.hpp @@ -0,0 +1,6 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include + +// Configure module wide pedestal type for cluster finding +using pd_type = double; \ No newline at end of file diff --git a/python/src/np_helper.hpp b/python/src/np_helper.hpp index c0f5c3fd..d21aceff 100644 --- a/python/src/np_helper.hpp +++ b/python/src/np_helper.hpp @@ -41,31 +41,60 @@ template py::array return_vector(std::vector *vec) { free_when_done); // numpy array references this parent } -// todo rewrite generic -template -auto get_shape_3d(const py::array_t &arr) { - return aare::Shape<3>{arr.shape(0), arr.shape(1), arr.shape(2)}; +// Create a NDView of a numpy array. NDView assumes C-order strides so we +// reject anything that is not Ndim dimensional and C-contiguous instead of +// silently reading the data in the wrong order. +template +auto get_checked_shape(const py::array_t &arr) { + if (arr.ndim() != Ndim) { + throw py::value_error( + fmt::format("Expected {}D array, got {}D", Ndim, arr.ndim())); + } + if (!(arr.flags() & py::array::c_style)) { + throw py::value_error( + "Array is not C-contiguous. Use np.ascontiguousarray(arr)"); + } + aare::Shape shape{}; + for (ssize_t i = 0; i < Ndim; ++i) { + shape[i] = arr.shape(i); + } + return shape; +} + +template +auto make_view(py::array_t &arr) { + auto shape = get_checked_shape(arr); + return aare::NDView(arr.mutable_data(), shape); +} + +// Read only view, also works for numpy arrays that are not writeable +template +auto make_const_view(const py::array_t &arr) { + auto shape = get_checked_shape(arr); + return aare::NDView(arr.data(), shape); } template auto make_view_3d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_3d(arr)); + return make_view<3>(arr); } - -template -auto get_shape_2d(const py::array_t &arr) { - return aare::Shape<2>{arr.shape(0), arr.shape(1)}; -} - -template -auto get_shape_1d(const py::array_t &arr) { - return aare::Shape<1>{arr.shape(0)}; -} - template auto make_view_2d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_2d(arr)); + return make_view<2>(arr); } template auto make_view_1d(py::array_t &arr) { - return aare::NDView(arr.mutable_data(), get_shape_1d(arr)); + return make_view<1>(arr); +} + +template +auto make_const_view_3d(const py::array_t &arr) { + return make_const_view<3>(arr); +} +template +auto make_const_view_2d(const py::array_t &arr) { + return make_const_view<2>(arr); +} +template +auto make_const_view_1d(const py::array_t &arr) { + return make_const_view<1>(arr); } template struct fmt_format_trait; // forward declaration diff --git a/python/src/pedestal.hpp b/python/src/pedestal.hpp deleted file mode 100644 index 238848b7..00000000 --- a/python/src/pedestal.hpp +++ /dev/null @@ -1,119 +0,0 @@ -// SPDX-License-Identifier: MPL-2.0 - -#include "aare/Pedestal.hpp" -#include "np_helper.hpp" - -#include -#include -#include -#include -#include - -namespace py = pybind11; - -template -void define_pedestal_bindings(py::module &m, const std::string &name) { - - py::class_>(m, name.c_str(), py::buffer_protocol()) - .def(py::init()) - .def(py::init()) - .def("mean", - [](Pedestal &self) { - auto mea = new NDArray{}; - *mea = self.mean(); - return return_image_data(mea); - }) - .def("view", - [](py::object self_py) { - auto &self = self_py.cast &>(); - auto v = self.view(); - std::array shape{ - static_cast(v.shape(0)), - static_cast(v.shape(1))}; - std::array byte_strides{ - static_cast(v.strides()[0]) * - static_cast(sizeof(SUM_TYPE)), - static_cast(v.strides()[1]) * - static_cast(sizeof(SUM_TYPE))}; - auto arr = py::array_t(shape, byte_strides, v.data(), - self_py); - arr.attr("setflags")(py::arg("write") = false); - return arr; - }) - .def("variance", - [](Pedestal &self) { - auto var = new NDArray{}; - *var = self.variance(); - return return_image_data(var); - }) - .def("std", - [](Pedestal &self) { - auto std = new NDArray{}; - *std = self.std(); - return return_image_data(std); - }) - .def( - "__array_ufunc__", - [](py::object self, py::object ufunc, const std::string &method, - py::args inputs, py::kwargs kwargs) -> py::object { - if (method != "__call__" || inputs.size() != 2 || - inputs[1].ptr() != self.ptr() || - py::cast(ufunc.attr("__name__")) != - "subtract") { - return py::reinterpret_borrow( - Py_NotImplemented); - } - - auto mean = - py::module_::import("builtins").attr("memoryview")(self); - return ufunc(inputs[0], mean, **kwargs); - }, - "Support subtracting a Pedestal from a NumPy array.") - .def("clear", py::overload_cast<>(&Pedestal::clear)) - .def_property_readonly("rows", &Pedestal::rows) - .def_property_readonly("cols", &Pedestal::cols) - .def_property_readonly("n_samples", &Pedestal::n_samples) - .def_property_readonly("sum", &Pedestal::get_sum) - .def_property_readonly("sum2", &Pedestal::get_sum2) - .def("clone", - [&](Pedestal &pedestal) { - return Pedestal(pedestal); - }) - // TODO! add push for other data types - .def("push", - [](Pedestal &pedestal, py::array_t &f) { - auto v = make_view_2d(f); - pedestal.push(v); - }) - .def( - "push_with_threshold", - [](Pedestal &pedestal, - py::array_t &f, - py::array_t &threshold) { - auto frame_view = make_view_2d(f); - auto threshold_view = make_view_2d(threshold); - pedestal.push_with_threshold(frame_view, threshold_view); - }, - py::arg("frame").noconvert(), py::arg("threshold").noconvert()) - .def( - "push_no_update", - [](Pedestal &pedestal, - py::array_t &f) { - auto v = make_view_2d(f); - pedestal.push_no_update(v); - }, - py::arg().noconvert()) - .def("update_mean", &Pedestal::update_mean) - .def_buffer([](Pedestal &self) { - auto mean = self.view(); - return py::buffer_info( - const_cast(mean.data()), sizeof(SUM_TYPE), - py::format_descriptor::format(), 2, - {static_cast(mean.shape(0)), - static_cast(mean.shape(1))}, - {static_cast(mean.strides()[0] * sizeof(SUM_TYPE)), - static_cast(mean.strides()[1] * - sizeof(SUM_TYPE))}, - true); - }); -} diff --git a/python/src/raw_master_file.hpp b/python/src/raw_master_file.hpp index 7f1837c0..40e942a9 100644 --- a/python/src/raw_master_file.hpp +++ b/python/src/raw_master_file.hpp @@ -66,7 +66,8 @@ void define_raw_master_file_bindings(py::module &m) { .def_property_readonly("total_frames_expected", &RawMasterFile::total_frames_expected) - .def_property_readonly("geometry", &RawMasterFile::geometry) + .def_property_readonly("detector_layout", + &RawMasterFile::detector_layout) .def_property_readonly("udp_interfaces_per_module", &RawMasterFile::udp_interfaces_per_module) .def_property_readonly("analog_samples", &RawMasterFile::analog_samples, @@ -107,9 +108,46 @@ void define_raw_master_file_bindings(py::module &m) { return std::nullopt; } }) - .def_property_readonly("period", [](RawMasterFile &self) { - double seconds = - std::chrono::duration(self.period()).count(); - return seconds; - }); + .def_property_readonly("rois", &RawMasterFile::rois, R"( + Get the ROIs defined in the master file + + Returns + ---------- + List[ROI] + List of ROIs (default complete ROI) + )") + .def_property_readonly("udp_port_types", &RawMasterFile::udp_port_types, + R"( + Get the types of UDP ports + + Returns + ---------- + Optional[List[UDPPortPosition]] + Optional vector of UDP port types as strings (only present for + masterfile version >= 8.1) + )") + .def_property_readonly("disabled_udp_ports", + &RawMasterFile::disabled_udp_ports, R"( + Get the indices of disabled UDP ports + + Returns + ---------- + List[int] + Vector of disabled UDP port indices relative to UDP port types (empty if none are disabled) + )") + .def_property_readonly( + "period", + [](RawMasterFile &self) { + double seconds = + std::chrono::duration(self.period()).count(); + return seconds; + }) + + .def("__enter__", + [](RawMasterFile &self) -> RawMasterFile & { return self; }) + + .def("__exit__", + []([[maybe_unused]] RawMasterFile &self, + [[maybe_unused]] py::object, [[maybe_unused]] py::object, + [[maybe_unused]] py::object) { return; }); } diff --git a/python/src/var_cluster.hpp b/python/src/var_cluster.hpp index 44d70db1..1a3b674c 100644 --- a/python/src/var_cluster.hpp +++ b/python/src/var_cluster.hpp @@ -21,40 +21,42 @@ void define_var_cluster_finder_bindings(py::module &m) { auto *ptr = new NDArray(self.labeled()); return return_image_data(ptr); }) - .def("set_noiseMap", - [](VarClusterFinder &self, - py::array_t - noise_map) { - auto noise_map_span = make_view_2d(noise_map); - self.set_noiseMap(noise_map_span); - }) + .def( + "set_noiseMap", // conversions ok + [](VarClusterFinder &self, + py::array_t + noise_map) { + auto noise_map_span = make_view_2d(noise_map); + self.set_noiseMap(noise_map_span); + }, + py::arg("noise_map")) .def("set_numberOfNeighbours", &VarClusterFinder::set_numberOfNeighbours) .def("set_empty_surroundingPixels", &VarClusterFinder::set_empty_surroundingPixels) .def("set_peripheralThresholdFactor", &VarClusterFinder::set_peripheralThresholdFactor) - .def("find_clusters", - [](VarClusterFinder &self, - py::array_t - img) { - auto view = make_view_2d(img); - self.find_clusters(view); - }) - .def("find_clusters_X", - [](VarClusterFinder &self, - py::array_t - img) { - auto img_span = make_view_2d(img); - self.find_clusters_X(img_span); - }) - .def("single_pass", - [](VarClusterFinder &self, - py::array_t - img) { - auto img_span = make_view_2d(img); - self.single_pass(img_span); - }) + .def( + "find_clusters", + [](VarClusterFinder &self, py::array_t img) { + auto view = make_view_2d(img); + self.find_clusters(view); + }, + py::arg("img").noconvert()) + .def( + "find_clusters_X", + [](VarClusterFinder &self, py::array_t img) { + auto img_span = make_view_2d(img); + self.find_clusters_X(img_span); + }, + py::arg("img").noconvert()) + .def( + "single_pass", + [](VarClusterFinder &self, py::array_t img) { + auto img_span = make_view_2d(img); + self.single_pass(img_span); + }, + py::arg("img").noconvert()) .def("hits", [](VarClusterFinder &self) { auto ptr = new std::vector::Hit>( diff --git a/python/tests/test_Cluster.py b/python/tests/test_Cluster.py index cafb118e..28b90e9d 100644 --- a/python/tests/test_Cluster.py +++ b/python/tests/test_Cluster.py @@ -3,7 +3,7 @@ import pytest import numpy as np from aare import _aare #import the C++ module -from aare import corner +from aare import ROI, corner from conftest import test_data_path @@ -91,6 +91,22 @@ def test_calculate_eta(): assert eta2.sum == 4 +def test_cluster_representations(): + """Test Cluster representation""" + + cluster = _aare.Cluster2x2i(5, 6, np.array([1, 2, 3, 4], dtype=np.int32)) + assert repr(cluster) == "Cluster2x2i(x=5, y=6, data=[[1, 2], [3, 4]])" + +def test_eta_representations(): + """Test Eta representation""" + eta = _aare.Etai() + eta.x = 0.25 + eta.y = 0.75 + eta.c = corner.cBottomRight + eta.sum = 42 + assert repr(eta) == "Etai(x=0.25, y=0.75, c=BottomRight, sum=42)" + + def test_max_sum(): """Max 2x2 Sum""" cluster = _aare.Cluster3x3i(5,5,np.array([1, 1, 1, 2, 3, 1, 2, 2, 1], dtype=np.int32)) @@ -103,19 +119,59 @@ def test_max_sum(): def test_cluster_finder(): """Test ClusterFinder""" + shape = [100,100] + cf = _aare.ClusterFinder_Cluster3x3i(shape) - clusterfinder = _aare.ClusterFinder_Cluster3x3i([100,100]) + #Push 1000 frames to the pedestal + for i in range(1000): + frame = np.random.normal(loc = 100, scale = 5, size = shape).astype(np.uint16) + cf.push_pedestal_frame(frame) + cf.update_threshold() + frame = np.zeros(shape=shape, dtype=np.uint16) + cf.find_clusters(frame) - #frame = np.random.rand(100,100) - frame = np.zeros(shape=[100,100]) - - clusterfinder.find_clusters(frame) - - clusters = clusterfinder.steal_clusters(False) #conversion does not work + clusters = cf.steal_clusters(False) #conversion does not work assert clusters.size == 0 +def test_cluster_finder_set_nSigma_before_pedestal_is_ready(): + shape = [10, 10] + n_samples = 4 + cf = _aare.ClusterFinder_Cluster3x3i(shape, min_pedestal_samples=n_samples) + assert not cf.pedestal_ready + + cf.nSigma = 50 + assert cf.nSigma == 50 + + # Pedestal of 100 with a noise of 2 + for i in range(n_samples): + cf.push_pedestal_frame(np.full(shape, 98 + 4 * (i % 2), dtype=np.uint16)) + assert cf.pedestal_ready + + # 30 ADU is a cluster at the default nSigma of 5 but not at 50 + frame = np.full(shape, 100, dtype=np.uint16) + frame[5, 5] = 130 + cf.find_clusters(frame) + assert cf.steal_clusters().size == 0 + + +def test_cluster_finder_mt_set_nSigma_before_pedestal_is_ready(): + shape = [10, 10] + n_samples = 4 + cf = _aare.ClusterFinderMT_Cluster3x3i( + shape, n_threads=2, min_pedestal_samples=n_samples + ) + assert not cf.pedestal_ready() + + cf.set_nSigma(50) + + for i in range(n_samples): + cf.push_pedestal_frame(np.full(shape, 98 + 4 * (i % 2), dtype=np.uint16)) + assert cf.pedestal_ready() + cf.stop() + + def test_2x2_reduction(): """Test 2x2 Reduction""" cluster = _aare.Cluster3x3i(5,5,np.array([1, 1, 1, 2, 3, 1, 2, 2, 1], dtype=np.int32)) @@ -144,3 +200,20 @@ def test_3x3_reduction(): + +@pytest.mark.parametrize( + "cluster_type, n_pixels", + [(_aare.Cluster3x3i, 9), (_aare.Cluster2x2i, 4)], +) +def test_cluster_constructor_requires_exact_data_length(cluster_type, n_pixels): + """Data must hold exactly one value per pixel""" + for bad in ( + np.ones(n_pixels - 1, dtype=np.int32), + np.ones(n_pixels + 1, dtype=np.int32), + np.ones((n_pixels, 1), dtype=np.int32), + ): + with pytest.raises(ValueError, match=f"{n_pixels} values"): + cluster_type(0, 0, bad) + + cluster = cluster_type(0, 0, np.arange(n_pixels, dtype=np.int32)) + assert (cluster.data.ravel() == np.arange(n_pixels)).all() diff --git a/python/tests/test_ClusterFile.py b/python/tests/test_ClusterFile.py index 3e2fb81d..6ce8a11b 100644 --- a/python/tests/test_ClusterFile.py +++ b/python/tests/test_ClusterFile.py @@ -6,10 +6,279 @@ import boost_histogram as bh import time from pathlib import Path import pickle +import struct +import gc +import weakref -from aare import ClusterFile +from aare import ClusterFile, ROI from conftest import test_data_path + +@pytest.fixture +def iterator_file(tmp_path): + fname = tmp_path / "iteration.clust" + records = [ + (-11, []), + (42, [(5, 6, 10), (7, 8, 20)]), + (103, []), + (104, [(5, 6, 30), (7, 8, 40)]), + (105, []), + ] + fname.write_bytes(b"".join( + struct.pack("@iI", number, len(clusters)) + + b"".join(struct.pack("@HH9i", x, y, *([value] * 9)) + for x, y, value in clusters) + for number, clusters in records + )) + return fname + + +def test_frames_preserve_empty_frames_and_frame_numbers(iterator_file): + with ClusterFile(iterator_file) as reader: + frames = reader.frames() + assert iter(frames) is frames + assert reader.tell() == 0 + results = list(frames) + assert [frame.frame_number for frame in results] == [-11, 42, 103, 104, 105] + assert [frame.size for frame in results] == [0, 2, 0, 2, 0] + for _ in range(2): + with pytest.raises(StopIteration): + next(frames) + assert list(reader.frames()) == [] + + +@pytest.mark.parametrize("chunk_size, sizes", [(1, [1, 1, 1, 1]), (2, [2, 2]), + (3, [3, 1]), (10, [4])]) +@pytest.mark.parametrize("explicit", [False, True]) +def test_chunks_cross_frames(iterator_file, chunk_size, sizes, explicit): + with ClusterFile(iterator_file, chunk_size=chunk_size) as reader: + chunks = reader.chunks(chunk_size=chunk_size) if explicit else reader.chunks() + assert iter(chunks) is chunks + assert reader.tell() == 0 + results = list(chunks) + assert [chunk.size for chunk in results] == sizes + data = np.concatenate([np.asarray(chunk) for chunk in results]) + np.testing.assert_array_equal(data["x"], [5, 7, 5, 7]) + np.testing.assert_array_equal(data["data"][:, 0, 0], [10, 20, 30, 40]) + for _ in range(2): + with pytest.raises(StopIteration): + next(chunks) + + +def test_chunk_override_preserves_default_iteration(iterator_file): + with ClusterFile(iterator_file, chunk_size=2) as reader: + assert iter(reader) is reader + assert next(reader.chunks(1)).size == 1 + assert [chunk.size for chunk in reader] == [2, 1] + + +@pytest.mark.parametrize("use_roi", [False, True]) +@pytest.mark.parametrize("use_noise", [False, True]) +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_apply_filters_and_gain(iterator_file, use_roi, use_noise, method): + with ClusterFile(iterator_file, chunk_size=3) as reader: + if use_roi: + reader.set_roi(ROI(0, 6, 0, 12)) + if use_noise: + reader.set_noise_map(np.full((12, 12), 15, dtype=np.int32)) + reader.set_gain_map(np.full((12, 12), 2.0)) + results = list(getattr(reader, method)()) + if method == "frames": + assert [frame.frame_number for frame in results] == [-11, 42, 103, 104, 105] + expected = [value / 2 for x, value in [(5, 10), (7, 20), (5, 30), (7, 40)] + if (not use_roi or x == 5) and (not use_noise or value > 15)] + data = np.concatenate([np.asarray(result) for result in results]) + np.testing.assert_array_equal(data["data"][:, 0, 0], expected) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_handle_all_rejected_clusters(iterator_file, method): + with ClusterFile(iterator_file) as reader: + reader.set_roi(ROI(0, 1, 0, 1)) + results = list(getattr(reader, method)()) + assert len(results) == (5 if method == "frames" else 0) + assert all(result.size == 0 for result in results) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterator_keeps_file_alive(iterator_file, method): + reader = ClusterFile(iterator_file) + owner = weakref.ref(reader) + iterator = getattr(reader, method)() + del reader + gc.collect() + assert owner() is not None + results = list(iterator) + assert sum(result.size for result in results) == 4 + del iterator + gc.collect() + assert owner() is None + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_retained_results_and_numpy_views_own_storage(iterator_file, method): + with ClusterFile(iterator_file, chunk_size=2) as reader: + reader.read_frame() # Skip the initial empty frame. + iterator = getattr(reader, method)() + first = next(iterator) + array = np.asarray(first) + expected = array.copy() + results = list(iterator) + np.testing.assert_array_equal(np.asarray(first), expected) + del first, results, iterator, reader + gc.collect() + np.testing.assert_array_equal(array, expected) + + +def test_frame_iteration_resumes_without_reading_ahead(iterator_file): + with ClusterFile(iterator_file) as reader: + reader.read_frame() + for frame in reader.frames(): + assert frame.frame_number == 42 + break + assert reader.read_frame().frame_number == 103 + assert [frame.frame_number for frame in reader.frames()] == [104, 105] + + +def test_partial_chunk_must_be_completed_before_frame_iteration(iterator_file): + with ClusterFile(iterator_file) as reader: + for chunk in reader.chunks(1): + assert chunk.size == 1 + break + with pytest.raises(RuntimeError, match="clusters left"): + next(reader.frames()) + assert reader.read_clusters(1).size == 1 + assert [frame.frame_number for frame in reader.frames()] == [103, 104, 105] + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_empty_iterators(tmp_path, method): + fname = tmp_path / "empty.clust" + fname.touch() + with ClusterFile(fname) as reader: + iterator = getattr(reader, method)() + for _ in range(2): + with pytest.raises(StopIteration): + next(iterator) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +@pytest.mark.parametrize("started", [False, True]) +def test_iterators_reject_closed_files(iterator_file, method, started): + reader = ClusterFile(iterator_file, chunk_size=1) + iterator = getattr(reader, method)() + if started: + next(iterator) + reader.close() + with pytest.raises(RuntimeError, match="not opened for reading"): + next(iterator) + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +@pytest.mark.parametrize("mode", ["w", "a"]) +def test_iterators_reject_writing_modes(tmp_path, method, mode): + with ClusterFile(tmp_path / "output.clust", mode=mode) as reader: + with pytest.raises(RuntimeError, match="not opened for reading"): + next(getattr(reader, method)()) + + +def test_chunk_sizes_must_be_positive(iterator_file): + with ClusterFile(iterator_file, chunk_size=0) as reader: + with pytest.raises(ValueError, match="greater than zero"): + reader.chunks() + with pytest.raises(ValueError, match="greater than zero"): + reader.chunks(0) + with pytest.raises(ValueError, match="greater than zero"): + next(reader) + with pytest.raises(TypeError): + reader.chunks(-1) + assert reader.tell() == 0 + assert reader.read_clusters(0).size == 0 + assert len(list(reader.frames())) == 5 + + +@pytest.mark.parametrize("method", ["frames", "chunks"]) +def test_iterators_defer_read_errors_until_next_result(tmp_path, method): + fname = tmp_path / "truncated.clust" + cluster = struct.pack("@HH9i", 5, 6, *range(9)) + frame = struct.pack("@iI", 42, 1) + cluster + fname.write_bytes(frame + frame[:-1]) + with ClusterFile(fname, chunk_size=1) as reader: + iterator = getattr(reader, method)() + assert next(iterator).size == 1 + assert reader.tell() == len(frame) + with pytest.raises(RuntimeError): + next(iterator) + + +@pytest.mark.parametrize( + "shape, dtype", + [(shape, dtype) + for shape in [(2, 2), (3, 3), (5, 5), (7, 7), (9, 9)] + for dtype in [np.int32, np.float32, np.float64]] + + [((3, 3), np.int16)], +) +def test_iterators_are_bound_for_each_cluster_type(tmp_path, shape, dtype): + fname = tmp_path / "typed_frames.clust" + values = list(range(shape[0] * shape[1])) + record = struct.pack("@HH" + np.dtype(dtype).char * len(values), 5, 6, *values) + fname.write_bytes((struct.pack("@iI", -42, 1) + record) * 3) + with ClusterFile(fname, cluster_size=shape, dtype=dtype) as reader: + frame = next(reader.frames()) + assert frame.frame_number == -42 + np.testing.assert_array_equal(np.asarray(frame)["data"].reshape(-1), values) + chunks = list(reader.chunks()) + assert len(chunks) == 1 + assert chunks[0].size == 2 + np.testing.assert_array_equal(np.asarray(chunks[0])["data"].reshape(-1), values * 2) + + +def test_read_frame_returns_none_at_eof(tmp_path): + fname = tmp_path / "empty.clust" + fname.touch() + + with ClusterFile(fname) as f: + assert f.read_frame() is None + + +def test_read_frame_raises_for_malformed_file(tmp_path): + fname = tmp_path / "malformed.clust" + fname.write_bytes(b"\x00") + + with ClusterFile(fname) as f, pytest.raises(RuntimeError): + f.read_frame() + + +@pytest.mark.parametrize("use_roi", [False, True]) +@pytest.mark.parametrize("method", ["read_clusters", "default", "frames", "chunks"]) +@pytest.mark.parametrize( + "size", + [ + pytest.param(3, id="partial-frame-number"), + pytest.param(4, id="missing-cluster-count"), + pytest.param(7, id="partial-cluster-count"), + pytest.param(48, id="missing-cluster-record"), + pytest.param(87, id="partial-cluster-record"), + ], +) +def test_chunk_reads_reject_incomplete_frames(tmp_path, use_roi, method, size): + fname = tmp_path / "incomplete.clust" + cluster = struct.pack("@HH9i", 5, 6, *range(9)) + frame = struct.pack("@iI", 42, 2) + cluster * 2 + fname.write_bytes(frame[:size]) + + with ClusterFile(fname) as reader: + if use_roi: + reader.set_roi(ROI(0, 10, 0, 10)) + with pytest.raises(RuntimeError): + if method == "default": + next(reader) + elif method == "read_clusters": + reader.read_clusters(10) + else: + next(getattr(reader, method)()) + + @pytest.mark.withdata def test_cluster_file(test_data_path): """Test ClusterFile""" diff --git a/python/tests/test_ClusterVector.py b/python/tests/test_ClusterVector.py index 30f934fa..b0e9cffa 100644 --- a/python/tests/test_ClusterVector.py +++ b/python/tests/test_ClusterVector.py @@ -83,12 +83,15 @@ def test_make_a_hitmap_from_cluster_vector(): def test_2x2_reduction(): cv = ClusterVector((3,3)) + cv.frame_number = -135 cv.push_back(_aare.Cluster3x3i(5, 5, np.array([1, 1, 1, 2, 3, 1, 2, 2, 1], dtype=np.int32))) cv.push_back(_aare.Cluster3x3i(5, 5, np.array([2, 2, 1, 2, 3, 1, 1, 1, 1], dtype=np.int32))) - reduced_cv = np.array(_aare.reduce_to_2x2(cv), copy=False) + reduced = _aare.reduce_to_2x2(cv) + reduced_cv = np.array(reduced, copy=False) + assert reduced.frame_number == cv.frame_number assert reduced_cv.size == 2 assert reduced_cv[0]["x"] == 5 assert reduced_cv[0]["y"] == 5 @@ -100,14 +103,17 @@ def test_2x2_reduction(): def test_3x3_reduction(): cv = _aare.ClusterVector_Cluster5x5d() + cv.frame_number = 246 cv.push_back(_aare.Cluster5x5d(5,5,np.array([1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 2.0, 2.0, 3.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], dtype=np.double))) cv.push_back(_aare.Cluster5x5d(5,5,np.array([1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 2.0, 2.0, 3.0, 1.0, 1.0, 1.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], dtype=np.double))) - reduced_cv = np.array(_aare.reduce_to_3x3(cv), copy=False) + reduced = _aare.reduce_to_3x3(cv) + reduced_cv = np.array(reduced, copy=False) + assert reduced.frame_number == cv.frame_number assert reduced_cv.size == 2 assert reduced_cv[0]["x"] == 5 assert reduced_cv[0]["y"] == 5 @@ -130,3 +136,26 @@ def test_masking(): assert cv_masked_array[0]["x"] == 1 assert cv_masked_array[0]["y"] == 2 assert (cv_masked_array[0]["data"] == np.ones((3,3),dtype=np.int32)).all() + + +def test_masking_requires_c_contiguous_array(): + cv = _aare.ClusterVector_Cluster3x3i() + cv.push_back(_aare.Cluster3x3i(1, 2, np.ones(9, dtype=np.int32))) + cv.push_back(_aare.Cluster3x3i(3, 4, np.ones(9, dtype=np.int32))) + + mask = np.array([True, False, True, False], dtype=bool)[::2] + assert not mask.flags.c_contiguous + + with pytest.raises(ValueError, match="C-contiguous"): + cv(mask) + + +def test_masking_requires_one_dimension(): + cv = _aare.ClusterVector_Cluster3x3i() + cv.push_back(_aare.Cluster3x3i(1, 2, np.ones(9, dtype=np.int32))) + cv.push_back(_aare.Cluster3x3i(3, 4, np.ones(9, dtype=np.int32))) + + mask = np.array([[True, False]], dtype=bool) + + with pytest.raises(ValueError): + cv(mask) diff --git a/python/tests/test_CtbRawFile.py b/python/tests/test_CtbRawFile.py new file mode 100644 index 00000000..e23c99f2 --- /dev/null +++ b/python/tests/test_CtbRawFile.py @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: MPL-2.0 + + +import pytest +import numpy as np +from aare import CtbRawFile + + +@pytest.mark.withdata +def test_basic_properites(test_data_path): + f = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + assert f.total_frames == 47 + assert f.image_size_in_bytes == 640 + assert f.master.analog_samples == 10 + assert f.master.digital_samples is None + assert f.master.transceiver_samples is None + + +@pytest.mark.withdata +def test_seek_and_read(test_data_path): + f = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f.seek(0) + header, raw_data = f.read_frame() + assert header['frameNumber'] == 48 #we know that the first frame in this file is 48 + + #Seek a multiple of frames in file + f.seek(9) + header, raw_data = f.read_frame() + assert header['frameNumber'] == 57 + +@pytest.mark.withdata +def test_read_all_frames_direct_and_iterate(test_data_path): + """ + We want to make sure that iterating and reading give the same result. + """ + f0 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f1 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + + for h0, d0 in f0: + h1, d1 = f1.read_frame() + assert h0 == h1 + assert np.all(d0 == d1) + +@pytest.mark.withdata +def test_read_all_frames_with_index_and_iterate(test_data_path): + """ + We want to make sure that iterating and reading give the same result. + """ + f0 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + f1 = CtbRawFile(test_data_path/'raw/ctb/run_master_0.json') + + frames_read = 0 + for i,(h0, d0) in enumerate(f0): + h1, d1 = f1.read_frame(i) + assert h0 == h1 + assert np.all(d0 == d1) + frames_read += 1 + + assert frames_read == f0.total_frames diff --git a/python/tests/test_FastPedestal.py b/python/tests/test_FastPedestal.py new file mode 100644 index 00000000..1feacaea --- /dev/null +++ b/python/tests/test_FastPedestal.py @@ -0,0 +1,160 @@ +# SPDX-License-Identifier: MPL-2.0 +import numpy as np +import pytest + +from aare import ( + FastPedestal, + FastPedestal_d, + FastPedestal_f, + FastPedestal_i16, +) + + +@pytest.mark.parametrize( + ("dtype", "pedestal_type"), + [ + (np.float64, FastPedestal_d), + (np.float32, FastPedestal_f), + (np.int16, FastPedestal_i16), + ], +) +def test_fast_pedestal_factory(dtype, pedestal_type): + pedestal = FastPedestal(2, 3, n_samples=4, dtype=dtype) + + assert isinstance(pedestal, pedestal_type) + assert pedestal.rows == 2 + assert pedestal.cols == 3 + assert pedestal.n_samples == 4 + + +def test_fast_pedestal_factory_defaults_to_double(): + assert isinstance(FastPedestal(2, 3), FastPedestal_d) + + +def test_fast_pedestal_factory_rejects_unbound_dtype(): + with pytest.raises(ValueError, match="Unsupported dtype for FastPedestal"): + FastPedestal(2, 3, dtype=np.int32) + + +@pytest.mark.parametrize( + ("kwargs", "expected_n_samples"), + [ + ({"rows": 2, "cols": 3}, 1000), + ({"rows": 2, "cols": 3, "n_samples": 4}, 4), + ], +) +def test_fast_pedestal_binding_accepts_constructor_keywords( + kwargs, expected_n_samples +): + pedestal = FastPedestal_d(**kwargs) + + assert pedestal.rows == 2 + assert pedestal.cols == 3 + assert pedestal.n_samples == expected_n_samples + + +@pytest.mark.parametrize( + ("dtype", "pedestal_type", "expected_dtype"), + [ + (np.float64, FastPedestal_d, np.float64), + (np.float32, FastPedestal_f, np.float32), + (np.int16, FastPedestal_i16, np.int16), + ], +) +def test_fast_pedestal_factory_from_file( + tmp_path, dtype, pedestal_type, expected_dtype +): + frames = np.array( + [[[100, 100]], [[2, 4]], [[4, 6]], [[5, 7]]], dtype=np.uint16 + ) + filename = tmp_path / "frames.npy" + np.save(filename, frames) + + pedestal = FastPedestal.from_file( + filename, n_samples=2, skip_first=1, dtype=dtype + ) + + assert isinstance(pedestal, pedestal_type) + assert pedestal.ready + assert pedestal.cur_samples == 2 + assert pedestal.mean().dtype == expected_dtype + np.testing.assert_array_equal(pedestal.mean(), [[4, 6]]) + + +def test_fast_pedestal_factory_from_file_rejects_unbound_dtype(): + with pytest.raises(ValueError, match="Unsupported dtype for FastPedestal"): + FastPedestal.from_file("unused.npy", dtype=np.int32) + + +def test_fast_pedestal_from_file_rejects_skip_beyond_end(tmp_path): + filename = tmp_path / "frames.npy" + np.save(filename, np.zeros((1, 1, 1), dtype=np.uint16)) + + with pytest.raises(RuntimeError, match="less frames"): + FastPedestal.from_file(filename, n_samples=1, skip_first=2) + + +@pytest.mark.parametrize( + ("pedestal_type", "expected_dtype"), + [(FastPedestal_d, np.float64), (FastPedestal_f, np.float32)], +) +def test_fast_pedestal_initialization(pedestal_type, expected_dtype): + pedestal = pedestal_type(2, 3, 2) + first = np.array([[2, 4, 6], [8, 10, 12]], dtype=np.uint16) + second = np.array([[4, 6, 8], [10, 12, 14]], dtype=np.uint16) + + pedestal.add_init_frame(first) + pedestal.add_init_frame(second) + + + expected_mean = np.array( + [[3, 5, 7], [9, 11, 13]], dtype=expected_dtype + ) + np.testing.assert_array_equal(pedestal.mean(), expected_mean) + np.testing.assert_array_equal(pedestal.std(), np.ones((2, 3))) + + +def test_fast_pedestal_steady_state_push_ema(): + pedestal = FastPedestal_d(1, 2, 2) + pedestal.add_init_frame(np.array([[2, 4]], dtype=np.uint16)) + pedestal.add_init_frame(np.array([[4, 6]], dtype=np.uint16)) + + + pedestal.push_ema(np.array([[6, 8]], dtype=np.uint16)) + + np.testing.assert_array_equal(pedestal.mean(), [[4.5, 6.5]]) + + +def test_fast_pedestal_exposes_read_only_buffer_and_subtraction(): + pedestal = FastPedestal_d(1, 2, 1) + pedestal.add_init_frame(np.array([[2, 4]], dtype=np.uint16)) + + + view = np.asarray(pedestal) + result = np.array([[12, 14]], dtype=np.uint16) - pedestal + + np.testing.assert_array_equal(view, [[2, 4]]) + np.testing.assert_array_equal(result, [[10, 10]]) + assert np.shares_memory(view, pedestal.view()) + assert not view.flags.writeable + + +def test_fast_pedestal_rejects_wrong_shape(): + pedestal = FastPedestal_d(2, 3) + + with pytest.raises(RuntimeError, match="shape"): + pedestal.add_init_frame(np.zeros((2, 2), dtype=np.uint16)) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +def test_fast_pedestal_std_uses_double_variance(dtype): + pedestal = FastPedestal(1, 1, n_samples=2, dtype=dtype) + pedestal.add_init_frame(np.array([[0]], dtype=np.uint16)) + pedestal.add_init_frame(np.array([[1000]], dtype=np.uint16)) + + assert pedestal.std().dtype == dtype + np.testing.assert_array_equal(pedestal.std(), [[500]]) + + pedestal.push_ema(np.array([[1000]], dtype=np.uint16)) + expected = np.array([[np.sqrt(187500.0)]], dtype=dtype) + np.testing.assert_array_equal(pedestal.std(), expected) diff --git a/python/tests/test_Fit.py b/python/tests/test_Fit.py index 3a477b27..2e876ed9 100644 --- a/python/tests/test_Fit.py +++ b/python/tests/test_Fit.py @@ -1,15 +1,401 @@ # SPDX-License-Identifier: MPL-2.0 import numpy as np +import pytest + import aare +MINIMIZERS = [ + aare.Minimizer.Migrad, + aare.Minimizer.Fumili, + aare.Minimizer.LevenbergMarquardt, + aare.Minimizer.VarPro, +] -def test_gaussian_model_evaluates_and_fits_data(): +def gaussian(x, A, mu, sigma): + return A * np.exp(-0.5 * ((x - mu) / sigma) ** 2) + + +def uses_minuit2(minimizer): + return minimizer not in ( + aare.Minimizer.LevenbergMarquardt, + aare.Minimizer.VarPro, + ) + + +@pytest.fixture(params=MINIMIZERS, ids=lambda m: m.name) +def minimizer(request): + return request.param + + +@pytest.fixture +def gaussian_data(): + x = np.linspace(-6.0, 6.0, 61) + y = gaussian(x, 120.0, 0.8, 1.3) + y_err = np.ones_like(y) + return x, y, y_err + + +def test_gaussian_model_evaluates_and_fits_data(minimizer): x = np.linspace(-5.0, 5.0, 51) expected = np.array([20.0, 0.5, 1.2]) - model = aare.Gaussian() + model = aare.Gaussian(minimizer=minimizer) y = model(x, expected) result = model.fit(x, y) np.testing.assert_allclose(result["par"], expected, atol=2e-3) + + +def test_minimizer_can_be_selected(): + assert aare.Gaussian().minimizer == aare.Minimizer.Migrad + assert set(aare.Minimizer.__members__) == { + "Migrad", + "Fumili", + "LevenbergMarquardt", + "VarPro", + } + + for minimizer in MINIMIZERS: + assert aare.Gaussian(minimizer=minimizer).minimizer == minimizer + + model = aare.Gaussian() + model.minimizer = aare.Minimizer.LevenbergMarquardt + assert model.minimizer == aare.Minimizer.LevenbergMarquardt + + +def test_fit_returns_errors_and_chi2(gaussian_data, minimizer): + x, y, y_err = gaussian_data + model = aare.Gaussian(compute_errors=True, minimizer=minimizer) + result = model.fit(x, y, y_err) + + assert set(result) == {"par", "par_err", "chi2"} + assert result["par"].shape == (3,) + assert result["par_err"].shape == (3,) + assert result["chi2"].shape == (1,) + np.testing.assert_allclose(result["par"], [120.0, 0.8, 1.3], rtol=1e-4) + np.testing.assert_allclose( + result["par_err"], [0.361, 0.00451, 0.00451], rtol=2e-2 + ) + assert result["chi2"][0] == pytest.approx(0.0, abs=1e-3) + + without_errors = aare.fit(aare.Gaussian(minimizer=minimizer), x, y) + assert set(without_errors) == {"par", "chi2"} + + +def test_pol1_weighted_errors_match_analytic_formula(minimizer): + x = np.arange(20, dtype=float) + y = 2.0 + 0.5 * x + y_err = 1.0 + 0.1 * x + w = 1.0 / y_err**2 + S, Sx, Sxx = w.sum(), (w * x).sum(), (w * x * x).sum() + delta = S * Sxx - Sx * Sx + + result = aare.Pol1(compute_errors=True, minimizer=minimizer).fit(x, y, y_err) + + np.testing.assert_allclose(result["par"], [2.0, 0.5], rtol=1e-6) + np.testing.assert_allclose( + result["par_err"], + [np.sqrt(Sxx / delta), np.sqrt(S / delta)], + rtol=1e-6, + ) + + +def test_fixed_parameter_reports_zero_error(gaussian_data, minimizer): + x, y, y_err = gaussian_data + model = aare.Gaussian(compute_errors=True, minimizer=minimizer) + model.FixParameter("sigma", 1.5) + result = model.fit(x, y, y_err) + + assert result["par"][2] == 1.5 + assert result["par_err"][2] == 0.0 + assert result["par"][0] == pytest.approx(111.15, rel=1e-3) + assert result["chi2"][0] == pytest.approx(1684.25, rel=1e-3) + + model.ReleaseParameter("sigma") + result = model.fit(x, y, y_err) + assert result["par"][2] == pytest.approx(1.3, rel=1e-4) + + +def test_active_limit_ends_on_the_limit(gaussian_data, minimizer): + x, y, y_err = gaussian_data + model = aare.Gaussian(compute_errors=True, minimizer=minimizer) + model.SetParLimits("mu", -5.0, 0.5) + result = model.fit(x, y, y_err) + + margin = 1e-6 if uses_minuit2(minimizer) else 1e-9 + assert result["par"][1] == pytest.approx(0.5, abs=margin) + assert result["par_err"][1] == 0.0 + assert result["par"][0] == pytest.approx(116.88, rel=1e-3) + assert result["chi2"][0] == pytest.approx(4301.95, rel=1e-3) + + inactive = aare.Gaussian(minimizer=minimizer) + inactive.SetParLimits(1, -5.0, 5.0) + np.testing.assert_allclose( + inactive.fit(x, y)["par"], [120.0, 0.8, 1.3], rtol=1e-4 + ) + + +def test_user_start_value(gaussian_data, minimizer): + x, y, _ = gaussian_data + model = aare.Gaussian(minimizer=minimizer) + model.SetParameter("mu", 0.0) + np.testing.assert_allclose(model.fit(x, y)["par"], [120.0, 0.8, 1.3], rtol=1e-4) + + +def test_max_calls_failure_returns_zeros(gaussian_data, minimizer): + x, y, y_err = gaussian_data + model = aare.Gaussian(max_calls=3, compute_errors=True, minimizer=minimizer) + # start far from the minimum so that no minimizer converges in 3 calls + model.SetParameter("mu", -3.0) + model.SetParameter("sigma", 3.0) + result = model.fit(x, y, y_err) + assert np.all(result["par"] == 0.0) + assert np.all(result["par_err"] == 0.0) + assert np.all(result["chi2"] == 0.0) + + +def test_strategy_is_ignored_by_levenberg_marquardt(gaussian_data): + x, y, _ = gaussian_data + lm = aare.Minimizer.LevenbergMarquardt + a = aare.Gaussian(strategy=0, minimizer=lm).fit(x, y) + b = aare.Gaussian(strategy=1, minimizer=lm).fit(x, y) + np.testing.assert_array_equal(a["par"], b["par"]) + + +def test_invalid_parameter_access_raises(): + model = aare.Gaussian() + assert model.par_names == ["A", "mu", "sigma"] + assert model.n_par == 3 + with pytest.raises(RuntimeError): + model.SetParameter("nope", 1.0) + with pytest.raises(IndexError): + model.SetParameter(3, 1.0) + with pytest.raises(RuntimeError): + model.SetParLimits("mu", 1.0, 1.0) + + +def test_mismatched_array_sizes_raise(gaussian_data, minimizer): + x, y, y_err = gaussian_data + model = aare.Gaussian(minimizer=minimizer) + with pytest.raises(RuntimeError): + model.fit(x, y[:-1]) + with pytest.raises(RuntimeError): + model.fit(x, y, y_err[:-1]) + + +@pytest.mark.parametrize("cls", [aare.RisingScurve, aare.FallingScurve]) +def test_scurve_fit(cls, minimizer): + truth = np.array([10.0, 0.1, 50.0, 3.0, 1000.0, 2.0]) + x = np.linspace(0.0, 100.0, 100) + model = cls(max_calls=500, compute_errors=True, minimizer=minimizer) + y = model(x, truth) + result = model.fit(x, y, np.ones_like(y)) + + np.testing.assert_allclose(result["par"], truth, rtol=1e-3) + assert np.all(result["par_err"] > 0.0) + + +@pytest.mark.parametrize("with_errors", [False, True]) +@pytest.mark.parametrize("weighted", [False, True]) +def test_fit_3d_matches_fit_1d(with_errors, weighted, minimizer): + x = np.linspace(-6.0, 6.0, 61) + rows, cols = 4, 3 + y = np.empty((rows, cols, x.size)) + for r in range(rows): + for c in range(cols): + y[r, c] = gaussian(x, 80.0 + 10.0 * r, -0.6 + 0.3 * c, 0.9 + 0.1 * r) + y_err = np.ones_like(y) if weighted else None + + model = aare.Gaussian(compute_errors=with_errors, minimizer=minimizer) + cube = model.fit(x, y, y_err, n_threads=3) + + expected_keys = {"par", "par_err", "chi2"} if with_errors else {"par", "chi2"} + assert set(cube) == expected_keys + assert cube["par"].shape == (rows, cols, 3) + assert cube["chi2"].shape == (rows, cols) + if with_errors: + assert cube["par_err"].shape == (rows, cols, 3) + for r in range(rows): + for c in range(cols): + single = model.fit(x, y[r, c], None if y_err is None else y_err[r, c]) + np.testing.assert_allclose(cube["par"][r, c], single["par"], rtol=1e-10) + assert cube["chi2"][r, c] == pytest.approx(single["chi2"][0], abs=1e-10) + if with_errors: + np.testing.assert_allclose( + cube["par_err"][r, c], single["par_err"], rtol=1e-10 + ) + np.testing.assert_allclose(cube["par"][3, 2], [110.0, 0.0, 1.2], rtol=1e-4, atol=1e-6) + + +def test_all_minimizers_agree_on_3d_data_with_errors(): + x = np.linspace(-5.0, 5.0, 51) + expected = np.array([20.0, 0.5, 1.2]) + single = aare.Gaussian()(x, expected) + y = np.tile(single, (2, 3, 1)) + y_err = np.ones_like(y) + + results = { + minimizer: aare.Gaussian(compute_errors=True, minimizer=minimizer).fit( + x, y, y_err, n_threads=2 + ) + for minimizer in MINIMIZERS + } + reference = results[aare.Minimizer.Migrad] + for minimizer, result in results.items(): + assert result["par"].shape == (2, 3, 3) + assert result["par_err"].shape == (2, 3, 3) + np.testing.assert_allclose(result["par"], np.broadcast_to(expected, (2, 3, 3)), atol=2e-3) + np.testing.assert_allclose(result["chi2"], 0.0, atol=1e-3) + np.testing.assert_allclose(result["par_err"], reference["par_err"], rtol=2e-2) + + +# Regressions of the built-in solvers: fits that used to be reported as valid +# although they had stopped away from the minimum or returned garbage. + + +def test_gaussian_fit_to_noise_fails_or_stays_bounded(minimizer): + # A width collapsing between two scan points let the amplitude diverge + # (up to 1e159) in a fit that was still reported as valid. + x = np.linspace(-6.0, 6.0, 61) + y = np.random.default_rng(0).normal(0.0, 1.0, (20, 20, x.size)) + result = aare.Gaussian(minimizer=minimizer).fit(x, y, n_threads=2) + + par = result["par"] + failed = np.all(par == 0.0, axis=-1) + bounded = np.all(np.isfinite(par) & (np.abs(par) < 1e6), axis=-1) + assert np.all(failed | bounded) + assert np.all(np.isfinite(result["chi2"])) + + +def test_step_reflected_onto_start_is_not_convergence(minimizer): + # The step of +2 reflects at the limit 1 back onto the start 0; the + # truncated step must still be tried. + model = aare.Pol1(minimizer=minimizer) + model.SetParameter("p0", 0.0) + model.FixParameter("p1", 0.0) + model.SetParLimits("p0", -1.0, 1.0) + result = model.fit(np.arange(4.0), np.full(4, 2.2)) + + assert result["par"][0] == pytest.approx(1.0, abs=1e-6) + assert result["chi2"][0] == pytest.approx(5.76, rel=1e-6) + + +@pytest.mark.parametrize( + "amplitude, offset", [(1e8, 0.0), (1e10, 0.0), (120.0, 1e9)] +) +def test_large_parameter_does_not_stop_the_fit(minimizer, amplitude, offset): + # One relative step norm over all parameters let a large amplitude or + # position stop the fit at the start estimate of the width. + x = np.linspace(-6.0, 6.0, 61) + y = gaussian(x, amplitude, 0.8, 1.3) + y_err = np.sqrt(y) + 1.0 + result = aare.Gaussian(minimizer=minimizer, max_calls=500).fit( + x + offset, y, y_err + ) + + par = result["par"] + assert par[0] == pytest.approx(amplitude, rel=1e-3) + assert par[1] - offset == pytest.approx(0.8, abs=1e-3) + assert par[2] == pytest.approx(1.3, rel=1e-3) + + +def test_rejected_trial_on_limit_does_not_freeze_interior_parameter(minimizer): + # A rejected trial truncated onto the lower limit froze mu at its start. + x = np.linspace(-0.1, 0.1, 11) + mu = np.sqrt(2.0 * np.log(2.0)) + y = gaussian(x, 10.0, mu, 1.0) + model = aare.Gaussian(minimizer=minimizer) + model.FixParameter("A", 10.0) + model.FixParameter("sigma", 1.0) + model.SetParameter("mu", 3.0) + model.SetParLimits("mu", 0.0, 5.0) + result = model.fit(x, y) + + assert result["par"][1] == pytest.approx(mu, abs=1e-3) + assert result["chi2"][0] < 1e-4 + + +# Samples with a zero error are ignored, also when they hold NaN (a masked +# dead frame): a zero weight does not remove a NaN from a sum. + + +@pytest.mark.parametrize("masked", [4, 0, 9]) +def test_masked_nan_sample_is_ignored(minimizer, masked): + # The start estimate of Pol1 uses the first and the last sample. + x = np.arange(10.0) + y = 2.0 + 0.5 * x + y_err = np.ones_like(y) + y[masked] = np.nan + y_err[masked] = 0.0 + result = aare.Pol1(minimizer=minimizer, compute_errors=True).fit(x, y, y_err) + + np.testing.assert_allclose(result["par"], [2.0, 0.5], atol=1e-5) + assert np.all(result["par_err"] > 0.0) + assert result["chi2"][0] == pytest.approx(0.0, abs=1e-6) + + +def test_too_few_unmasked_samples_fail(minimizer): + x = np.arange(10.0) + y = np.full_like(x, np.nan) + y_err = np.zeros_like(x) + y[0], y_err[0] = 2.0, 1.0 # one sample left for two free parameters + result = aare.Pol1(minimizer=minimizer).fit(x, y, y_err) + assert np.all(result["par"] == 0.0) + assert np.all(result["chi2"] == 0.0) + + +def test_masked_samples_in_cube_match_1d_fits(minimizer): + x = np.linspace(-6.0, 6.0, 61) + y = np.stack([gaussian(x, 80.0 + 10.0 * c, 0.8, 1.2) for c in range(4)]) + y = y.reshape(1, 4, x.size) + y_err = np.ones_like(y) + for col, masked in [(1, 10), (2, 0), (3, 34)]: + y[0, col, masked] = np.nan + y_err[0, col, masked] = 0.0 + model = aare.Gaussian(minimizer=minimizer, compute_errors=True) + cube = model.fit(x, y, y_err, n_threads=1) + + for col in range(4): + single = model.fit(x, y[0, col], y_err[0, col]) + np.testing.assert_allclose(cube["par"][0, col], single["par"], atol=1e-12) + np.testing.assert_allclose(cube["par_err"][0, col], single["par_err"], atol=1e-12) + np.testing.assert_allclose( + cube["par"][0, col], [80.0 + 10.0 * col, 0.8, 1.2], rtol=1e-3 + ) + + +def test_parameter_whose_minimum_lies_on_its_limit_reports_no_error(minimizer): + # At a minimum that lies exactly on the limit the gradient is zero. + x = np.array([-1.0, -1.0, 1.0, 1.0]) + model = aare.Pol1(minimizer=minimizer, compute_errors=True) + model.SetParLimits("p0", 0.0, 1.0) + result = model.fit(x, 1.0 + 2.0 * x) + + assert result["par"][0] == pytest.approx(1.0, abs=1e-6) + assert result["par_err"][0] == 0.0 + assert result["par_err"][1] == pytest.approx(0.5, rel=1e-6) + + +@pytest.mark.parametrize("amplitude", [0.1, 1e-4]) +def test_fit_does_not_stop_at_start_without_a_step(minimizer, amplitude): + # The EDM tolerance is absolute in chi2 units: an unweighted fit of small + # data passes it at the start estimate (sigma 1.3617) already. + x = np.linspace(-6.0, 6.0, 61) + y = gaussian(x, amplitude, 0.8, 1.3) + result = aare.Gaussian(minimizer=minimizer).fit(x, y) + assert abs(result["par"][2] - 1.3) < 0.03 # Migrad ends at 1.3205 + + one_call = aare.Gaussian(minimizer=minimizer, max_calls=1).fit(x, y) + assert np.all(one_call["par"] == 0.0) + + +@pytest.mark.parametrize("lo, span", [(8000, 1000), (20000, 500), (60000, 200)]) +def test_polynomial_far_from_zero_keeps_its_precision(minimizer, lo, span): + # Nearly collinear columns 1, x, x^2: VarPro's normal equations lost six + # digits at x = 20000. + x = np.linspace(lo, lo + span, 40) + c = np.array([1.5 - 2e-3 * lo + 3e-7 * lo**2, 2e-3 - 6e-7 * lo, 3e-7]) + y = c[0] + c[1] * x + c[2] * x**2 + result = aare.Pol2(minimizer=minimizer).fit(x, y) + np.testing.assert_allclose(result["par"], c, rtol=1e-9) diff --git a/python/tests/test_InterpolationAPI.py b/python/tests/test_InterpolationAPI.py index e110087a..48686253 100644 --- a/python/tests/test_InterpolationAPI.py +++ b/python/tests/test_InterpolationAPI.py @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 import pytest from aare import Interpolator, ClusterVector, Etai, Cluster diff --git a/python/tests/test_Pedestal.py b/python/tests/test_Pedestal.py index 2c94811e..d2a3ddb3 100644 --- a/python/tests/test_Pedestal.py +++ b/python/tests/test_Pedestal.py @@ -1,7 +1,107 @@ +# SPDX-License-Identifier: MPL-2.0 import numpy as np import pytest -from aare import Pedestal_d, Pedestal_f +from aare import Pedestal, Pedestal_d, Pedestal_f, Pedestal_i16 + + +@pytest.mark.parametrize( + ("dtype", "pedestal_type"), + [(np.float64, Pedestal_d), (np.float32, Pedestal_f), (np.int16, Pedestal_i16)], +) +def test_pedestal_factory(dtype, pedestal_type): + pedestal = Pedestal(rows=2, cols=3, n_samples=4, dtype=dtype) + + assert isinstance(pedestal, pedestal_type) + assert pedestal.rows == 2 + assert pedestal.cols == 3 + assert pedestal.n_samples == 4 + + +def test_pedestal_factory_defaults_to_double(): + pedestal = Pedestal(2, 3) + + assert isinstance(pedestal, Pedestal_d) + assert pedestal.n_samples == 1000 + + +def test_pedestal_factory_rejects_unbound_dtype(): + with pytest.raises(ValueError, match="Unsupported dtype for Pedestal"): + Pedestal(2, 3, dtype=np.int32) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("name", ["rows", "cols", "n_samples"]) +def test_pedestal_factory_rejects_negative_parameters(dtype, name): + parameters = dict(rows=1, cols=1, n_samples=10) + parameters[name] = -1 + with pytest.raises(TypeError): + Pedestal(**parameters, dtype=dtype) + + +@pytest.mark.parametrize("pedestal_type", [Pedestal_d, Pedestal_f, Pedestal_i16]) +@pytest.mark.parametrize( + "args", [(-1, 1), (1, -1), (-1, 1, 10), (1, -1, 10), (1, 1, -1)] +) +def test_pedestal_constructor_rejects_negative_parameters(pedestal_type, args): + with pytest.raises(TypeError): + pedestal_type(*args) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +def test_pedestal_std_stays_finite_after_settling(dtype): + pedestal = Pedestal(1, 1, n_samples=10, dtype=dtype) + pedestal.push(np.array([[16382]], dtype=np.uint16)) + frame = np.array([[16383]], dtype=np.uint16) + for _ in range(201): + pedestal.push(frame) + + noise = pedestal.std().item() + assert np.isfinite(noise) + assert 0 <= noise < 1e-3 + + +@pytest.mark.parametrize( + ("pedestal_type", "expected_dtype"), + [(Pedestal_d, np.float64), (Pedestal_f, np.float32), (Pedestal_i16, np.int16)], +) +def test_double_precision_moments(pedestal_type, expected_dtype): + pedestal = pedestal_type(1, 1, 2) + for value in [30000, 30003, 30002]: + pedestal.push(np.array([[value]], dtype=np.uint16)) + + assert pedestal.mean().dtype == expected_dtype + assert pedestal.std().dtype == expected_dtype + np.testing.assert_array_equal( + pedestal.mean(), np.array([[30001.75]], dtype=expected_dtype) + ) + np.testing.assert_allclose( + pedestal.std(), + np.array([[np.sqrt(1.1875)]], dtype=expected_dtype), + rtol=1e-6, + ) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("method", ["push", "push_with_threshold"]) +@pytest.mark.parametrize("shape", [(2, 2), (3, 2)]) +def test_pedestal_rejects_mismatched_frame_shapes(dtype, method, shape): + pedestal = Pedestal(2, 3, dtype=dtype) + initial = np.full((2, 3), 7, dtype=np.uint16) + pedestal.push(initial) + frame = np.zeros(shape, dtype=np.uint16) + args = ( + (np.full((2, 3), 10, dtype=dtype),) + if method == "push_with_threshold" + else () + ) + + with pytest.raises( + RuntimeError, match="Frame shape does not match pedestal shape" + ): + getattr(pedestal, method)(frame, *args) + + np.testing.assert_array_equal(pedestal.mean(), initial) @pytest.mark.parametrize( @@ -38,3 +138,67 @@ def test_pedestal_exposes_mean_as_read_only_buffer(): np.testing.assert_array_equal(mean, pedestal.view()) assert np.shares_memory(mean, pedestal.view()) assert not mean.flags.writeable + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("method", ["push", "push_with_threshold"]) +def test_pedestal_push_accepts_contiguous_frames(dtype, method): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.arange(6, dtype=np.uint16).reshape(2, 3) + args = ( + (np.full((2, 3), 10, dtype=dtype),) + if method == "push_with_threshold" + else () + ) + + getattr(pedestal, method)(frame, *args) + + np.testing.assert_array_equal(pedestal.mean(), frame) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("input_name", ["push_frame", "threshold_frame", "threshold"]) +@pytest.mark.parametrize("layout", ["transpose", "slice", "reverse"]) +def test_pedestal_rejects_noncontiguous_inputs(dtype, input_name, layout): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.zeros((2, 3), dtype=np.uint16) + threshold = np.full((2, 3), 10, dtype=dtype) + input_dtype = dtype if input_name == "threshold" else np.uint16 + if layout == "transpose": + invalid = np.ones((3, 2), dtype=input_dtype).T + elif layout == "slice": + invalid = np.ones((2, 6), dtype=input_dtype)[:, ::2] + else: + invalid = np.ones((4, 6), dtype=input_dtype)[:2, :3][:, ::-1] + assert not invalid.flags.c_contiguous + + with pytest.raises(ValueError, match="C-contiguous"): + if input_name == "push_frame": + pedestal.push(invalid) + elif input_name == "threshold_frame": + pedestal.push_with_threshold(invalid, threshold) + else: + pedestal.push_with_threshold(frame, invalid) + + np.testing.assert_array_equal(pedestal.mean(), np.zeros((2, 3))) + + +@pytest.mark.parametrize("dtype", [np.float64, np.float32, np.int16]) +@pytest.mark.parametrize("input_name", ["push_frame", "threshold_frame", "threshold"]) +@pytest.mark.parametrize("shape", [(), (6,), (2, 3, 2)]) +def test_pedestal_rejects_inputs_with_wrong_ndim(dtype, input_name, shape): + pedestal = Pedestal(2, 3, dtype=dtype) + frame = np.ones((2, 3), dtype=np.uint16) + threshold = np.full((2, 3), 10, dtype=dtype) + input_dtype = dtype if input_name == "threshold" else np.uint16 + invalid = np.ones(shape, dtype=input_dtype) + + with pytest.raises(ValueError): + if input_name == "push_frame": + pedestal.push(invalid) + elif input_name == "threshold_frame": + pedestal.push_with_threshold(invalid, threshold) + else: + pedestal.push_with_threshold(frame, invalid) + + np.testing.assert_array_equal(pedestal.mean(), np.zeros((2, 3))) diff --git a/python/tests/test_ROI.py b/python/tests/test_ROI.py new file mode 100644 index 00000000..3a117016 --- /dev/null +++ b/python/tests/test_ROI.py @@ -0,0 +1,22 @@ +# SPDX-License-Identifier: MPL-2.0 +import pytest + +import numpy as np + +from aare import ROI + + +def test_roi_slice(): + """test the ROI slice method""" + roi = ROI(0, 4, 2, 4) + + array = np.array([[1, 2, 3, 4], + [5, 6, 7, 8], + [9, 10, 11, 12], + [13, 14, 15, 16]]) + + sliced_array = array[roi.slice()] + + assert sliced_array.shape == (2, 4) + assert np.array_equal(sliced_array, np.array([[9, 10, 11, 12], + [13, 14, 15, 16]])) \ No newline at end of file diff --git a/python/tests/test_RawFile.py b/python/tests/test_RawFile.py index ca844a0e..1fb6c3f7 100644 --- a/python/tests/test_RawFile.py +++ b/python/tests/test_RawFile.py @@ -1,8 +1,299 @@ # SPDX-License-Identifier: MPL-2.0 import pytest -from aare import RawFile +import json +from aare import File, RawFile, RawSubFile, DetectorType, ROI, UDPPortPosition import numpy as np + +@pytest.fixture +def small_raw_file(tmp_path): + master_path = tmp_path / "run_master_0.json" + master_path.write_text(json.dumps({ + "Version": 7.2, + "Detector Type": "Jungfrau", + "Timing Mode": "auto", + "Geometry": {"x": 1, "y": 1}, + "Image Size in bytes": 12, + "Pixels": {"x": 3, "y": 2}, + "Max Frames Per File": 1, + "Total Frames": 2, + "Frames in File": 2, + "Frame Padding": 1, + "Frame Discard Policy": "nodiscard", + })) + for index in range(2): + data_path = tmp_path / f"run_d0_f{index}_0.raw" + # A detector header followed by six uint16 pixels. + data_path.write_bytes(bytes(112) + np.arange(6, dtype=np.uint16).tobytes()) + return master_path + + +@pytest.mark.parametrize("reader_type", [RawFile, File]) +@pytest.mark.parametrize("legacy_master", [False, True]) +@pytest.mark.parametrize("padding, policy, supported", [ + (0, "nodiscard", False), + (0, "discard", False), + (0, "discardpartial", True), + (1, "nodiscard", True), + (1, "discard", True), + (1, "discardpartial", True), +]) +def test_raw_frame_policy(small_raw_file, reader_type, legacy_master, + padding, policy, supported): + master_path = small_raw_file + if legacy_master: + master_path = master_path.with_suffix(".raw") + master_path.write_text( + "Version : 6.4\n" + "Detector Type : Jungfrau\n" + "Timing Mode : auto\n" + "Geometry : [1, 1]\n" + "Image Size : 12\n" + "Pixels : [3, 2]\n" + "Dynamic Range : 16\n" + "Max Frames Per File : 1\n" + "Total Frames : 2\n" + "Frames in File : 2\n" + f"Frame Padding : {padding}\n" + f"Frame Discard Policy : {policy}\n" + ) + else: + metadata = json.loads(master_path.read_text()) + metadata["Frame Padding"] = padding + metadata["Frame Discard Policy"] = policy + master_path.write_text(json.dumps(metadata)) + + if supported: + reader = reader_type(master_path) + assert reader.total_frames == 2 + frame = reader.read_frame() + if reader_type is RawFile: + _, frame = frame + np.testing.assert_array_equal(frame, np.arange(6).reshape(2, 3)) + else: + message = "requires frame padding or discardpartial" + with pytest.raises(RuntimeError, match=message) as error: + reader_type(master_path) + assert str(master_path) in str(error.value) + + for index in range(2): + (master_path.parent / f"run_d0_f{index}_0.raw").unlink() + with pytest.raises(RuntimeError, match=message) as error: + reader_type(master_path) + assert str(master_path) in str(error.value) + + +@pytest.mark.parametrize("method, args, kwargs", [ + ("read_frame", (), {}), + ("read_n", (1,), {}), + ("read_roi", (0,), {}), + ("read_rois", (), {}), + ("read_n_with_roi", (1,), {"roi_index": 0}), + ("frame_number", (2,), {}), +]) +def test_raw_read_error_context(small_raw_file, method, args, kwargs): + reader = RawFile(small_raw_file) + reader.seek(2) + with pytest.raises(RuntimeError) as error: + getattr(reader, method)(*args, **kwargs) + assert "frame index 2" in str(error.value) + assert str(small_raw_file) in str(error.value) + assert ".raw" not in str(error.value) + if method == "read_frame": + assert "file contains 2 frames (indices are zero-based)" in str(error.value) + + +@pytest.mark.parametrize("frame_index", [2, 17]) +def test_generic_raw_read_error_context(small_raw_file, frame_index): + reader = File(small_raw_file) + with pytest.raises(RuntimeError) as error: + reader.read_frame(frame_index) + assert f"frame index {frame_index}" in str(error.value) + assert str(small_raw_file) in str(error.value) + + +@pytest.mark.parametrize("raw_subfile", [False, True]) +def test_truncated_raw_read_error_context(small_raw_file, raw_subfile): + data_path = small_raw_file.parent / "run_d0_f1_0.raw" + if raw_subfile: + reader = RawSubFile( + small_raw_file.parent / "run_d0_f0_0.raw", + DetectorType.Jungfrau, 2, 3, 16, + ) + else: + reader = RawFile(small_raw_file) + data_path.write_bytes(bytes(112)) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + reader.read_frame() + assert "frame index 1" in str(error.value) + assert str(data_path) in str(error.value) + assert "End of file" in str(error.value) + assert "master" not in str(error.value) + assert "RawSubFile.cpp" not in str(error.value) + + +@pytest.mark.parametrize("method, args", [ + ("read_frame", ()), + ("read_n", (1,)), + ("read_roi", (0,)), + ("frame_number", (1,)), +]) +def test_short_raw_subfile_sets_read_bounds(small_raw_file, method, args): + (small_raw_file.parent / "run_d0_f1_0.raw").unlink() + reader = RawFile(small_raw_file) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + getattr(reader, method)(*args) + assert reader.total_frames == 1 + assert "frame index 1" in str(error.value) + assert str(small_raw_file) in str(error.value) + assert ".raw" not in str(error.value) + + +@pytest.mark.parametrize("counts, master_count, warn", [ + ([2], 2, False), + ([2, 2], 2, False), + ([2, 2], 0, True), + ([2, 2], 1, True), + ([2, 2], 5, True), + ([1, 2], 2, True), + ([2, 1], 2, True), + ([1, 2], 1, True), + ([0, 2], 2, True), + ([0, 0], 0, False), +]) +def test_raw_subfile_frame_counts(small_raw_file, capfd, counts, master_count, warn): + metadata = json.loads(small_raw_file.read_text()) + metadata["Geometry"]["y"] = len(counts) + metadata["Frames in File"] = master_count + metadata["Total Frames"] = 1000 + small_raw_file.write_text(json.dumps(metadata)) + data = (small_raw_file.parent / "run_d0_f0_0.raw").read_bytes() + for module, count in enumerate(counts): + for index in range(2): + path = small_raw_file.parent / f"run_d{module}_f{index}_0.raw" + if index < count: + path.write_bytes(data) + elif index == 0: + path.write_bytes(b"") + else: + path.unlink(missing_ok=True) + + reader = RawFile(small_raw_file) + warning = capfd.readouterr().err + if warn: + assert warning.count("WARNING") == 1 + assert str(small_raw_file) in warning + else: + assert "WARNING" not in warning + + assert reader.total_frames == min(counts) + assert len(reader) == min(counts) + assert reader.master.frames_in_file == master_count + if min(counts): + _, frames = reader.read() + assert frames.shape == (min(counts), 2 * len(counts), 3) + reader.seek(0) + _, frames = reader.read_n(10) + assert len(frames) == min(counts) + assert reader.frame_number(min(counts) - 1) == 0 + with pytest.raises(RuntimeError): + reader.frame_number(min(counts)) + reader.seek(min(counts)) + with pytest.raises(RuntimeError): + reader.read_frame() + assert "WARNING" not in capfd.readouterr().err + + +@pytest.mark.parametrize("short_module", [0, 1]) +def test_raw_frame_count_across_rois(small_raw_file, short_module): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({ + "Version": 8.0, + "Geometry": {"x": 1, "y": 2}, + "Image Size": 12, + "Receiver Rois": [ + {"xmin": 0, "xmax": 2, "ymin": 0, "ymax": 1}, + {"xmin": 0, "xmax": 2, "ymin": 2, "ymax": 3}, + ], + }) + small_raw_file.write_text(json.dumps(metadata)) + for index in range(2): + data = (small_raw_file.parent / f"run_d0_f{index}_0.raw").read_bytes() + (small_raw_file.parent / f"run_d1_f{index}_0.raw").write_bytes(data) + (small_raw_file.parent / f"run_d{short_module}_f1_0.raw").unlink() + + reader = RawFile(small_raw_file) + assert reader.total_frames == 1 + _, rois = reader.read_rois() + assert len(rois) == 2 + assert all(roi.shape == (2, 3) for roi in rois) + for roi_index in range(2): + reader.seek(0) + _, frames = reader.read_n_with_roi(10, roi_index=roi_index) + assert frames.shape == (1, 2, 3) + with pytest.raises(RuntimeError): + reader.read_roi(roi_index) + + +@pytest.mark.parametrize("disabled_port", [0, 1]) +def test_raw_frame_count_with_disabled_udp_port(small_raw_file, disabled_port): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({ + "Geometry": {"x": 1, "y": 2}, + "Frames in File": 0, + "Number of UDP Interfaces": 2, + "UDP Ports Type": ["bottom", "top"], + "UDP Ports Disabled": [disabled_port], + }) + small_raw_file.write_text(json.dumps(metadata)) + data = (small_raw_file.parent / "run_d0_f0_0.raw").read_bytes() + for index in range(2): + (small_raw_file.parent / f"run_d0_f{index}_0.raw").unlink() + path = small_raw_file.parent / f"run_d{1 - disabled_port}_f{index}_0.raw" + path.write_bytes(data) + + reader = RawFile(small_raw_file) + assert reader.total_frames == 2 + assert reader.master.frames_in_file == 0 + assert reader.frame_number(1) == 0 + _, frames = reader.read() + assert frames.shape == (2, 2, 3) + + +def test_raw_frame_count_without_selected_subfiles(small_raw_file): + metadata = json.loads(small_raw_file.read_text()) + metadata.update({"Version": 8.0, "Image Size": 12, "Receiver Rois": []}) + small_raw_file.write_text(json.dumps(metadata)) + with pytest.raises(RuntimeError, match="No raw subfiles selected"): + RawFile(small_raw_file) + + +@pytest.mark.parametrize("lagging_module", [0, 1]) +def test_raw_synchronization_error_identifies_subfile( + small_raw_file, lagging_module +): + metadata = json.loads(small_raw_file.read_text()) + metadata["Geometry"]["y"] = 2 + small_raw_file.write_text(json.dumps(metadata)) + for index in range(2): + data = (small_raw_file.parent / f"run_d0_f{index}_0.raw").read_bytes() + (small_raw_file.parent / f"run_d1_f{index}_0.raw").write_bytes(data) + + ahead_path = small_raw_file.parent / f"run_d{1 - lagging_module}_f1_0.raw" + with ahead_path.open("r+b") as output: + output.write(np.array([1], dtype=np.uint64).tobytes()) + + reader = RawFile(small_raw_file) + reader.seek(1) + with pytest.raises(RuntimeError) as error: + reader.read_frame() + assert "frame index 1" in str(error.value) + assert "synchroniz" in str(error.value) + lagging_path = small_raw_file.parent / f"run_d{lagging_module}_f1_0.raw" + assert str(lagging_path) in str(error.value) + @pytest.mark.withdata def test_read_rawfile_with_roi_spanning_over_one_module(test_data_path): @@ -110,3 +401,41 @@ def test_read_rawfile_eiger_and_compare_to_numpy(test_data_path): header, image1 = f.read_frame() assert (image == image1).all() + + +@pytest.mark.withdata +def test_read_eiger_udp_port_disabled(test_data_path): + with RawFile(test_data_path / "raw/eiger/one_udp_port_disabled_master_0.json") as f: + _, frame = f.read_rois() + + assert(len(frame) == 2) + assert frame[0].shape == (256, 512) + assert frame[1].shape == (256, 1024) + assert f.master.udp_port_types == [UDPPortPosition.LEFT, UDPPortPosition.RIGHT] + rois = f.master.rois + assert len(rois) == 2 + assert rois[0] == ROI(512, 1024, 0, 256) + assert rois[1] == ROI(0, 1024, 256, 512) + + with RawFile(test_data_path / "raw/eiger/quad_eiger_disabled_bottom_port_master_0.json") as f: + _, frame = f.read_frame() + + assert frame.shape == (256, 512) + assert(f.master.disabled_udp_ports == [1]) + assert f.master.udp_port_types == [UDPPortPosition.TOP, UDPPortPosition.BOTTOM] + rois = f.master.rois + assert len(rois) == 1 + assert rois[0] == ROI(0, 512, 256, 512) + + with RawFile(test_data_path / "raw/eiger/2_modules_eiger_disabled_udp_port_master_0.json") as f: + _, frame = f.read_rois() + + assert(len(frame) == 2) + assert frame[0].shape == (512, 512) + assert frame[1].shape == (512, 512) + assert (f.master.disabled_udp_ports == [1, 3, 5, 7]) + assert f.master.udp_port_types == [UDPPortPosition.LEFT, UDPPortPosition.RIGHT] + rois = f.master.rois + assert len(rois) == 2 + assert rois[0] == ROI(0, 512, 0, 512) + assert rois[1] == ROI(1024, 1536, 0, 512) diff --git a/python/tests/test_RawMasterFile.py b/python/tests/test_RawMasterFile.py index 6ea9ff74..94669d77 100644 --- a/python/tests/test_RawMasterFile.py +++ b/python/tests/test_RawMasterFile.py @@ -1,7 +1,10 @@ +# SPDX-License-Identifier: MPL-2.0 +import json + import pytest -from aare import RawMasterFile, ReadoutMode, DetectorType +from aare import RawMasterFile, ReadoutMode, DetectorType, FrameDiscardPolicy @pytest.mark.withdata @@ -11,4 +14,37 @@ def test_read_rawfile_quad_eiger_and_compare_to_numpy(test_data_path): f = RawMasterFile(file_name) assert(f.reading_mode == ReadoutMode.UNKNOWN) - assert(f.detector_type == DetectorType.Jungfrau) \ No newline at end of file + assert(f.detector_type == DetectorType.Jungfrau) + + +@pytest.mark.parametrize( + "policy, expected_policy", + [ + ("nodiscard", FrameDiscardPolicy.NoDiscard), + ("discard", FrameDiscardPolicy.Discard), + ("discardpartial", FrameDiscardPolicy.DiscardPartial), + ], +) +def test_raw_master_file_context_manager(tmp_path, policy, expected_policy): + file_name = tmp_path / "run_master_0.json" + file_name.write_text(json.dumps({ + "Version": 7.2, + "Detector Type": "Jungfrau", + "Timing Mode": "auto", + "Geometry": {"x": 1, "y": 1}, + "Image Size in bytes": 12, + "Pixels": {"x": 3, "y": 2}, + "Max Frames Per File": 1, + "Total Frames": 2, + "Frames in File": 2, + "Frame Padding": 1, + "Frame Discard Policy": policy, + })) + + + with RawMasterFile(file_name) as context_file: + assert context_file.reading_mode == ReadoutMode.UNKNOWN + assert context_file.detector_type == DetectorType.Jungfrau + frame_discard_policy = context_file.frame_discard_policy + assert isinstance(frame_discard_policy, FrameDiscardPolicy) + assert frame_discard_policy == expected_policy diff --git a/python/tests/test_Transformation.py b/python/tests/test_Transformation.py index 2b381085..0d91c10a 100644 --- a/python/tests/test_Transformation.py +++ b/python/tests/test_Transformation.py @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 import pytest import numpy as np from aare import CtbRawFile, transform diff --git a/python/tests/test_interpolation_simulated.py b/python/tests/test_interpolation_simulated.py index 5a3cdf3c..57a6ab63 100644 --- a/python/tests/test_interpolation_simulated.py +++ b/python/tests/test_interpolation_simulated.py @@ -1,3 +1,4 @@ +# SPDX-License-Identifier: MPL-2.0 #test script to test interpolation on simulated data diff --git a/python/tests/test_make_view.py b/python/tests/test_make_view.py new file mode 100644 index 00000000..2df947ca --- /dev/null +++ b/python/tests/test_make_view.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: MPL-2.0 +"""Probe make_view in np_helper.hpp through the bindings in _aare.testing""" +import numpy as np +import pytest + +from aare._aare import testing + + + + +def test_make_view_roundtrip(): + """Send an array to C++ and check that we get the same array back""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + out = testing.view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + + #Sanity check to make sure we got a copy an not just the same thing back + arr[3,2] = 31655 + assert arr[3,2] != out[3,2] + +def test_make_view_const_roundtrip(): + """Const works if the array is not writeable""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr.flags.writeable = False + out = testing.const_view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + +def test_make_view_const_roundtrip_writable(): + """Const works also if the array is writeable""" + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + out = testing.const_view_roundtrip_2d(arr) + np.testing.assert_array_equal(out, arr) + +def test_make_view_fill(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + testing.view_fill_2d(arr, 7) + assert (arr == 7).all() + +def test_make_view_fill_rejects_nonwriteable(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr.flags.writeable = False + with pytest.raises(ValueError, match="not writeable"): + testing.view_fill_2d(arr, 7) + +def test_make_view_2d_rejects_1d(): + arr = np.arange(30, dtype=np.uint16) + with pytest.raises(ValueError, match="Expected 2D array, got 1D"): + testing.view_roundtrip_2d(arr) + +def test_make_view_2d_rejects_3d(): + arr = np.arange(30, dtype=np.uint16).reshape(3, 5, 2) + with pytest.raises(ValueError, match="Expected 2D array, got 3D"): + testing.view_roundtrip_2d(arr) + +def test_make_view_rejects_noncontiguous(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr = arr[::2, ::3] # non-contiguous + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.view_roundtrip_2d(arr) + +def test_make_view_rejects_noncontiguous_const(): + arr = np.arange(30, dtype=np.float64).reshape(5, 6) + arr = arr[::2, ::3] # non-contiguous + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.const_view_roundtrip_2d(arr) + +def test_make_view_rejects_fortran_order(): + arr = np.asfortranarray(np.arange(30, dtype=np.float64).reshape(5, 6)) + assert not arr.flags.c_contiguous + with pytest.raises(ValueError, match="not C-contiguous"): + testing.view_roundtrip_2d(arr) + diff --git a/src/Chi2.hpp b/src/Chi2.hpp deleted file mode 100644 index 2aeb0b93..00000000 --- a/src/Chi2.hpp +++ /dev/null @@ -1,140 +0,0 @@ -// SPDX-License-Identifier: MPL-2.0 -#pragma once -#include "aare/Models.hpp" -#include "aare/NDView.hpp" -#include -#include -#include -#include -#include - -namespace aare { - -namespace func { - -/** - * @brief Generic chi-squared FCN with analytic gradient for a 1D model. - * - * @tparam Model A model struct that satisfies: - * - static constexpr std::size_t npar; - * - static double eval(double x, const std::vector& par); - * - static void eval_and_grad(double x, const std::vector& par, - * double& f, std::array& g); - * - static bool is_valid(const std::vector& par); - * - * Gradient: - * d(chi2)/dp_k = -2 * sum_i w_i * (y_i - f_i) * df_i/dp_k - * - * where w_i = 1/sigma_i^2 (weighted) or 1 (unweighted). - * - * By providing analytic gradients we avoid 2*npar extra function evaluations - * per Minuit step that would otherwise be spent on finite differences. - * - * @throws std::invalid_argument if par.size() != Model::npar. - * - * Invalid model parameters do not throw; they return a large penalty - * (and a zero gradient fallback) so the minimizer can remain in control. - */ -template -class Chi2Model1DGrad : public ROOT::Minuit2::FCNGradientBase { - public: - Chi2Model1DGrad(NDView x, NDView y) - : x_(x), y_(y), s_(), weighted_(false) {} - - Chi2Model1DGrad(NDView x, NDView y, - NDView y_err) - : x_(x), y_(y), s_(y_err), weighted_(true) {} - - ~Chi2Model1DGrad() override = default; - - double operator()(const std::vector &par) const override { - if (par.size() != Model::npar) { - throw std::invalid_argument( - "Chi2Model1DGrad: wrong parameter vector size."); - } - - if (!Model::is_valid(par)) - return 1e20; - - double chi2 = 0.0; - - if (weighted_) { - for (ssize_t i = 0; i < x_.size(); ++i) { - const double si = s_[i]; - if (si == 0.0) - continue; - - const double f_i = Model::eval(x_[i], par); - const double r_i = y_[i] - f_i; - chi2 += (r_i * r_i) / (si * si); - } - } else { - for (ssize_t i = 0; i < x_.size(); ++i) { - const double f_i = Model::eval(x_[i], par); - const double r_i = y_[i] - f_i; - chi2 += r_i * r_i; - } - } - - return chi2; - } - - std::vector - Gradient(const std::vector &par) const override { - if (par.size() != Model::npar) { - throw std::invalid_argument( - "Chi2Model1DGrad: wrong parameter vector size."); - } - - std::vector grad(Model::npar, 0.0); - if (!Model::is_valid(par)) - return grad; - - std::array df{}; - double f_i = 0.0; - - if (weighted_) { - for (ssize_t i = 0; i < x_.size(); ++i) { - const double si = s_[i]; - if (si == 0.0) - continue; - - Model::eval_and_grad(x_[i], par, f_i, df); - - const double r_i = y_[i] - f_i; - const double c = -2.0 * r_i / (si * si); - - for (std::size_t k = 0; k < Model::npar; ++k) { - grad[k] += c * df[k]; - } - } - } else { - for (ssize_t i = 0; i < x_.size(); ++i) { - Model::eval_and_grad(x_[i], par, f_i, df); - - const double r_i = y_[i] - f_i; - const double c = -2.0 * r_i; - - for (std::size_t k = 0; k < Model::npar; ++k) { - grad[k] += c * df[k]; - } - } - } - - return grad; - } - - /** @brief Error definition: 1.0 for chi-squared (delta_chi2 = 1 -> - * 1-sigma). */ - double Up() const override { return 1.0; } - - private: - NDView x_; - NDView y_; - NDView s_; - bool weighted_; -}; - -} // namespace func - -} // namespace aare \ No newline at end of file diff --git a/src/Cluster.test.cpp b/src/Cluster.test.cpp index 4919b859..ddcd2c4b 100644 --- a/src/Cluster.test.cpp +++ b/src/Cluster.test.cpp @@ -24,8 +24,8 @@ TEST_CASE("Test sum of Cluster", "[cluster]") { using ClusterTypes = std::variant, Cluster, Cluster, Cluster>; -using ClusterTypesLargerThan2x2 = - std::variant, Cluster, Cluster>; +using ReducibleTo3x3ClusterTypes = + std::variant, Cluster, Cluster>; TEST_CASE("Test reduce to 2x2 Cluster", "[cluster]") { auto [cluster, expected_reduced_cluster] = GENERATE( @@ -68,14 +68,15 @@ TEST_CASE("Test reduce to 2x2 Cluster", "[cluster]") { TEST_CASE("Test reduce to 3x3 Cluster", "[cluster]") { auto [cluster, expected_reduced_cluster] = GENERATE( - std::make_tuple(ClusterTypesLargerThan2x2{Cluster{ - 5, 5, {1, 1, 1, 1, 3, 1, 1, 1, 1}}}, - Cluster{5, 5, {1, 1, 1, 1, 3, 1, 1, 1, 1}}), std::make_tuple( - ClusterTypesLargerThan2x2{Cluster{ - 5, 5, {2, 2, 1, 1, 2, 2, 1, 1, 1, 1, 3, 1, 1, 1, 1, 1}}}, - Cluster{5, 5, {2, 1, 1, 1, 3, 1, 1, 1, 1}}), - std::make_tuple(ClusterTypesLargerThan2x2{Cluster{ + ReducibleTo3x3ClusterTypes{Cluster{ + 5, 5, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}}}, + Cluster{5, 5, {3, 4, 5, 6, 7, 8, 9, 10, 11}}), + std::make_tuple( + ReducibleTo3x3ClusterTypes{Cluster{ + 5, 5, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}}}, + Cluster{5, 5, {1, 2, 3, 5, 6, 7, 9, 10, 11}}), + std::make_tuple(ReducibleTo3x3ClusterTypes{Cluster{ 5, 5, {1, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 3, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1}}}, Cluster{5, 5, {2, 1, 1, 2, 3, 1, 2, 1, 1}})); @@ -89,4 +90,4 @@ TEST_CASE("Test reduce to 3x3 Cluster", "[cluster]") { CHECK(std::equal(reduced_cluster.data.begin(), reduced_cluster.data.begin() + 9, expected_reduced_cluster.data.begin())); -} \ No newline at end of file +} diff --git a/src/ClusterFile.test.cpp b/src/ClusterFile.test.cpp index 15caf6bb..6c009ba1 100644 --- a/src/ClusterFile.test.cpp +++ b/src/ClusterFile.test.cpp @@ -5,12 +5,74 @@ #include "aare/defs.hpp" #include #include +#include +#include +#include #include +#include +#include +#include using aare::Cluster; using aare::ClusterFile; using aare::ClusterVector; +namespace { + +class TemporaryClusterFile { + public: + TemporaryClusterFile() { + const auto unique = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_path = std::filesystem::temp_directory_path() / + ("aare-cluster-file-" + std::to_string(unique) + ".clust"); + } + + TemporaryClusterFile(const TemporaryClusterFile &) = delete; + TemporaryClusterFile &operator=(const TemporaryClusterFile &) = delete; + + ~TemporaryClusterFile() { + std::error_code error; + std::filesystem::remove(m_path, error); + } + + const std::filesystem::path &path() const { return m_path; } + + private: + std::filesystem::path m_path; +}; + +using TestCluster = Cluster; + +ClusterVector make_test_frame(int32_t frame_number, + double offset) { + ClusterVector clusters(2, frame_number); + clusters.push_back( + TestCluster{5, + 6, + {offset, offset + 1, offset + 2, offset + 3, offset + 4, + offset + 5, offset + 6, offset + 7, offset + 8}}); + clusters.push_back(TestCluster{7, + 8, + {offset + 9, offset + 10, offset + 11, + offset + 12, offset + 13, offset + 14, + offset + 15, offset + 16, offset + 17}}); + return clusters; +} + +void check_frame(const ClusterVector &actual, + const ClusterVector &expected) { + REQUIRE(actual.frame_number() == expected.frame_number()); + REQUIRE(actual.size() == expected.size()); + for (size_t i = 0; i < expected.size(); ++i) { + CHECK(actual[i].x == expected[i].x); + CHECK(actual[i].y == expected[i].y); + CHECK(actual[i].data == expected[i].data); + } +} + +} // namespace + TEST_CASE("Read one frame from a cluster file", "[.with-data]") { // We know that the frame has 97 clusters auto fpath = test_data_path() / "clust" / "single_frame_97_clustrers.clust"; @@ -20,12 +82,14 @@ TEST_CASE("Read one frame from a cluster file", "[.with-data]") { CHECK(f.estimate_n_clusters() == 97); CHECK(f.tell() == 0); auto clusters = f.read_frame(); - CHECK(clusters.size() == 97); - CHECK(clusters.frame_number() == 135); - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + REQUIRE(clusters); + CHECK(clusters->size() == 97); + CHECK(clusters->frame_number() == 135); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } @@ -42,22 +106,24 @@ TEST_CASE("Read one frame using ROI", "[.with-data]") { roi.ymax = 249; f.set_roi(roi); auto clusters = f.read_frame(); - REQUIRE(clusters.size() == 49); - REQUIRE(clusters.frame_number() == 135); + REQUIRE(clusters); + REQUIRE(clusters->size() == 49); + REQUIRE(clusters->frame_number() == 135); // Check that all clusters are within the ROI - for (size_t i = 0; i < clusters.size(); i++) { - auto c = clusters[i]; + for (size_t i = 0; i < clusters->size(); i++) { + auto c = (*clusters)[i]; REQUIRE(c.x >= roi.xmin); REQUIRE(c.x <= roi.xmax); REQUIRE(c.y >= roi.ymin); REQUIRE(c.y <= roi.ymax); } - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } @@ -284,70 +350,551 @@ TEST_CASE("Read cluster from multiple frame file", "[.with-data]") { } } -TEST_CASE("Write cluster with potential padding", - "[.with-data][.ClusterFile]") { +TEST_CASE("ClusterFile flushes a frame when the writer is destroyed", + "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); - using ClusterType = Cluster; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } - REQUIRE(std::filesystem::exists(test_data_path() / "clust")); + const auto expected_file_size = sizeof(int32_t) + sizeof(uint32_t) + + expected.size() * sizeof(TestCluster); + CHECK(std::filesystem::file_size(file.path()) == expected_file_size); - auto fpath = test_data_path() / "clust" / "single_frame_2_clusters.clust"; - - ClusterFile file(fpath, 1000, "w"); - - ClusterVector clustervec(2); - uint16_t coordinate = 5; - clustervec.push_back(ClusterType{ - coordinate, coordinate, {0., 0., 0., 0., 0., 0., 0., 0., 0.}}); - clustervec.push_back(ClusterType{ - coordinate, coordinate, {0., 0., 0., 0., 0., 0., 0., 0., 0.}}); - - file.write_frame(clustervec); - - file.close(); - - file.open("r"); - - auto read_cluster_vector = file.read_frame(); - - CHECK(read_cluster_vector.size() == 2); - CHECK(read_cluster_vector.frame_number() == 0); - - CHECK(read_cluster_vector[0].x == clustervec[0].x); - CHECK(read_cluster_vector[0].y == clustervec[0].y); - CHECK(std::equal( - clustervec[0].data.begin(), clustervec[0].data.end(), - read_cluster_vector[0].data.begin(), [](double a, double b) { - return std::abs(a - b) < std::numeric_limits::epsilon(); - })); - - CHECK(read_cluster_vector[1].x == clustervec[1].x); - CHECK(read_cluster_vector[1].y == clustervec[1].y); - CHECK(std::equal( - clustervec[1].data.begin(), clustervec[1].data.end(), - read_cluster_vector[1].data.begin(), [](double a, double b) { - return std::abs(a - b) < std::numeric_limits::epsilon(); - })); + ClusterFile reader(file.path()); + auto actual = reader.read_frame(); + REQUIRE(actual); + check_frame(*actual, expected); } -TEST_CASE("Read frame and modify cluster data", "[.with-data][.ClusterFile]") { +TEST_CASE("ClusterFile appends frames", "[ClusterFile]") { + TemporaryClusterFile file; + auto first_expected = make_test_frame(42, 0.0); + auto second_expected = make_test_frame(43, 100.0); + + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first_expected); + } + { + ClusterFile writer(file.path(), 1000, "a"); + writer.write_frame(second_expected); + } + + ClusterFile reader(file.path()); + auto first_actual = reader.read_frame(); + auto second_actual = reader.read_frame(); + REQUIRE(first_actual); + REQUIRE(second_actual); + check_frame(*first_actual, first_expected); + check_frame(*second_actual, second_expected); +} + +TEST_CASE("ClusterFile reads frames into an existing cluster vector", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first_expected = make_test_frame(42, 0.0); + auto second_expected = make_test_frame(43, 100.0); + + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first_expected); + writer.write_frame(second_expected); + } + + ClusterFile reader(file.path()); + ClusterVector actual(2); + const auto initial_data = actual.data(); + + REQUIRE(reader.read_frame(actual)); + CHECK(actual.data() == initial_data); + check_frame(actual, first_expected); + + REQUIRE(reader.read_frame(actual)); + CHECK(actual.data() == initial_data); + check_frame(actual, second_expected); + + CHECK_FALSE(reader.read_frame(actual)); + check_frame(actual, second_expected); +} + +TEST_CASE("ClusterFile reports a clean end of file", "[ClusterFile]") { + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + } + + ClusterFile reader(file.path()); + ClusterVector clusters; + CHECK_FALSE(reader.read_frame(clusters)); + CHECK(clusters.empty()); + CHECK_FALSE(reader.read_frame().has_value()); +} + +TEST_CASE("ClusterFile reports complete frames with no output clusters", + "[ClusterFile]") { + TemporaryClusterFile file; + ClusterVector empty_frame(0, 41); + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(empty_frame); + writer.write_frame(expected); + } + + ClusterFile reader(file.path()); + ClusterVector clusters; + auto empty_actual = reader.read_frame(); + REQUIRE(empty_actual); + CHECK(empty_actual->empty()); + CHECK(empty_actual->frame_number() == 41); + + aare::ROI roi; + roi.xmin = 100; + roi.xmax = 200; + roi.ymin = 100; + roi.ymax = 200; + reader.set_roi(roi); + + REQUIRE(reader.read_frame(clusters)); + CHECK(clusters.empty()); + CHECK(clusters.frame_number() == expected.frame_number()); + CHECK_FALSE(reader.read_frame(clusters)); +} + +TEST_CASE("ClusterFile rejects incomplete frames", "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + + SECTION("incomplete frame number") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) - 1); + } + SECTION("incomplete cluster count") { + std::filesystem::resize_file(file.path(), + sizeof(int32_t) + sizeof(uint32_t) - 1); + } + SECTION("incomplete cluster data") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + } + + { + ClusterFile reader(file.path()); + ClusterVector clusters; + CHECK_THROWS_AS(reader.read_frame(clusters), std::runtime_error); + } + { + ClusterFile reader(file.path()); + CHECK_THROWS_AS(reader.read_frame(), std::runtime_error); + } +} + +TEST_CASE("ClusterFile close is idempotent", "[ClusterFile]") { + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + writer.close(); + + CHECK_NOTHROW(writer.close()); + CHECK_THROWS_AS(writer.tell(), std::runtime_error); + CHECK_THROWS_AS(writer.write_frame(expected), std::runtime_error); + + ClusterFile reader(file.path()); + auto actual = reader.read_frame(); + REQUIRE(actual); + check_frame(*actual, expected); +} + +TEST_CASE("ClusterFile chunk reads reject incomplete frames", "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + CAPTURE(use_roi); + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + + SECTION("incomplete frame number") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) - 1); + } + SECTION("missing cluster count") { + std::filesystem::resize_file(file.path(), sizeof(int32_t)); + } + SECTION("incomplete cluster count") { + std::filesystem::resize_file(file.path(), + sizeof(int32_t) + sizeof(uint32_t) - 1); + } + SECTION("missing cluster record") { + std::filesystem::resize_file(file.path(), sizeof(int32_t) + + sizeof(uint32_t) + + sizeof(TestCluster)); + } + SECTION("incomplete cluster record") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + } + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + CHECK_THROWS_AS(reader.read_clusters(10), std::runtime_error); +} + +TEST_CASE("ClusterFile chunk reads reject truncated leftover clusters", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + const auto requested = GENERATE(1, 10); + CAPTURE(use_roi, requested); + TemporaryClusterFile file; + auto expected = make_test_frame(42, 0.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(expected); + } + std::filesystem::resize_file(file.path(), + std::filesystem::file_size(file.path()) - 1); + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + auto first = reader.read_clusters(1); + REQUIRE(first.size() == 1); + CHECK(first[0].data == expected[0].data); + CHECK_THROWS_AS(reader.read_clusters(requested), std::runtime_error); +} + +TEST_CASE("ClusterFile chunk reads preserve valid frame traversal", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + CAPTURE(use_roi); + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + ClusterVector empty(0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(empty); + writer.write_frame(first); + writer.write_frame(empty); + writer.write_frame(second); + writer.write_frame(empty); + } + + ClusterFile reader(file.path()); + if (use_roi) { + reader.set_roi(aare::ROI{0, 10, 0, 10}); + } + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.tell() == 0); + + auto initial = reader.read_clusters(1); + REQUIRE(initial.size() == 1); + CHECK(initial[0].data == first[0].data); + const auto position = reader.tell(); + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.tell() == position); + CHECK_THROWS_AS(reader.read_frame(), std::runtime_error); + + auto crossing = reader.read_clusters(2); + REQUIRE(crossing.size() == 2); + CHECK(crossing[0].data == first[1].data); + CHECK(crossing[1].data == second[0].data); + + auto final = reader.read_clusters(10); + REQUIRE(final.size() == 1); + CHECK(final[0].data == second[1].data); + CHECK(static_cast(reader.tell()) == + std::filesystem::file_size(file.path())); + CHECK(reader.read_clusters(10).empty()); + CHECK_FALSE(reader.read_frame()); +} + +TEST_CASE("ClusterFile frames preserve empty frames and reuse storage", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(-42, 0.0); + auto second = make_test_frame(105, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(ClusterVector(0, -43)); + writer.write_frame(first); + writer.write_frame(ClusterVector(0, 0)); + writer.write_frame(second); + } + + ClusterFile reader(file.path()); + auto frames = reader.frames(); + CHECK(reader.tell() == 0); + const TestCluster *storage = nullptr; + std::vector frame_numbers; + for (auto &frame : frames) { + frame_numbers.push_back(frame.frame_number()); + if (frame.frame_number() == -42) { + check_frame(frame, first); + storage = frame.data(); + } else if (frame.frame_number() == 105) { + check_frame(frame, second); + CHECK(frame.data() == storage); + } else { + CHECK(frame.empty()); + } + } + CHECK(frame_numbers == std::vector{-43, -42, 0, 105}); + CHECK(frames.begin() == frames.end()); +} + +TEST_CASE("ClusterFile chunks preserve cluster order across frame boundaries", + "[ClusterFile]") { + const auto chunk_size = GENERATE(1, 2, 3, 10); + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(ClusterVector(0)); + writer.write_frame(first); + writer.write_frame(ClusterVector(0)); + writer.write_frame(second); + writer.write_frame(ClusterVector(0)); + } + + ClusterFile reader(file.path(), chunk_size); + auto chunks = reader.chunks(); + auto requested = static_cast(chunk_size); + SECTION("constructor size") {} + SECTION("explicit size overrides the constructor") { + chunks = reader.chunks(3); + requested = 3; + CHECK(reader.chunk_size() == static_cast(chunk_size)); + } + CHECK(reader.tell() == 0); + size_t count = 0; + for (auto &chunk : chunks) { + REQUIRE(chunk.size() == std::min(requested, 4 - count)); + for (const auto &cluster : chunk) { + REQUIRE(count < 4); + const auto &expected = count < 2 ? first[count] : second[count - 2]; + CHECK(cluster.x == expected.x); + CHECK(cluster.y == expected.y); + CHECK(cluster.data == expected.data); + ++count; + } + } + CHECK(count == 4); + CHECK(chunks.begin() == chunks.end()); +} + +TEST_CASE("ClusterFile iterator results can be moved out without copying", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first); + writer.write_frame(second); + } + ClusterFile reader(file.path(), 2); + auto check_moves = [&](auto range) { + auto it = range.begin(); + REQUIRE(it != range.end()); + CHECK(range.end() != it); + const auto storage = it->data(); + auto retained = std::move(*it); + CHECK(retained.data() == storage); + ++it; + REQUIRE(it != range.end()); + check_frame(*it, second); + check_frame(retained, first); + it++; + CHECK(it == range.end()); + CHECK(range.end() == it); + ++it; + CHECK(it == range.end()); + reader.close(); + check_frame(retained, first); + }; + SECTION("frames") { check_moves(reader.frames()); } + SECTION("chunks") { check_moves(reader.chunks()); } +} + +TEST_CASE("ClusterFile iteration preserves filtering and gain correction", + "[ClusterFile]") { + const auto use_roi = GENERATE(false, true); + const auto use_noise = GENERATE(false, true); + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(make_test_frame(42, 0.0)); + writer.write_frame(make_test_frame(43, 100.0)); + } + ClusterFile reader(file.path(), 3); + ClusterFile reference(file.path()); + aare::NDArray gain({12, 12}, 2.0); + aare::NDArray noise({12, 12}, 5); + for (auto *file_reader : {&reader, &reference}) { + file_reader->set_gain_map(gain.view()); + if (use_roi) { + file_reader->set_roi(aare::ROI{0, 6, 0, 12}); + } + if (use_noise) { + file_reader->set_noise_map(noise.view()); + } + } + SECTION("frames include fully rejected frames") { + size_t count = 0; + for (auto &frame : reader.frames()) { + auto expected = reference.read_frame(); + REQUIRE(expected); + check_frame(frame, *expected); + ++count; + } + CHECK(count == 2); + CHECK_FALSE(reference.read_frame()); + } + SECTION("chunks count selected clusters") { + for (auto &chunk : reader.chunks()) { + auto expected = reference.read_clusters(3); + check_frame(chunk, expected); + } + CHECK(reference.read_clusters(3).empty()); + } +} + +TEST_CASE("ClusterFile iteration resumes after an early exit", + "[ClusterFile]") { + TemporaryClusterFile file; + auto first = make_test_frame(42, 0.0); + auto second = make_test_frame(43, 100.0); + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(first); + writer.write_frame(second); + } + ClusterFile reader(file.path()); + SECTION("breaking a frame loop does not consume the following frame") { + for (auto &frame : reader.frames()) { + check_frame(frame, first); + break; + } + auto next = reader.read_frame(); + REQUIRE(next); + check_frame(*next, second); + } + SECTION("partial chunks must be completed before frame iteration") { + for (auto &chunk : reader.chunks(1)) { + REQUIRE(chunk.size() == 1); + CHECK(chunk[0].data == first[0].data); + break; + } + CHECK_THROWS_AS(reader.frames().begin(), std::runtime_error); + auto remainder = reader.read_clusters(1); + REQUIRE(remainder.size() == 1); + CHECK(remainder[0].data == first[1].data); + auto it = reader.frames().begin(); + REQUIRE(it != reader.frames().end()); + check_frame(*it, second); + } +} + +TEST_CASE("ClusterFile iterators distinguish EOF from read errors", + "[ClusterFile]") { + TemporaryClusterFile file; + { + ClusterFile writer(file.path(), 1000, "w"); + writer.write_frame(make_test_frame(42, 0.0)); + } + auto check_error = [&] { + ClusterFile frame_reader(file.path()); + ClusterFile chunk_reader(file.path()); + CHECK_THROWS_AS(frame_reader.frames().begin(), std::runtime_error); + CHECK_THROWS_AS(chunk_reader.chunks().begin(), std::runtime_error); + }; + SECTION("empty file") { + std::filesystem::resize_file(file.path(), 0); + ClusterFile reader(file.path()); + CHECK(reader.frames().begin() == reader.frames().end()); + CHECK(reader.chunks().begin() == reader.chunks().end()); + } + SECTION("partial header") { + std::filesystem::resize_file(file.path(), 7); + check_error(); + } + SECTION("partial record") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + check_error(); + } + SECTION("partial record encountered on increment") { + std::filesystem::resize_file( + file.path(), std::filesystem::file_size(file.path()) - 1); + ClusterFile reader(file.path()); + auto it = reader.chunks(1).begin(); + REQUIRE(it != reader.chunks(1).end()); + CHECK_THROWS_AS(++it, std::runtime_error); + } + SECTION("closed before first read") { + ClusterFile reader(file.path()); + auto frames = reader.frames(); + auto chunks = reader.chunks(); + reader.close(); + CHECK_THROWS_AS(frames.begin(), std::runtime_error); + CHECK_THROWS_AS(chunks.begin(), std::runtime_error); + } + SECTION("closed during traversal") { + ClusterFile reader(file.path()); + auto it = reader.frames().begin(); + reader.close(); + CHECK_THROWS_AS(++it, std::runtime_error); + } + SECTION("write and append modes") { + const auto mode = GENERATE("w", "a"); + ClusterFile writer(file.path(), 1000, mode); + CHECK_THROWS_AS(writer.frames().begin(), std::runtime_error); + CHECK_THROWS_AS(writer.chunks().begin(), std::runtime_error); + } + SECTION("zero chunk sizes are rejected without reading") { + ClusterFile reader(file.path(), 0); + CHECK_THROWS_AS(reader.chunks(), std::invalid_argument); + CHECK_THROWS_AS(reader.chunks(0), std::invalid_argument); + CHECK(reader.tell() == 0); + CHECK(reader.read_clusters(0).empty()); + CHECK(reader.frames().begin() != reader.frames().end()); + } +} + +TEST_CASE("Read frame and modify cluster data", "[.with-data]") { auto fpath = test_data_path() / "clust" / "single_frame_97_clustrers.clust"; REQUIRE(std::filesystem::exists(fpath)); ClusterFile> f(fpath); auto clusters = f.read_frame(); - CHECK(clusters.size() == 97); - CHECK(clusters.frame_number() == 135); + REQUIRE(clusters); + CHECK(clusters->size() == 97); + CHECK(clusters->frame_number() == 135); int32_t expected_cluster_data[] = {0, 1, 2, 3, 4, 5, 6, 7, 8}; - clusters.push_back( + clusters->push_back( Cluster{0, 0, {0, 1, 2, 3, 4, 5, 6, 7, 8}}); - CHECK(clusters.size() == 98); - CHECK(clusters[0].x == 1); - CHECK(clusters[0].y == 200); + CHECK(clusters->size() == 98); + CHECK((*clusters)[0].x == 1); + CHECK((*clusters)[0].y == 200); - CHECK(std::equal(std::begin(clusters[0].data), std::end(clusters[0].data), + CHECK(std::equal(std::begin((*clusters)[0].data), + std::end((*clusters)[0].data), std::begin(expected_cluster_data))); } diff --git a/src/ClusterFinder.test.cpp b/src/ClusterFinder.test.cpp index fd7eeb37..5ce214fd 100644 --- a/src/ClusterFinder.test.cpp +++ b/src/ClusterFinder.test.cpp @@ -2,6 +2,7 @@ #include "aare/ClusterFinder.hpp" #include "aare/Pedestal.hpp" #include +#include #include #include #include @@ -45,6 +46,42 @@ TEST_CASE("Construct a cluster finder") { // REQUIRE(clusterFinder.get_nSigma() == 1); } +TEST_CASE("nSigma set before the pedestal is ready is used for the threshold") { + const Shape<2> image_size{10, 10}; + const size_t min_pedestal_samples = 4; + + // Pedestal of 100 with a noise of 2 and a 30 ADU hit: a cluster at the + // default nSigma of 5, below threshold at nSigma 50. + NDArray low(image_size, 98); + NDArray high(image_size, 102); + NDArray hit(image_size, 100); + hit(5, 5) = 130; + + const double nSigma = GENERATE(5.0, 50.0); + const size_t expected_clusters = nSigma == 5.0 ? 1 : 0; + + ClusterFinder> cf(image_size, 5.0, 100, + min_pedestal_samples); + REQUIRE_FALSE(cf.pedestal_ready()); + REQUIRE_NOTHROW(cf.set_nSigma(nSigma)); + REQUIRE(cf.get_nSigma() == nSigma); + + for (size_t i = 0; i < min_pedestal_samples / 2; ++i) { + cf.push_pedestal_frame(low.view()); + REQUIRE_FALSE(cf.pedestal_ready()); + cf.push_pedestal_frame(high.view()); + } + REQUIRE(cf.pedestal_ready()); + + cf.find_clusters(hit.view()); + REQUIRE(cf.steal_clusters().size() == expected_clusters); +} + +TEST_CASE("update_threshold throws until the pedestal is ready") { + ClusterFinder> cf({10, 10}); + REQUIRE_THROWS_AS(cf.update_threshold(), std::runtime_error); +} + // TEST_CASE("test cluster finder") { // aare::Pedestal pedestal(10, 10, 5); // NDArray frame({10, 10}); diff --git a/src/ClusterFinderMT.test.cpp b/src/ClusterFinderMT.test.cpp index e8904814..6973d0df 100644 --- a/src/ClusterFinderMT.test.cpp +++ b/src/ClusterFinderMT.test.cpp @@ -21,14 +21,23 @@ class ClusterFinderMTWrapper public: ClusterFinderMTWrapper(Shape<2> image_size, PEDESTAL_TYPE nSigma = 5.0, - size_t capacity = 2000, size_t n_threads = 3) + size_t capacity = 2000, size_t n_threads = 3, + size_t queue_depth = 16, + size_t minimum_pedestal_samples = 1000) : ClusterFinderMT( - image_size, nSigma, capacity, n_threads) {} + image_size, nSigma, capacity, n_threads, queue_depth, + minimum_pedestal_samples) {} size_t get_m_input_queues_size() const { return this->m_input_queues.size(); } + size_t get_m_frame_pools_size() const { return this->m_frame_pools.size(); } + + size_t get_frame_pool_depth(size_t thread_index) const { + return this->m_frame_pools[thread_index]->size(); + } + size_t get_m_output_queues_size() const { return this->m_output_queues.size(); } @@ -66,13 +75,14 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { File file(fpath); size_t n_threads = 2; - size_t n_frames_pd = 10; + size_t minimum_pedestal_samples = 1; + size_t n_frames = 10; using ClusterType = Cluster; ClusterFinderMTWrapper cf( {static_cast(file.rows()), static_cast(file.cols())}, - 5, 2000, n_threads); // no idea what frame type is!!! default uint16_t + 5, 2000, n_threads, 16, minimum_pedestal_samples); CHECK(cf.get_m_input_queues_size() == n_threads); CHECK(cf.get_m_output_queues_size() == n_threads); @@ -80,8 +90,11 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_output_queues_are_empty() == true); CHECK(cf.m_input_queues_are_empty() == true); - for (size_t i = 0; i < n_frames_pd; ++i) { - auto frame = file.read_frame(); + auto frame = file.read_frame(); + cf.push_pedestal_frame(frame.view()); + + for (size_t i = 0; i < n_frames; ++i) { + frame = file.read_frame(); cf.find_clusters(frame.view()); } @@ -90,7 +103,7 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_output_queues_are_empty() == true); CHECK(cf.m_input_queues_are_empty() == true); - CHECK(cf.m_sink_size() == n_frames_pd); + CHECK(cf.m_sink_size() == n_frames); ClusterCollector clustercollector(&cf); clustercollector.stop(); @@ -98,5 +111,115 @@ TEST_CASE("multithreaded cluster finder", "[.with-data]") { CHECK(cf.m_sink_size() == 0); auto clustervec = clustercollector.steal_clusters(); - // CHECK(clustervec.size() == ) //dont know how many clusters to expect +} + +TEST_CASE("frame buffers are recycled when pushing more frames than the pool " + "holds", + "[.files]") { + using ClusterType = Cluster; + + const size_t n_threads = 2; + const size_t queue_depth = 4; + const size_t n_frames = 5 * queue_depth; + const Shape<2> image_size{10, 10}; + + ClusterFinderMTWrapper cf(image_size, 5, 200, n_threads, + queue_depth); + + CHECK(cf.get_m_frame_pools_size() == n_threads); + for (size_t i = 0; i < n_threads; ++i) { + CHECK(cf.get_frame_pool_depth(i) == queue_depth); + } + + NDArray frame(image_size, 0); + + // More frames than the pool holds, so this only completes if the workers + // return the buffers to the free list. + for (size_t i = 0; i < n_frames; ++i) { + cf.find_clusters(frame.view(), i); + } + + cf.stop(); + + CHECK(cf.m_input_queues_are_empty() == true); +} + +TEST_CASE("nSigma set before the pedestal is ready reaches all threads") { + using ClusterType = Cluster; + + const size_t n_threads = 2; + const size_t min_pedestal_samples = 4; + const Shape<2> image_size{10, 10}; + + ClusterFinderMTWrapper cf(image_size, 5, 200, n_threads, 16, + min_pedestal_samples); + + // Pedestal of 100 with a noise of 2 and a 30 ADU hit: a cluster at the + // default nSigma of 5, below threshold at nSigma 50. + NDArray low(image_size, 98); + NDArray high(image_size, 102); + NDArray hit(image_size, 100); + hit(5, 5) = 130; + + CHECK_FALSE(cf.pedestal_ready()); + REQUIRE_NOTHROW(cf.set_nSigma(50)); + + for (size_t i = 0; i < min_pedestal_samples / 2; ++i) { + cf.push_pedestal_frame(low.view()); + cf.push_pedestal_frame(high.view()); + } + CHECK(cf.pedestal_ready()); + + for (size_t i = 0; i < n_threads; ++i) { + cf.find_clusters(hit.view(), i); + } + cf.stop(); + + ClusterCollector collector(&cf); + collector.stop(); + + auto clusters = collector.steal_clusters(); + REQUIRE(clusters.size() == n_threads); + for (const auto &frame_clusters : clusters) { + CHECK(frame_clusters.size() == 0); + } +} + +TEST_CASE("cluster collector accepts finders with a matching cluster type") { + using ClusterType = Cluster; + using Finder = ClusterFinderMTWrapper; + using OtherFinder = + ClusterFinderMTWrapper, uint16_t, float>; + + static_assert( + std::is_constructible_v, Finder *>); + static_assert( + !std::is_constructible_v, OtherFinder *>); + + Finder cf({10, 10}); + cf.stop(); + + ClusterCollector collector(&cf); + collector.stop(); + + CHECK(collector.steal_clusters().empty()); +} + +TEST_CASE("cluster collector drains queued clusters when stopped") { + using ClusterType = Cluster; + ProducerConsumerQueue> source(4); + + for (uint64_t frame_number = 1; frame_number <= 3; ++frame_number) { + REQUIRE(source.write(ClusterVector(4, frame_number))); + } + + ClusterCollector collector(&source); + collector.stop(); + + auto clusters = collector.steal_clusters(); + REQUIRE(clusters.size() == 3); + CHECK(source.isEmpty()); + for (size_t i = 0; i < clusters.size(); ++i) { + CHECK(clusters[i].frame_number() == static_cast(i + 1)); + } } diff --git a/src/ClusterVector.test.cpp b/src/ClusterVector.test.cpp index 33b0b98c..135719c1 100644 --- a/src/ClusterVector.test.cpp +++ b/src/ClusterVector.test.cpp @@ -1,6 +1,10 @@ // SPDX-License-Identifier: MPL-2.0 #include "aare/ClusterVector.hpp" +#include "aare/GainMap.hpp" +#include "aare/NDArray.hpp" +#include #include +#include #include #include @@ -21,6 +25,81 @@ TEST_CASE("After pushing back one element the ClusterVector is not empty") { REQUIRE(!cv.empty()); } +TEST_CASE("Filtering a ClusterVector selects the masked elements") { + ClusterVector source(8, 123); + source.push_back(C1{1, 2, {3, 4, 5, 6}}); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); + source.push_back(C1{13, 14, {15, 16, 17, 18}}); + source.push_back(C1{19, 20, {21, 22, 23, 24}}); + + SECTION("Some elements are selected") { + std::array mask{true, false, false, true}; + + auto filtered = source(aare::NDView{mask.data(), {4}}); + + CHECK(filtered.size() == 2); + CHECK(filtered.frame_number() == 123); + CHECK(filtered[0].x == 1); + CHECK(filtered[1].x == 19); + } + + SECTION("No elements are selected") { + std::array mask{false, false, false, false}; + + auto filtered = source(aare::NDView{mask.data(), {4}}); + + CHECK(filtered.empty()); + CHECK(filtered.frame_number() == 123); + } + + SECTION("An empty vector accepts a default empty mask") { + ClusterVector empty_source(0, -123); + + auto filtered = empty_source(aare::NDView{}); + + CHECK(filtered.empty()); + CHECK(filtered.frame_number() == -123); + } +} + +TEST_CASE("Move constructing a ClusterVector transfers its storage") { + ClusterVector source(4, 123); + source.push_back(C1{1, 2, {3, 4, 5, 6}}); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); + + const auto *source_data = source.data(); + const auto source_capacity = source.capacity(); + + ClusterVector destination(std::move(source)); + + CHECK(destination.data() == source_data); + CHECK(destination.capacity() == source_capacity); + CHECK(destination.size() == 2); + CHECK(destination.frame_number() == 123); + CHECK(destination[0].x == 1); + CHECK(destination[1].x == 7); +} + +TEST_CASE("Move assigning a ClusterVector transfers its storage") { + ClusterVector source(4, 123); + source.push_back(C1{1, 2, {3, 4, 5, 6}}); + source.push_back(C1{7, 8, {9, 10, 11, 12}}); + + const auto *source_data = source.data(); + const auto source_capacity = source.capacity(); + + ClusterVector destination(2, 456); + destination.push_back(C1{13, 14, {15, 16, 17, 18}}); + destination = std::move(source); + + CHECK(destination.data() == source_data); + CHECK(destination.capacity() == source_capacity); + CHECK(destination.size() == 2); + CHECK(destination.frame_number() == 123); + CHECK(destination[0].x == 1); + CHECK(destination[1].x == 7); +} + TEST_CASE("item_size return the size of the cluster stored") { ClusterVector cv(4); CHECK(cv.item_size() == sizeof(C1)); @@ -232,6 +311,28 @@ TEST_CASE("Concatenate two cluster vectors where we need to allocate") { REQUIRE(ptr[3].y == 17); } +TEST_CASE("Reducing a ClusterVector preserves its frame number") { + SECTION("Reduce to 2x2") { + ClusterVector> source(1, -135); + source.push_back(Cluster{}); + + auto reduced = aare::reduce_to_2x2(source); + + CHECK(reduced.size() == source.size()); + CHECK(reduced.frame_number() == source.frame_number()); + } + + SECTION("Reduce to 3x3") { + ClusterVector> source(1, -246); + source.push_back(Cluster{}); + + auto reduced = aare::reduce_to_3x3(source); + + CHECK(reduced.size() == source.size()); + CHECK(reduced.frame_number() == source.frame_number()); + } +} + struct ClusterTestData { uint8_t ClusterSizeX; uint8_t ClusterSizeY; @@ -275,4 +376,74 @@ TEST_CASE("Gain Map Calculation Index Map") { CHECK(index_map_x == clustertestdata.index_map_x); CHECK(index_map_y == clustertestdata.index_map_y); -} \ No newline at end of file +} + +namespace { + +template +void check_gain_map_cluster_bounds() { + using ClusterType = Cluster; + + constexpr ssize_t rows = 16; + constexpr ssize_t cols = 16; + constexpr uint16_t left = ClusterSizeX / 2; + constexpr uint16_t right = ClusterSizeX - left - 1; + constexpr uint16_t top = ClusterSizeY / 2; + constexpr uint16_t bottom = ClusterSizeY - top - 1; + + aare::NDArray gain_map({rows, cols}, 2.0); + aare::InvertedGainMap inverted_gain_map(gain_map); + ClusterVector clusters(6); + + const auto add_cluster = [&clusters](uint16_t x, uint16_t y) { + ClusterType cluster{}; + cluster.x = x; + cluster.y = y; + cluster.data.fill(2.0); + clusters.push_back(cluster); + }; + + add_cluster(left, top); + add_cluster(cols - right - 1, rows - bottom - 1); + add_cluster(left - 1, top); + add_cluster(left, top - 1); + add_cluster(cols - right, top); + add_cluster(left, rows - bottom); + + inverted_gain_map.apply_gain_map(clusters); + + for (size_t i = 0; i < 2; ++i) { + CHECK(std::all_of(clusters[i].data.begin(), clusters[i].data.end(), + [](double value) { return value == 1.0; })); + } + for (size_t i = 2; i < clusters.size(); ++i) { + CHECK(std::all_of(clusters[i].data.begin(), clusters[i].data.end(), + [](double value) { return value == 0.0; })); + } + + aare::NDArray small_gain_map( + {ClusterSizeY - 1, ClusterSizeX - 1}, 2.0); + aare::InvertedGainMap small_inverted_gain_map(small_gain_map); + ClusterVector oversized_cluster(1); + ClusterType cluster{}; + cluster.x = left; + cluster.y = top; + cluster.data.fill(2.0); + oversized_cluster.push_back(cluster); + + small_inverted_gain_map.apply_gain_map(oversized_cluster); + + CHECK(std::all_of(oversized_cluster[0].data.begin(), + oversized_cluster[0].data.end(), + [](double value) { return value == 0.0; })); +} + +} // namespace + +TEST_CASE("Gain map bounds cover the full cluster footprint", "[GainMap]") { + SECTION("3x3") { check_gain_map_cluster_bounds<3, 3>(); } + SECTION("5x5") { check_gain_map_cluster_bounds<5, 5>(); } + SECTION("7x7") { check_gain_map_cluster_bounds<7, 7>(); } + SECTION("9x9") { check_gain_map_cluster_bounds<9, 9>(); } + SECTION("5x7") { check_gain_map_cluster_bounds<5, 7>(); } +} diff --git a/src/CtbRawFile.cpp b/src/CtbRawFile.cpp index 260ab668..e4d0b958 100644 --- a/src/CtbRawFile.cpp +++ b/src/CtbRawFile.cpp @@ -24,9 +24,9 @@ void CtbRawFile::read_into(std::byte *image_buf, DetectorHeader *header) { throw std::runtime_error(LOCATION + " End of file reached"); } - if (m_current_frame != 0 && - m_current_frame % m_master.max_frames_per_file() == 0) { - open_data_file(m_current_subfile + 1); + const auto index = sub_file_index(m_current_frame); + if (index != m_current_subfile) { + open_data_file(index); } if (header) { @@ -79,4 +79,4 @@ void CtbRawFile::open_data_file(size_t subfile_index) { } } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/src/DetectorGeometry.cpp b/src/DetectorGeometry.cpp index ee3525f2..c0ceef9a 100644 --- a/src/DetectorGeometry.cpp +++ b/src/DetectorGeometry.cpp @@ -11,7 +11,8 @@ DetectorGeometry::DetectorGeometry(const xy &geometry, const ssize_t module_pixels_x, const ssize_t module_pixels_y, const xy udp_interfaces_per_module, - const bool quad) { + const bool quad) + : m_udp_interfaces_per_module(udp_interfaces_per_module) { size_t num_modules = geometry.col * geometry.row; module_geometries.reserve(num_modules); @@ -43,6 +44,10 @@ DetectorGeometry::DetectorGeometry(const xy &geometry, m_pixels_y += static_cast((geometry.row - 1) * cfg.module_gap_row); } +xy DetectorGeometry::udp_interfaces_per_module() const { + return m_udp_interfaces_per_module; +} + size_t DetectorGeometry::n_modules() const { return m_modules_x * m_modules_y; } size_t DetectorGeometry::pixels_x() const { return m_pixels_x; } diff --git a/src/FastPedestal.test.cpp b/src/FastPedestal.test.cpp new file mode 100644 index 00000000..7c003849 --- /dev/null +++ b/src/FastPedestal.test.cpp @@ -0,0 +1,60 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/FastPedestal.hpp" + +#include +#include + +using namespace aare; + +TEMPLATE_TEST_CASE("fast pedestal keeps variance in double precision", + "[fast-pedestal]", double, float, int16_t) { + FastPedestal pedestal(1, 1, 2); + NDArray frame({1, 1}, 0); + pedestal.add_init_frame(frame.view()); + frame[0] = 1000; + pedestal.add_init_frame(frame.view()); + + const auto &ready = pedestal; + REQUIRE(ready.std(0, 0) == static_cast(500)); + REQUIRE(ready.std_unchecked(0) == ready.std(0, 0)); + REQUIRE(ready.std()[0] == ready.std(0, 0)); + + pedestal.push_ema(frame.view()); + REQUIRE(ready.std(0, 0) == static_cast(std::sqrt(187500.0))); + REQUIRE(ready.std_unchecked(0) == ready.std(0, 0)); + REQUIRE(ready.std()[0] == ready.std(0, 0)); +} + +TEST_CASE("fast pedestal validates standard deviation access", + "[fast-pedestal]") { + FastPedestal pedestal(1, 1, 1); + REQUIRE_THROWS(pedestal.std()); + REQUIRE_THROWS(pedestal.std(0, 0)); + + NDArray frame({1, 1}, 42); + pedestal.add_init_frame(frame.view()); + REQUIRE(pedestal.std(0, 0) == 0.0); + REQUIRE_THROWS(pedestal.std(1, 0)); + REQUIRE_THROWS(pedestal.std(0, 1)); +} + +TEMPLATE_TEST_CASE("fast pedestal std stays finite after settling", + "[fast-pedestal]", double, float, int16_t) { + FastPedestal pedestal(1, 1, 10); + NDArray frame({1, 1}, 16382); + pedestal.add_init_frame(frame.view()); + frame[0] = 16383; + for (int i = 0; i < 9; ++i) { + pedestal.add_init_frame(frame.view()); + } + for (int i = 0; i < 300; ++i) { + pedestal.push_ema(frame.view()); + } + + const auto noise = pedestal.std(0, 0); + REQUIRE(std::isfinite(noise)); + REQUIRE(noise >= 0); + REQUIRE(static_cast(noise) < 1e-3); + REQUIRE(pedestal.std_unchecked(0) == noise); + REQUIRE(pedestal.std()[0] == noise); +} diff --git a/src/FilePtr.cpp b/src/FilePtr.cpp index dd907bda..45e65d46 100644 --- a/src/FilePtr.cpp +++ b/src/FilePtr.cpp @@ -22,6 +22,8 @@ FilePtr &FilePtr::operator=(FilePtr &&other) { return *this; } +FilePtr::operator bool() const noexcept { return fp_ != nullptr; } + FILE *FilePtr::get() { return fp_; } ssize_t FilePtr::tell() { diff --git a/src/FilePtr.test.cpp b/src/FilePtr.test.cpp new file mode 100644 index 00000000..6b23e62a --- /dev/null +++ b/src/FilePtr.test.cpp @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/FilePtr.hpp" + +#include +#include + +TEST_CASE("FilePtr converts to its open state", "[FilePtr]") { + aare::FilePtr empty; + CHECK_FALSE(empty); + + aare::FilePtr open(__FILE__, "rb"); + CHECK(open); + + aare::FilePtr moved(std::move(open)); + CHECK(moved); + CHECK_FALSE(open); +} diff --git a/src/Fit.cpp b/src/Fit.cpp deleted file mode 100644 index f11aa611..00000000 --- a/src/Fit.cpp +++ /dev/null @@ -1,308 +0,0 @@ -// SPDX-License-Identifier: MPL-2.0 -#include "aare/Fit.hpp" -#include "Chi2.hpp" -#include "Minuit2/FunctionMinimum.h" -#include "Minuit2/MnHesse.h" -#include "Minuit2/MnMigrad.h" -#include "Minuit2/MnStrategy.h" -#include "Minuit2/MnUserParameters.h" -#include "aare/Models.hpp" -#include "aare/utils/par.hpp" -#include "aare/utils/task.hpp" -#include -#include -#include -#include -#include - -namespace aare { -// ============================================================================ -// FitModel — method definitions -// (constructor, destructor, copy, and all methods that touch Minuit2 state) -// ============================================================================ - -template struct FitModel::FitModelImpl { - ROOT::Minuit2::MnUserParameters upar; - ROOT::Minuit2::MnStrategy strategy; - - explicit FitModelImpl(unsigned int strategy_level) - : strategy(strategy_level) {} - - FitModelImpl(const FitModelImpl &) = default; - FitModelImpl &operator=(const FitModelImpl &) = default; -}; - -template -FitModel::FitModel(unsigned int strategy, unsigned int max_calls, - double tolerance, bool compute_errors) - : impl_(std::make_unique(strategy)), max_calls_(max_calls), - tolerance_(tolerance), compute_errors_(compute_errors) { - for (std::size_t i = 0; i < npar; ++i) { - const auto pi = Model::param_info[i]; - const bool has_lo = std::isfinite(pi.default_lo); - const bool has_hi = std::isfinite(pi.default_hi); - if (has_lo && has_hi) { - impl_->upar.Add(pi.name, 0.0, 1.0, pi.default_lo, pi.default_hi); - } else if (has_lo) { - impl_->upar.Add(pi.name, 0.0, 1.0, pi.default_lo, 1e6); - } else { - impl_->upar.Add(pi.name, 0.0, 1.0); - } - } -} - -template FitModel::~FitModel() = default; - -template -FitModel::FitModel(const FitModel &other) - : impl_(std::make_unique(*other.impl_)), - max_calls_(other.max_calls_), tolerance_(other.tolerance_), - compute_errors_(other.compute_errors_), user_fixed_(other.user_fixed_), - user_start_(other.user_start_) {} - -template -FitModel &FitModel::operator=(const FitModel &other) { - if (this != &other) { - impl_ = std::make_unique(*other.impl_); - max_calls_ = other.max_calls_; - tolerance_ = other.tolerance_; - compute_errors_ = other.compute_errors_; - user_fixed_ = other.user_fixed_; - user_start_ = other.user_start_; - } - return *this; -} - -template -unsigned int FitModel::checked_index(const std::string &name) const { - for (std::size_t i = 0; i < npar; ++i) { - if (impl_->upar.Name(i) == name) - return static_cast(i); - } - throw std::runtime_error("FitModel: unknown parameter name '" + name + "'"); -} - -template -void FitModel::SetParLimits(unsigned int idx, double lo, double hi) { - impl_->upar.SetLimits(idx, lo, hi); -} - -template -void FitModel::FixParameter(unsigned int idx, double val) { - SetParameter(idx, val); - impl_->upar.Fix(idx); - user_fixed_[idx] = true; -} - -template -void FitModel::ReleaseParameter(unsigned int idx) { - impl_->upar.Release(idx); - user_fixed_[idx] = false; -} - -template -void FitModel::ReleaseParameter(const std::string &name) { - ReleaseParameter(checked_index(name)); -} - -template -void FitModel::SetParameter(unsigned int idx, double val) { - impl_->upar.SetValue(idx, val); - user_start_[idx] = true; -} - -template -void FitModel::SetParameter(const std::string &name, double val) { - SetParameter(checked_index(name), val); -} - -template -void FitModel::FixParameter(const std::string &name, double val) { - FixParameter(checked_index(name), val); -} - -template -void FitModel::SetParLimits(const std::string &name, double lo, - double hi) { - SetParLimits(checked_index(name), lo, hi); -} - -template -std::string FitModel::GetParName(unsigned int idx) const { - return impl_->upar.GetName(idx); -} - -template -std::vector FitModel::GetParNames() const { - std::vector names; - for (std::size_t i = 0; i < npar; ++i) - names.push_back(GetParName(i)); - return names; -} - -// ============================================================================ -// fit_pixel / fit_3d — Minuit2 template implementations -// ============================================================================ - -template -NDArray fit_pixel(const FitModel &model, NDView x, - NDView y, NDView y_err) { - using FCN = func::Chi2Model1DGrad; - constexpr std::size_t npar = Model::npar; - const bool want_errors = model.compute_errors(); - const ssize_t result_size = want_errors ? (2 * npar + 1) : (npar + 1); - - auto start = Model::estimate_par(x, y); - - if (!Model::is_valid(std::vector(start.begin(), start.end()))) { - return NDArray({result_size}, 0.0); - } - - double x_range, y_range, slope_scale; - model::compute_ranges(x, y, x_range, y_range, slope_scale); - - std::array steps{}; - Model::compute_steps(start, x_range, y_range, slope_scale, steps); - - // thread-local copy of starting parameters - auto upar_local = model.impl()->upar; - - for (std::size_t i = 0; i < npar; ++i) { - if (model.is_user_fixed(i)) - continue; - if (!model.is_user_start(i)) - upar_local.SetValue(i, start[i]); - upar_local.SetError(i, steps[i]); - } - - auto chi2 = (y_err.size() > 0) ? FCN(x, y, y_err) : FCN(x, y); - - ROOT::Minuit2::MnMigrad migrad(chi2, upar_local, model.impl()->strategy); - ROOT::Minuit2::FunctionMinimum min = - migrad(model.max_calls(), model.tolerance()); - - if (!min.IsValid()) - return NDArray({result_size}, 0.0); - - if (want_errors) { - ROOT::Minuit2::MnHesse hesse; - hesse(chi2, min); - - const auto &values = min.UserState().Params(); - const auto &errors = min.UserState().Errors(); - - NDArray result({result_size}); - for (std::size_t k = 0; k < npar; ++k) { - result[k] = values[k]; - result[npar + k] = errors[k]; - } - result[2 * npar] = min.Fval(); - return result; - } - - const auto &values = min.UserState().Params(); - NDArray result({result_size}); - for (std::size_t k = 0; k < npar; ++k) - result[k] = values[k]; - result[npar] = min.Fval(); - return result; -} - -template -NDArray fit_pixel(const FitModel &model, NDView x, - NDView y) { - return fit_pixel(model, x, y, NDView{}); -} - -template -void fit_3d(const FitModel &model, NDView x, - NDView y, NDView y_err, - NDView par_out, NDView err_out, - NDView chi2_out, int n_threads) { - const std::size_t npar = Model::npar; - - if (x.size() != y.shape(2)) - throw std::runtime_error("fit_3d: x.size() must match y.shape(2)."); - - if (par_out.shape(0) != y.shape(0) || par_out.shape(1) != y.shape(1) || - par_out.shape(2) != npar) - throw std::runtime_error("par_out must have shape [rows, cols, npar]."); - - if (chi2_out.shape(0) != y.shape(0) || chi2_out.shape(1) != y.shape(1)) - throw std::runtime_error("chi2_out must have shape [rows, cols]."); - - const bool has_errors = (y_err.size() > 0); - const bool want_par_errors = (err_out.size() > 0) && model.compute_errors(); - - if (has_errors) { - if (y.shape(0) != y_err.shape(0) || y.shape(1) != y_err.shape(1) || - y.shape(2) != y_err.shape(2)) - throw std::runtime_error( - "fit_3d: y and y_err must have identical shape."); - - if (err_out.shape(0) != y.shape(0) || err_out.shape(1) != y.shape(1) || - err_out.shape(2) != npar) - throw std::runtime_error( - "err_out must have shape [rows, cols, npar]."); - } - - auto process = [&](ssize_t first_row, ssize_t last_row) { - for (ssize_t row = first_row; row < last_row; row++) { - for (ssize_t col = 0; col < y.shape(1); col++) { - - NDView values(&y(row, col, 0), {y.shape(2)}); - NDView errors = - has_errors ? NDView(&y_err(row, col, 0), - {y_err.shape(2)}) - : NDView{}; - - auto res = fit_pixel(model, x, values, errors); - - for (std::size_t k = 0; k < npar; ++k) - par_out(row, col, k) = res(k); - - if (want_par_errors) { - for (std::size_t k = 0; k < npar; ++k) - err_out(row, col, k) = res(npar + k); - chi2_out(row, col) = res(2 * npar); - } else { - chi2_out(row, col) = res(npar); - } - } - } - }; - - auto tasks = split_task(0, static_cast(y.shape(0)), n_threads); - RunInParallel(process, tasks); -} - -// ============================================================================ -// Explicit instantiations for all supported model types -// ============================================================================ - -// NOLINTBEGIN -#define AARE_INSTANTIATE_FIT(Model) \ - template class FitModel; \ - template NDArray fit_pixel( \ - const FitModel &, NDView, NDView, \ - NDView); \ - template NDArray fit_pixel( \ - const FitModel &, NDView, NDView); \ - template void fit_3d(const FitModel &, NDView, \ - NDView, NDView, \ - NDView, NDView, \ - NDView, int); - -AARE_INSTANTIATE_FIT(model::Gaussian) -AARE_INSTANTIATE_FIT(model::GaussianErfcPlateau) -AARE_INSTANTIATE_FIT(model::GaussianChargeSharing) -AARE_INSTANTIATE_FIT(model::GaussianChargeSharingKb) -AARE_INSTANTIATE_FIT(model::Pol1) -AARE_INSTANTIATE_FIT(model::Pol2) -AARE_INSTANTIATE_FIT(model::RisingScurve) -AARE_INSTANTIATE_FIT(model::FallingScurve) - -#undef AARE_INSTANTIATE_FIT -// NOLINTEND - -} // namespace aare diff --git a/src/Fit.test.cpp b/src/Fit.test.cpp deleted file mode 100644 index 2b0fd1bb..00000000 --- a/src/Fit.test.cpp +++ /dev/null @@ -1,81 +0,0 @@ -// SPDX-License-Identifier: MPL-2.0 -#include "aare/Fit.hpp" -#include "aare/FitModel.hpp" -#include "aare/Models.hpp" - -#include -#include -#include - -namespace { - -constexpr ssize_t n_points = 61; - -void fill_gaussian(aare::NDArray &x, aare::NDArray &y, - double amplitude, double mean, double sigma) { - for (ssize_t i = 0; i < n_points; ++i) { - x(i) = -6.0 + 0.2 * static_cast(i); - const double z = (x(i) - mean) / sigma; - y(i) = amplitude * std::exp(-0.5 * z * z); - } -} - -} // namespace - -TEST_CASE("Minuit2 fits weighted and unweighted Gaussian data", "[fit]") { - aare::NDArray x({n_points}); - aare::NDArray y({n_points}); - aare::NDArray y_err({n_points}, 1.0); - fill_gaussian(x, y, 120.0, 0.8, 1.3); - - const aare::FitModel unweighted_model; - const auto unweighted = - aare::fit_pixel(unweighted_model, x.view(), y.view()); - - REQUIRE(unweighted.size() == 4); - CHECK(unweighted(0) == Catch::Approx(120.0).epsilon(1e-5)); - CHECK(unweighted(1) == Catch::Approx(0.8).epsilon(1e-5)); - CHECK(unweighted(2) == Catch::Approx(1.3).epsilon(1e-5)); - CHECK(unweighted(3) == Catch::Approx(0.0).margin(1e-4)); - - const aare::FitModel weighted_model(0, 100, 0.5, - true); - const auto weighted = - aare::fit_pixel(weighted_model, x.view(), y.view(), y_err.view()); - - REQUIRE(weighted.size() == 7); - CHECK(weighted(0) == Catch::Approx(120.0).epsilon(1e-5)); - CHECK(weighted(1) == Catch::Approx(0.8).epsilon(1e-5)); - CHECK(weighted(2) == Catch::Approx(1.3).epsilon(1e-5)); - CHECK(weighted(6) == Catch::Approx(0.0).margin(1e-4)); -} - -TEST_CASE("Minuit2 fits a Gaussian data cube in parallel", "[fit]") { - aare::NDArray x({n_points}); - aare::NDArray values({n_points}); - fill_gaussian(x, values, 80.0, -0.6, 0.9); - - aare::NDArray y({2, 2, n_points}); - for (ssize_t row = 0; row < 2; ++row) { - for (ssize_t col = 0; col < 2; ++col) { - for (ssize_t i = 0; i < n_points; ++i) - y(row, col, i) = values(i); - } - } - - aare::NDArray par({2, 2, 3}); - aare::NDArray chi2({2, 2}); - const aare::FitModel model; - - aare::fit_3d(model, x.view(), y.view(), aare::NDView{}, - par.view(), aare::NDView{}, chi2.view(), 2); - - for (ssize_t row = 0; row < 2; ++row) { - for (ssize_t col = 0; col < 2; ++col) { - CHECK(par(row, col, 0) == Catch::Approx(80.0).epsilon(1e-5)); - CHECK(par(row, col, 1) == Catch::Approx(-0.6).epsilon(1e-5)); - CHECK(par(row, col, 2) == Catch::Approx(0.9).epsilon(1e-5)); - CHECK(chi2(row, col) == Catch::Approx(0.0).margin(1e-8)); - } - } -} diff --git a/src/Interpolation.test.cpp b/src/Interpolation.test.cpp index 41a8b70a..764de414 100644 --- a/src/Interpolation.test.cpp +++ b/src/Interpolation.test.cpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #include "aare/ClusterVector.hpp" #include "aare/Interpolator.hpp" #include "aare/NDArray.hpp" diff --git a/src/NDArray.test.cpp b/src/NDArray.test.cpp index 3a3cafc0..872f22cd 100644 --- a/src/NDArray.test.cpp +++ b/src/NDArray.test.cpp @@ -86,6 +86,40 @@ TEST_CASE("A const NDArray returns a read-only view") { REQUIRE(view(1, 2) == 1); } +TEST_CASE("NDArray can be constructed from a read-only view") { + NDArray source({2, 3}, 0); + for (ssize_t i = 0; i < source.size(); ++i) + source(i) = static_cast(i); + const auto &const_source = source; + + NDArray from_const(const_source.view()); + NDArray from_mutable(source.view()); + + REQUIRE(from_const.shape() == source.shape()); + REQUIRE(from_mutable.shape() == source.shape()); + for (ssize_t i = 0; i < source.size(); ++i) { + REQUIRE(from_const(i) == source(i)); + REQUIRE(from_mutable(i) == source(i)); + } + + // Independent copy, not a view of the source + source(0) = 42; + REQUIRE(from_const(0) == 0); +} + +TEST_CASE("NDArray::copy_from accepts a read-only view") { + NDArray source({4}, 1.5); + const auto &const_source = source; + + NDArray target({4}, 0.0); + target.copy_from(const_source.view()); + for (ssize_t i = 0; i < target.size(); ++i) + REQUIRE(target(i) == 1.5); + + NDArray wrong_shape({3}, 0.0); + REQUIRE_THROWS(wrong_shape.copy_from(const_source.view())); +} + TEST_CASE("Indexing of a 2D image") { std::array shape{{3, 7}}; NDArray img(shape, 5); @@ -412,7 +446,7 @@ TEST_CASE("Elementwise operations on images") { NDArray A(shape, a_val); NDArray B(shape, b_val); double v = 1.0; - auto C = A - v; + NDArray C = A - v; REQUIRE(C.data() != A.data()); // Value of C matches @@ -429,7 +463,7 @@ TEST_CASE("Elementwise operations on images") { NDArray A(shape, a_val); NDArray B(shape, b_val); double v = 1.0; - auto C = A + v; + NDArray C = A + v; REQUIRE(C.data() != A.data()); // Value of C matches @@ -446,7 +480,7 @@ TEST_CASE("Elementwise operations on images") { NDArray A(shape, a_val); NDArray B(shape, b_val); double v = 3.7; - auto C = A / v; + NDArray C = A / v; REQUIRE(C.data() != A.data()); // Value of C matches @@ -463,12 +497,12 @@ TEST_CASE("Elementwise operations on images") { NDArray A(shape, a_val); NDArray B(shape, b_val); double v = 3.7; - auto C = A / v; + NDArray C = A * v; REQUIRE(C.data() != A.data()); // Value of C matches for (uint32_t i = 0; i < C.size(); ++i) { - REQUIRE(C(i) == a_val / v); + REQUIRE(C(i) == a_val * v); } // Value of A is not changed @@ -478,6 +512,51 @@ TEST_CASE("Elementwise operations on images") { } } +TEST_CASE("Expressions with scalars") { + NDArray A({5, 5}, 3); + NDArray B({5, 5}, 8); + + NDArray C = 2 * A + B / 4 - 1; + for (ssize_t i = 0; i < C.size(); ++i) { + REQUIRE(C(i) == 7); + } + + // The scalar is not converted to the element type of the array + NDArray D = A * 0.5; + for (ssize_t i = 0; i < D.size(); ++i) { + REQUIRE(D(i) == 1.5); + } +} + +TEST_CASE("Assign an expression to an existing NDArray") { + NDArray A({5, 5}, 3); + NDArray B({5, 5}, 8); + + SECTION("same shape reuses the buffer") { + NDArray C({5, 5}, 0); + auto *ptr = C.data(); + C = A + B; + REQUIRE(C.data() == ptr); + for (ssize_t i = 0; i < C.size(); ++i) { + REQUIRE(C(i) == 11); + } + } + SECTION("the target can be part of the expression") { + A = A + B; + for (ssize_t i = 0; i < A.size(); ++i) { + REQUIRE(A(i) == 11); + } + } + SECTION("different shape reallocates") { + NDArray C({2, 3}, 0); + C = A + B; + REQUIRE(C.shape() == A.shape()); + for (ssize_t i = 0; i < C.size(); ++i) { + REQUIRE(C(i) == 11); + } + } +} + TEST_CASE("Assign an std::array to a 1D NDArray") { NDArray a{{5}, 0}; std::array b{1, 2, 3, 4, 5}; diff --git a/src/NumpyHelpers.cpp b/src/NumpyHelpers.cpp index a5448338..75462953 100644 --- a/src/NumpyHelpers.cpp +++ b/src/NumpyHelpers.cpp @@ -1,4 +1,4 @@ -// SPDX-License-Identifier: MPL-2.0 +// SPDX-License-Identifier: MIT /* 28-03-2024 modified by: Bechir Braham diff --git a/src/Pedestal.test.cpp b/src/Pedestal.test.cpp index 099c47cf..abab15d2 100644 --- a/src/Pedestal.test.cpp +++ b/src/Pedestal.test.cpp @@ -1,12 +1,35 @@ // SPDX-License-Identifier: MPL-2.0 #include "aare/Pedestal.hpp" +#include #include #include #include #include using namespace aare; + +TEMPLATE_TEST_CASE("pedestal uses double precision moments", "[pedestal]", + double, float, int16_t) { + Pedestal pedestal(1, 1, 2); + pedestal.push(0, 0, uint16_t{30000}); + pedestal.push(0, 0, uint16_t{30003}); + + REQUIRE(pedestal.mean(0, 0) == static_cast(30001.5)); + REQUIRE(pedestal.std(0, 0) == static_cast(1.5)); + + pedestal.push(0, 0, uint16_t{30002}); + + REQUIRE(pedestal.mean(0, 0) == static_cast(30001.75)); + // Static cast to double is needed to avoid a Catch2 warning about + // float to double conversion on mac. + REQUIRE_THAT( + static_cast(pedestal.std(0, 0)), + Catch::Matchers::WithinAbs(static_cast(static_cast( + std::sqrt(TestType(1.1875)))), + static_cast(TestType(1e-6)))); +} + TEST_CASE("test pedestal constructor") { aare::Pedestal pedestal(10, 10, 5); REQUIRE(pedestal.rows() == 10); @@ -14,13 +37,47 @@ TEST_CASE("test pedestal constructor") { REQUIRE(pedestal.n_samples() == 5); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == 0); - REQUIRE(pedestal.get_sum2()(i, j) == 0); + REQUIRE(pedestal.mean(i, j) == 0); REQUIRE(pedestal.cur_samples()(i, j) == 0); } } } +TEST_CASE("pedestal rejects mismatched frame shapes", "[pedestal]") { + Pedestal<> pedestal(2, 3); + pedestal.push(0, 0, uint16_t{7}); + NDArray threshold({2, 3}, 10.0); + + for (const auto shape : + {std::array{2, 2}, std::array{3, 2}}) { + Frame frame(shape[0], shape[1], Dtype::UINT16); + REQUIRE_THROWS_WITH(pedestal.push(frame.view()), + "Frame shape does not match pedestal shape"); + REQUIRE_THROWS_WITH(pedestal.push_with_threshold(frame.view(), + threshold.view()), + "Frame shape does not match pedestal shape"); + REQUIRE_THROWS_WITH(pedestal.push(frame), + "Frame shape does not match pedestal shape"); + REQUIRE(pedestal.mean(0, 0) == 7); + REQUIRE(pedestal.cur_samples()(0, 0) == 1); + } +} + +TEMPLATE_TEST_CASE("pedestal std stays finite after settling", "[pedestal]", + double, float, int16_t) { + Pedestal pedestal(1, 1, 10); + pedestal.push(0, 0, uint16_t{16382}); + for (int i = 0; i < 201; ++i) { + pedestal.push(0, 0, uint16_t{16383}); + } + + const auto noise = pedestal.std(0, 0); + REQUIRE(std::isfinite(noise)); + REQUIRE(noise >= 0); + REQUIRE(static_cast(noise) < 1e-3); + REQUIRE(pedestal.std()(0, 0) == noise); +} + TEST_CASE("test pedestal push") { aare::Pedestal pedestal(10, 10, 5); aare::Frame frame(10, 10, Dtype::UINT16); @@ -34,8 +91,7 @@ TEST_CASE("test pedestal push") { pedestal.push(frame); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == i + j); - REQUIRE(pedestal.get_sum2()(i, j) == (i + j) * (i + j)); + REQUIRE(pedestal.mean(i, j) == i + j); REQUIRE(pedestal.cur_samples()(i, j) == 1); } } @@ -44,8 +100,7 @@ TEST_CASE("test pedestal push") { pedestal.clear(); for (int i = 0; i < 10; i++) { for (int j = 0; j < 10; j++) { - REQUIRE(pedestal.get_sum()(i, j) == 0); - REQUIRE(pedestal.get_sum2()(i, j) == 0); + REQUIRE(pedestal.mean(i, j) == 0); REQUIRE(pedestal.cur_samples()(i, j) == 0); } } @@ -57,16 +112,10 @@ TEST_CASE("test pedestal push") { for (uint32_t j = 0; j < 10; j++) { if (k < 5) { REQUIRE(pedestal.cur_samples()(i, j) == k + 1); - REQUIRE(pedestal.get_sum()(i, j) == (k + 1) * (i + j)); - REQUIRE(pedestal.get_sum2()(i, j) == - (k + 1) * (i + j) * (i + j)); } else { REQUIRE(pedestal.cur_samples()(i, j) == 5); - REQUIRE(pedestal.get_sum()(i, j) == 5 * (i + j)); - REQUIRE(pedestal.get_sum2()(i, j) == 5 * (i + j) * (i + j)); } REQUIRE(pedestal.mean(i, j) == (i + j)); - REQUIRE(pedestal.variance(i, j) == 0); REQUIRE(pedestal.std(i, j) == 0); } } @@ -74,7 +123,7 @@ TEST_CASE("test pedestal push") { } TEST_CASE("test pedestal with normal distribution") { - const double MEAN = 5.0, STD = 2.0, VAR = STD * STD, TOLERANCE = 0.1; + const double MEAN = 5.0, STD = 2.0, TOLERANCE = 0.1; unsigned seed = std::chrono::system_clock::now().time_since_epoch().count(); std::default_random_engine generator(seed); @@ -91,17 +140,14 @@ TEST_CASE("test pedestal with normal distribution") { pedestal.push(frame); } auto mean = pedestal.mean(); - auto variance = pedestal.variance(); auto standard_deviation = pedestal.std(); for (int i = 0; i < 3; i++) { for (int j = 0; j < 5; j++) { REQUIRE_THAT(mean(i, j), Catch::Matchers::WithinAbs(MEAN, MEAN * TOLERANCE)); - REQUIRE_THAT(variance(i, j), - Catch::Matchers::WithinAbs(VAR, VAR * TOLERANCE)); REQUIRE_THAT(standard_deviation(i, j), Catch::Matchers::WithinAbs(STD, STD * TOLERANCE)); } } -} \ No newline at end of file +} diff --git a/src/ProducerConsumerQueue.test.cpp b/src/ProducerConsumerQueue.test.cpp new file mode 100644 index 00000000..05cb4904 --- /dev/null +++ b/src/ProducerConsumerQueue.test.cpp @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: MPL-2.0 + +#include "aare/ProducerConsumerQueue.hpp" + +#include +#include +#include +#include + +using aare::ProducerConsumerQueue; + +// The queue is a fixed-capacity SPSC ring with const size and buffer, as in +// the upstream folly implementation. Moving one is not thread safe and the +// previous move operations left the source pointing at a null buffer, so the +// type is deliberately neither copyable nor movable. Hold it by value or in a +// unique_ptr instead. +static_assert(!std::is_copy_constructible_v>); +static_assert(!std::is_copy_assignable_v>); +static_assert(!std::is_move_constructible_v>); +static_assert(!std::is_move_assignable_v>); + +TEST_CASE("ProducerConsumerQueue round trips non trivial elements", "[queue]") { + ProducerConsumerQueue q(3); // capacity 2 + REQUIRE(q.capacity() == 2); + REQUIRE(q.isEmpty()); + + REQUIRE(q.write(std::string(64, 'a'))); + REQUIRE(q.write(std::string(64, 'b'))); + CHECK(q.isFull()); + CHECK_FALSE(q.write(std::string(64, 'c'))); + + std::string out; + REQUIRE(q.read(out)); + CHECK(out == std::string(64, 'a')); + + // Wrap around the ring and leave an element behind for the destructor. + REQUIRE(q.write(std::string(64, 'd'))); + REQUIRE(q.frontPtr() != nullptr); + CHECK(*q.frontPtr() == std::string(64, 'b')); + q.popFront(); + CHECK(q.sizeGuess() == 1); +} + +TEST_CASE("ProducerConsumerQueue rejects sizes below two in every build type", + "[queue]") { + // A plain assert would be compiled out in Release builds and a zero-size + // queue would then overflow its buffer on the first write. + CHECK_THROWS_AS(ProducerConsumerQueue(0), std::invalid_argument); + CHECK_THROWS_AS(ProducerConsumerQueue(1), std::invalid_argument); + + ProducerConsumerQueue smallest(2); + CHECK(smallest.capacity() == 1); + CHECK(smallest.write(7)); + CHECK(smallest.isFull()); +} diff --git a/src/ROI.cpp b/src/ROI.cpp new file mode 100644 index 00000000..7e91c49d --- /dev/null +++ b/src/ROI.cpp @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/ROI.hpp" +#include "aare/DetectorGeometry.hpp" +#include "aare/ROIGeometry.hpp" + +namespace aare { + +/** + * @brief Check if the ROI covers the entire detector geometry + * @param roi Region of interest + * @param geometry Detector geometry + * @return true if the ROI covers the entire detector geometry, false otherwise + */ +bool complete_ROI(const ROI &roi, const DetectorGeometry &geometry) { + return roi.xmin == 0 && + roi.xmax == static_cast(geometry.pixels_x()) && + roi.ymin == 0 && + roi.ymax == static_cast(geometry.pixels_y()); +} + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry) { + if (rois.empty() or rois.size() > 1) { + return false; + } else { + return complete_ROI(rois[0], geometry); + } +} + +bool complete_ROI(const ROIGeometry &roi, const DetectorGeometry &geometry) { + return roi.pixels_x() == geometry.pixels_x() && + roi.pixels_y() == geometry.pixels_y(); +} + +bool complete_ROI(const std::vector &rois, + const DetectorGeometry &geometry) { + if (rois.empty() or rois.size() > 1) { + return false; + } else { + return complete_ROI(rois[0], geometry); + } +} + +} // namespace aare \ No newline at end of file diff --git a/src/ROI.test.cpp b/src/ROI.test.cpp new file mode 100644 index 00000000..9bbe4dff --- /dev/null +++ b/src/ROI.test.cpp @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/ROI.hpp" +#include + +namespace aare { + +TEST_CASE("merge ROIs", "[utility_functions]") { + + SECTION("not fully contiguous") { + + std::vector rois = {ROI{20, 30, 50, 60}, ROI{20, 30, 40, 50}, + ROI{10, 20, 40, 50}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 2); + + REQUIRE(merged_rois[0] == ROI{10, 30, 40, 50}); + REQUIRE(merged_rois[1] == ROI{20, 30, 50, 60}); + } + SECTION("complex merge") { + + std::vector rois = {ROI{40, 50, 20, 30}, ROI{10, 20, 30, 40}, + ROI{10, 20, 50, 60}, ROI{20, 30, 30, 40}, + ROI{60, 70, 30, 40}, ROI{60, 70, 20, 30}, + ROI{20, 30, 20, 30}, ROI{10, 20, 20, 30}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 4); + + REQUIRE(merged_rois[0] == ROI{10, 30, 20, 40}); + REQUIRE(merged_rois[1] == ROI{10, 20, 50, 60}); + REQUIRE(merged_rois[2] == ROI{40, 50, 20, 30}); + REQUIRE(merged_rois[3] == ROI{60, 70, 20, 40}); + } + SECTION("horizontally aligned") { + std::vector rois = {ROI{10, 20, 30, 40}, ROI{10, 20, 50, 60}, + ROI{10, 20, 20, 30}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 2); + REQUIRE(merged_rois[0] == ROI{10, 20, 20, 40}); + REQUIRE(merged_rois[1] == ROI{10, 20, 50, 60}); + } + SECTION("vertically aligned") { + std::vector rois = {ROI{10, 20, 30, 40}, ROI{30, 40, 30, 40}, + ROI{20, 30, 30, 40}}; + + auto merged_rois = merge_consecutive_rois(rois); + + REQUIRE(merged_rois.size() == 1); + REQUIRE(merged_rois[0] == ROI{10, 40, 30, 40}); + } +} + +} // namespace aare diff --git a/src/ROIGeometry.cpp b/src/ROIGeometry.cpp index 04cf9187..8008118d 100644 --- a/src/ROIGeometry.cpp +++ b/src/ROIGeometry.cpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #include "aare/ROIGeometry.hpp" #include @@ -5,25 +6,24 @@ namespace aare { ROIGeometry::ROIGeometry(const ROI &roi, DetectorGeometry &geometry) : m_pixels_x(roi.width()), m_pixels_y(roi.height()), m_geometry(geometry) { - m_module_indices_in_roi.reserve(m_geometry.n_modules()); - // determine which modules are in the roi - for (size_t i = 0; i < m_geometry.n_modules(); ++i) { - auto &module_geometry = m_geometry.get_module_geometries(i); - if (module_geometry.module_in_roi(roi)) { - module_geometry.update_geometry_with_roi(roi); - m_module_indices_in_roi.push_back(i); + + if (complete_ROI({roi}, geometry)) { + m_module_indices_in_roi.resize(m_geometry.n_modules()); + std::iota(m_module_indices_in_roi.begin(), + m_module_indices_in_roi.end(), 0); + } else { + m_module_indices_in_roi.reserve(m_geometry.n_modules()); + // determine which modules are in the roi + for (size_t i = 0; i < m_geometry.n_modules(); ++i) { + auto &module_geometry = m_geometry.get_module_geometries(i); + if (module_geometry.module_in_roi(roi)) { + module_geometry.update_geometry_with_roi(roi); + m_module_indices_in_roi.push_back(i); + } } } } -ROIGeometry::ROIGeometry(DetectorGeometry &geometry) - : m_pixels_x(geometry.pixels_x()), m_pixels_y(geometry.pixels_y()), - m_geometry(geometry) { - m_module_indices_in_roi.resize(m_geometry.n_modules()); - std::iota(m_module_indices_in_roi.begin(), m_module_indices_in_roi.end(), - 0); -} - size_t ROIGeometry::num_modules_in_roi() const { return m_module_indices_in_roi.size(); } diff --git a/src/RawFile.cpp b/src/RawFile.cpp index cecf3e82..1e33bf83 100644 --- a/src/RawFile.cpp +++ b/src/RawFile.cpp @@ -2,11 +2,13 @@ #include "aare/RawFile.hpp" #include "aare/DetectorGeometry.hpp" #include "aare/PixelMap.hpp" +#include "aare/ROI.hpp" #include "aare/ROIGeometry.hpp" #include "aare/algorithm.hpp" #include "aare/defs.hpp" #include "aare/logger.hpp" +#include #include #include @@ -14,36 +16,62 @@ using json = nlohmann::json; namespace aare { +std::runtime_error RawFile::frame_error(size_t frame_index, + const std::string &message) const { + return std::runtime_error( + fmt::format("Error reading frame index {} from file '{}': {}", + frame_index, m_master.master_fname().string(), message)); +} + RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode) - : m_master(fname), - m_geometry(m_master.geometry(), m_master.pixels_x(), m_master.pixels_y(), - m_master.udp_interfaces_per_module(), m_master.quad()) { + : m_master(fname) { m_mode = mode; - m_subfiles.resize(m_master.rois().has_value() ? m_master.rois()->size() - : 1); - if (mode == "r") { - if (m_master.rois().has_value()) { - m_ROI_geometries.reserve(m_master.rois()->size()); - - // iterate over all ROIS - size_t roi_index = 0; - const auto rois = m_master.rois().value(); - for (const auto &roi : rois) { - m_ROI_geometries.push_back(ROIGeometry(roi, m_geometry)); - // open subfiles - open_subfiles(roi_index); - ++roi_index; - } - - } else { - // no ROI use full detector - m_ROI_geometries.reserve(1); - m_ROI_geometries.push_back(ROIGeometry(m_geometry)); - open_subfiles(0); + if (m_master.frame_padding() == 0 && + m_master.frame_discard_policy() != + FrameDiscardPolicy::DiscardPartial) { + throw std::runtime_error(fmt::format( + "Cannot open '{}': RawFile requires frame padding or " + "discardpartial.", + m_master.master_fname().string())); } + + m_subfiles.resize(m_master.roi_geometries().size()); + // iterate over all ROIS + const size_t num_rois = m_master.roi_geometries().size(); + + for (size_t roi_index = 0; roi_index < num_rois; ++roi_index) { + // open subfiles + open_subfiles(roi_index); + } + + std::optional min_frames; + size_t max_frames = 0; + for (const auto &subfiles : m_subfiles) { + for (const auto &subfile : subfiles) { + const auto count = subfile->frames_in_file(); + min_frames = min_frames ? std::min(*min_frames, count) : count; + max_frames = std::max(max_frames, count); + } + } + if (!min_frames) { + throw std::runtime_error(fmt::format( + "No raw subfiles selected by '{}'", fname.string())); + } + m_frames_in_file = *min_frames; + + if (m_frames_in_file != max_frames || + m_frames_in_file != m_master.frames_in_file()) { + LOG(logWARNING) << fmt::format( + "'{}': Different number of frames across subfiles. Expected {} " + "frames but found min/max {}/{}, " + "using {} frames.", + fname.string(), m_master.frames_in_file(), m_frames_in_file, + max_frames, m_frames_in_file); + } + LOG(logDEBUG) << "Frames in file: " << m_frames_in_file; } else { throw std::runtime_error(LOCATION + " Unsupported mode. Can only read RawFiles."); @@ -52,24 +80,15 @@ RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode) Frame RawFile::read_roi(const size_t roi_index) { - if (!m_master.rois()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } - if (roi_index >= m_ROI_geometries.size()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + if (roi_index >= m_master.roi_geometries().size()) { + throw frame_error(m_current_frame, + LOCATION + "ROI index out of range."); } return get_frame(m_current_frame++, roi_index); } std::vector RawFile::read_rois() { - - if (!m_master.rois()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } - - const size_t num_rois = m_ROI_geometries.size(); + const size_t num_rois = m_master.roi_geometries().size(); std::vector frames; frames.reserve(num_rois); @@ -83,18 +102,19 @@ std::vector RawFile::read_rois() { } Frame RawFile::read_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Multiple ROIs defined in the master file. " - "Use read_ROIs() instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, LOCATION + + "Multiple ROIs present in file. " + "Use read_ROIs() instead."); } return get_frame(m_current_frame++); } Frame RawFile::read_frame(size_t frame_number) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error( - LOCATION + "Multiple ROIs defined in the master file. " + if (m_master.roi_geometries().size() > 1) { + throw frame_error( + frame_number, + LOCATION + "Multiple ROIs present in file. " "Use read_ROIs(const size_t frame_number) instead."); } seek(frame_number); @@ -103,9 +123,9 @@ Frame RawFile::read_frame(size_t frame_number) { void RawFile::read_into(std::byte *image_buf, size_t n_frames) { // TODO: implement this in a more efficient way - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + "Cannot use read_into for multiple ROIs."); } for (size_t i = 0; i < n_frames; i++) { @@ -115,31 +135,29 @@ void RawFile::read_into(std::byte *image_buf, size_t n_frames) { } void RawFile::read_into(std::byte *image_buf) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs. Use " - "read_roi_into() for a single ROI instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + + "Cannot use read_into for multiple ROIs. Use " + "read_roi_into() for a single ROI instead."); } return get_frame_into(m_current_frame++, image_buf); } void RawFile::read_roi_into(std::byte *image_buf, const size_t roi_index, const size_t frame_number, DetectorHeader *header) { - if (!m_master.rois().has_value()) { - throw std::runtime_error(LOCATION + - "No ROIs defined in the master file."); - } if (roi_index >= num_rois()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + throw frame_error(frame_number, LOCATION + "ROI index out of range."); } return get_frame_into(frame_number, image_buf, roi_index, header); } void RawFile::read_into(std::byte *image_buf, DetectorHeader *header) { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error(LOCATION + - "Cannot use read_into for multiple ROIs. Use " - "read_roi_into() for a single ROI instead."); + if (m_master.roi_geometries().size() > 1) { + throw frame_error(m_current_frame, + LOCATION + + "Cannot use read_into for multiple ROIs. Use " + "read_roi_into() for a single ROI instead."); } return get_frame_into(m_current_frame++, image_buf, 0, header); } @@ -148,25 +166,29 @@ void RawFile::read_into(std::byte *image_buf, size_t n_frames, DetectorHeader *header) { // return get_frame_into(m_current_frame++, image_buf, header); - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { - throw std::runtime_error( + if (m_master.roi_geometries().size() > 1) { + throw frame_error( + m_current_frame, LOCATION + - "Cannot use read_into for multiple ROIs."); // TODO: maybe pass - // roi_index so one can - // use read_into for a - // specific ROI + "Cannot use read_into for multiple ROIs."); // TODO: maybe + // pass + // roi_index so + // one can use + // read_into for + // a specific + // ROI } for (size_t i = 0; i < n_frames; i++) { this->get_frame_into(m_current_frame++, image_buf, 0, header); image_buf += bytes_per_frame(); if (header) - header += m_ROI_geometries[0].num_modules_in_roi(); + header += m_master.roi_geometries()[0].num_modules_in_roi(); } } size_t RawFile::bytes_per_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error( LOCATION + "Pass the desired roi_index to bytes_per_frame to get " "bytes_per_frame for the specific ROI. "); @@ -175,13 +197,13 @@ size_t RawFile::bytes_per_frame() { } size_t RawFile::bytes_per_frame(const size_t roi_index) { - return m_ROI_geometries.at(roi_index).pixels_x() * - m_ROI_geometries.at(roi_index).pixels_y() * m_master.bitdepth() / - bits_per_byte; + return m_master.roi_geometries().at(roi_index).pixels_x() * + m_master.roi_geometries().at(roi_index).pixels_y() * + m_master.bitdepth() / bits_per_byte; } size_t RawFile::pixels_per_frame() { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error( LOCATION + "Pass the desired roi_index to pixels_per_frame to get " "pixels_per_frame for the specific ROI. "); @@ -190,8 +212,8 @@ size_t RawFile::pixels_per_frame() { } size_t RawFile::pixels_per_frame(const size_t roi_index) { - return m_ROI_geometries.at(roi_index).pixels_x() * - m_ROI_geometries.at(roi_index).pixels_y(); + return m_master.roi_geometries().at(roi_index).pixels_x() * + m_master.roi_geometries().at(roi_index).pixels_y(); } DetectorType RawFile::detector_type() const { return m_master.detector_type(); } @@ -201,7 +223,8 @@ void RawFile::seek(size_t frame_index) { // if frame_number == total_frames, then the next read will throw an // error if (frame_index > total_frames()) { - throw std::runtime_error( + throw frame_error( + frame_index, fmt::format("frame number {} is greater than total frames {}", frame_index, total_frames())); } @@ -210,10 +233,10 @@ void RawFile::seek(size_t frame_index) { size_t RawFile::tell() { return m_current_frame; } -size_t RawFile::total_frames() const { return m_master.frames_in_file(); } +size_t RawFile::total_frames() const { return m_frames_in_file; } size_t RawFile::rows() const { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error(LOCATION + "Pass the desired roi_index to rows to get " "rows for the specific ROI. "); @@ -221,10 +244,10 @@ size_t RawFile::rows() const { return rows(0); } size_t RawFile::rows(const size_t roi_index) const { - return m_ROI_geometries.at(roi_index).pixels_y(); + return m_master.roi_geometries().at(roi_index).pixels_y(); } size_t RawFile::cols() const { - if (m_master.rois().has_value() && m_master.rois()->size() > 1) { + if (m_master.roi_geometries().size() > 1) { throw std::runtime_error(LOCATION + "Pass the desired roi_index to cols to get " "cols for the specific ROI. "); @@ -232,50 +255,41 @@ size_t RawFile::cols() const { return cols(0); } size_t RawFile::cols(const size_t roi_index) const { - return m_ROI_geometries.at(roi_index).pixels_x(); + return m_master.roi_geometries().at(roi_index).pixels_x(); } size_t RawFile::bitdepth() const { return m_master.bitdepth(); } -xy RawFile::geometry() const { - return xy{static_cast(m_geometry.modules_y()), - static_cast(m_geometry.modules_x())}; -} -size_t RawFile::n_modules() const { return m_geometry.n_modules(); }; +xy RawFile::geometry() const { return m_master.detector_layout(); } -size_t RawFile::num_rois() const { - if (m_master.rois().has_value()) { - return m_master.rois()->size(); - } else { - return 0; - } -} +size_t RawFile::n_modules() const { return m_master.n_modules(); }; + +size_t RawFile::num_rois() const { return m_master.roi_geometries().size(); } const ROIGeometry &RawFile::roi_geometries(size_t roi_index) const { - return m_ROI_geometries[roi_index]; + return m_master.roi_geometries().at(roi_index); } std::vector RawFile::n_modules_in_roi() const { - std::vector results(m_ROI_geometries.size()); + std::vector results(m_master.roi_geometries().size()); std::transform( - m_ROI_geometries.begin(), m_ROI_geometries.end(), results.begin(), + m_master.roi_geometries().begin(), m_master.roi_geometries().end(), + results.begin(), [](const ROIGeometry &roi) { return roi.num_modules_in_roi(); }); return results; } void RawFile::open_subfiles(const size_t roi_index) { - if (m_mode == "r") { m_subfiles[roi_index].reserve( - m_ROI_geometries[roi_index].num_modules_in_roi()); + m_master.roi_geometries().at(roi_index).num_modules_in_roi()); auto module_indices = - m_ROI_geometries[roi_index].module_indices_in_roi(); + m_master.roi_geometries().at(roi_index).module_indices_in_roi(); - for (const size_t i : - m_ROI_geometries[roi_index].module_indices_in_roi()) { - const auto pos = m_geometry.get_module_geometries(i); + for (const size_t i : module_indices) { + const auto pos = m_master.geometry().get_module_geometries(i); m_subfiles[roi_index].emplace_back(std::make_unique( m_master.data_fname(i, 0), m_master.detector_type(), pos.height, pos.width, m_master.bitdepth(), pos.row_index, pos.col_index)); @@ -290,14 +304,22 @@ DetectorHeader RawFile::read_header(const std::filesystem::path &fname) { DetectorHeader h{}; FILE *fp = fopen(fname.string().c_str(), "r"); if (!fp) - throw std::runtime_error( - fmt::format("Could not open: {} for reading", fname.string())); + throw std::runtime_error(fmt::format( + "Could not open file '{}' for frame index 0", fname.string())); size_t const rc = fread(reinterpret_cast(&h), sizeof(h), 1, fp); - if (rc != 1) - throw std::runtime_error(LOCATION + "Could not read header from file"); + if (rc != 1) { + fclose(fp); + throw std::runtime_error( + LOCATION + fmt::format("Could not read header for frame index 0 " + "from file '{}'", + fname.string())); + } if (fclose(fp)) { - throw std::runtime_error(LOCATION + "Could not close file"); + throw std::runtime_error( + LOCATION + fmt::format("Could not close file '{}' after reading " + "frame index 0", + fname.string())); } return h; @@ -306,8 +328,8 @@ DetectorHeader RawFile::read_header(const std::filesystem::path &fname) { RawMasterFile RawFile::master() const { return m_master; } Frame RawFile::get_frame(size_t frame_index, const size_t roi_index) { - auto f = Frame(m_ROI_geometries[roi_index].pixels_y(), - m_ROI_geometries[roi_index].pixels_x(), + auto f = Frame(m_master.roi_geometries().at(roi_index).pixels_y(), + m_master.roi_geometries().at(roi_index).pixels_x(), Dtype::from_bitdepth(m_master.bitdepth())); std::byte *frame_buffer = f.data(); get_frame_into(frame_index, frame_buffer, roi_index); @@ -320,19 +342,26 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, const size_t roi_index, DetectorHeader *header) { LOG(logDEBUG) << "RawFile::get_frame_into(" << frame_index << ")"; if (frame_index >= total_frames()) { - throw std::runtime_error(LOCATION + "Frame number out of range"); + throw frame_error( + frame_index, + LOCATION + + fmt::format("Frame index {} is out of range: file contains {} " + "frames (indices are zero-based)", + frame_index, total_frames())); } std::vector frame_numbers( - m_ROI_geometries[roi_index].num_modules_in_roi()); + m_master.roi_geometries().at(roi_index).num_modules_in_roi()); std::vector frame_indices( - m_ROI_geometries[roi_index].num_modules_in_roi(), frame_index); + m_master.roi_geometries().at(roi_index).num_modules_in_roi(), + frame_index); // sync the frame numbers - if (m_ROI_geometries[roi_index].num_modules_in_roi() != + if (m_master.roi_geometries().at(roi_index).num_modules_in_roi() != 1) { // if we have more than one module for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { frame_numbers[part_idx] = m_subfiles[roi_index][part_idx]->frame_number(frame_index); @@ -352,8 +381,17 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, // 4. if we can't increase its index => throw error if (frame_indices[min_frame_idx] >= total_frames()) { - throw std::runtime_error(LOCATION + - "Frame number out of range"); + throw frame_error( + frame_index, + LOCATION + + fmt::format( + "Frame index {} out of range while synchronizing " + "module {} for ROI {}; last data file '{}'", + frame_indices[min_frame_idx], min_frame_idx, + roi_index, + m_subfiles[roi_index][min_frame_idx] + ->current_path() + .string())); } frame_numbers[min_frame_idx] = @@ -362,41 +400,45 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, } } - if (m_master.geometry().col == 1) { + if (m_master.detector_layout().col == 1) { // get the part from each subfile and copy it to the frame for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { auto corrected_idx = frame_indices[part_idx]; // This is where we start writing auto offset = - (m_geometry + (m_master.geometry() .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_y * - m_ROI_geometries[roi_index].pixels_x() + - m_geometry + m_master.roi_geometries().at(roi_index).pixels_x() + + m_master.geometry() .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_x) * m_master.bitdepth() / 8; - if (m_geometry - .get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi( - part_idx)) + if (m_master.geometry() + .get_module_geometries(m_master.roi_geometries() + .at(roi_index) + .module_indices_in_roi(part_idx)) .origin_x != 0) - throw std::runtime_error( + throw frame_error( + frame_index, LOCATION + - " Implementation error. x pos not 0."); // TODO: - // origin - // can still - // change if - // roi - // changes + " Implementation error. x pos not 0."); // TODO: + // origin + // can still + // change if + // roi + // changes // TODO! What if the files don't match? m_subfiles[roi_index][part_idx]->seek(corrected_idx); m_subfiles[roi_index][part_idx]->read_into(frame_buffer + offset, @@ -420,10 +462,12 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, // the module level for (size_t part_idx = 0; - part_idx != m_ROI_geometries[roi_index].num_modules_in_roi(); + part_idx != + m_master.roi_geometries().at(roi_index).num_modules_in_roi(); ++part_idx) { - auto pos = m_geometry.get_module_geometries( - m_ROI_geometries[roi_index].module_indices_in_roi(part_idx)); + auto pos = m_master.geometry().get_module_geometries( + m_master.roi_geometries().at(roi_index).module_indices_in_roi( + part_idx)); auto corrected_idx = frame_indices[part_idx]; m_subfiles[roi_index][part_idx]->seek(corrected_idx); @@ -437,7 +481,8 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, auto irow = (pos.origin_y + cur_row); auto icol = pos.origin_x; auto dest = - (irow * m_ROI_geometries[roi_index].pixels_x() + icol); + (irow * m_master.roi_geometries().at(roi_index).pixels_x() + + icol); dest = dest * m_master.bitdepth() / 8; memcpy(frame_buffer + dest, part_buffer + @@ -452,12 +497,12 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer, std::vector RawFile::read_n(size_t n_frames) { // TODO: implement this in a more efficient way if (num_rois() > 1) { - throw std::runtime_error(LOCATION + - "Multiple ROIs defined in the master " - "file. Use " - "read_num_rois for a specific ROI or use " - "read_ROIs to read one frame after " - "the other."); + throw frame_error(m_current_frame, + LOCATION + "Multiple ROIs defined in the master " + "file. Use " + "read_num_rois for a specific ROI or use " + "read_ROIs to read one frame after " + "the other."); } std::vector frames; @@ -472,7 +517,8 @@ std::vector RawFile::read_n(size_t n_frames) { std::vector RawFile::read_n_with_roi(const size_t n_frames, const size_t roi_index) { if (roi_index >= num_rois()) { - throw std::runtime_error(LOCATION + "ROI index out of range."); + throw frame_error(m_current_frame, + LOCATION + "ROI index out of range."); } std::vector frames; @@ -485,8 +531,8 @@ std::vector RawFile::read_n_with_roi(const size_t n_frames, } size_t RawFile::frame_number(size_t frame_index) { - if (frame_index >= m_master.frames_in_file()) { - throw std::runtime_error(LOCATION + " Frame number out of range"); + if (frame_index >= total_frames()) { + throw frame_error(frame_index, "Frame number out of range"); } return m_subfiles[0][0]->frame_number(frame_index); } diff --git a/src/RawFile.test.cpp b/src/RawFile.test.cpp index 88efc027..ee0b9e24 100644 --- a/src/RawFile.test.cpp +++ b/src/RawFile.test.cpp @@ -5,8 +5,10 @@ #include #include +#include #include +#include "raw_file_helpers.hpp" #include "test_config.hpp" #include "test_macros.hpp" @@ -14,6 +16,212 @@ using aare::File; using aare::RawFile; using namespace aare; +TEST_CASE("RawFile requires padding or discarding partial frames", + "[RawFile][frame-policy]") { + const auto [padding, policy, supported] = + GENERATE(table({ + {0, "nodiscard", false}, + {0, "discard", false}, + {0, "discardpartial", true}, + {1, "nodiscard", true}, + {1, "discard", true}, + {1, "discardpartial", true}, + })); + CAPTURE(padding, policy); + TemporaryRawFiles files; + nlohmann::json metadata; + std::ifstream(files.master_path()) >> metadata; + metadata["Frame Padding"] = padding; + metadata["Frame Discard Policy"] = policy; + std::ofstream(files.master_path()) << metadata; + + REQUIRE(RawMasterFile(files.master_path()).frame_padding() == padding); + if (supported) { + RawFile reader(files.master_path()); + REQUIRE(reader.total_frames() == 2); + auto frame = reader.read_frame(); + REQUIRE(frame.view()(0, 0) == 1); + File generic_reader(files.master_path()); + auto generic_frame = generic_reader.read_frame(); + REQUIRE(generic_frame.view()(0, 0) == 1); + } else { + SECTION("with data subfiles") {} + SECTION("without data subfiles") { + std::filesystem::remove(files.data_path(0, 0)); + std::filesystem::remove(files.data_path(0, 1)); + } + const auto path_matcher = + Catch::Matchers::ContainsSubstring(files.master_path().string()); + const auto policy_matcher = Catch::Matchers::ContainsSubstring( + "requires frame padding or discardpartial"); + const auto message = path_matcher && policy_matcher; + REQUIRE_THROWS_AS(RawFile(files.master_path()), std::runtime_error); + REQUIRE_THROWS_WITH(RawFile(files.master_path()), message); + REQUIRE_THROWS_WITH(File(files.master_path()), message); + } +} + +TEST_CASE("RawFile read errors identify the frame index and master path", + "[RawFile][read-errors]") { + TemporaryRawFiles files; + RawFile file(files.master_path()); + REQUIRE(file.frame_number(1) == 101); + file.seek(2); + const auto index_matcher = + Catch::Matchers::ContainsSubstring("frame index 2"); + const auto path_matcher = + Catch::Matchers::ContainsSubstring(files.master_path().string()); + const auto context = index_matcher && path_matcher; + std::vector buffer(file.bytes_per_frame() * 2); + DetectorHeader headers[2]{}; + + SECTION("sequential frame") { + REQUIRE_THROWS_WITH( + file.read_frame(), + index_matcher && path_matcher && + Catch::Matchers::ContainsSubstring( + "file contains 2 frames (indices are zero-based)") && + !Catch::Matchers::ContainsSubstring(".raw")); + } + SECTION("indexed frame at EOF") { + REQUIRE_THROWS_WITH(file.read_frame(2), context); + } + SECTION("indexed frame beyond EOF") { + REQUIRE_THROWS_WITH( + file.read_frame(17), + Catch::Matchers::ContainsSubstring("frame index 17") && + Catch::Matchers::ContainsSubstring( + files.master_path().string())); + } + SECTION("batch crosses EOF") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_n(2), context); + } + SECTION("read into buffer") { + FileInterface &reader = file; + REQUIRE_THROWS_WITH(reader.read_into(buffer.data()), context); + } + SECTION("batch into buffer") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), size_t{2}), context); + } + SECTION("read with header") { + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), headers), context); + } + SECTION("batch with headers") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), 2, headers), context); + } + SECTION("single ROI") { REQUIRE_THROWS_WITH(file.read_roi(0), context); } + SECTION("all ROIs") { REQUIRE_THROWS_WITH(file.read_rois(), context); } + SECTION("ROI into buffer") { + REQUIRE_THROWS_WITH(file.read_roi_into(buffer.data(), 0, 2), context); + } + SECTION("ROI batch") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_n_with_roi(2, 0), context); + } + SECTION("frame number") { + REQUIRE_THROWS_WITH(file.frame_number(2), context); + } + SECTION("generic File") { + File reader(files.master_path()); + REQUIRE_THROWS_WITH(reader.read_frame(2), context); + } +} + +TEST_CASE("RawFile synchronization errors retain the requested index", + "[RawFile][read-errors]") { + TemporaryRawFiles files(2); + const size_t lagging_module = GENERATE(0, 1); + { + std::fstream output(files.data_path(1 - lagging_module, 1), + std::ios::binary | std::ios::in | std::ios::out); + DetectorHeader header{}; + header.frameNumber = 102; + output.write(reinterpret_cast(&header), sizeof(header)); + } + RawFile file(files.master_path()); + REQUIRE_THROWS_WITH( + file.read_frame(1), + Catch::Matchers::ContainsSubstring("frame index 1") && + Catch::Matchers::ContainsSubstring(files.master_path().string()) && + Catch::Matchers::ContainsSubstring( + files.data_path(lagging_module, 1).string()) && + Catch::Matchers::ContainsSubstring("synchroniz")); +} + +TEST_CASE("RawFile reports the failing data file and requested frame", + "[RawFile][read-errors]") { + TemporaryRawFiles files; + RawFile file(files.master_path()); + std::filesystem::resize_file(files.data_path(0, 1), sizeof(DetectorHeader)); + REQUIRE_THROWS_WITH( + file.read_frame(1), + Catch::Matchers::ContainsSubstring("frame index 1") && + !Catch::Matchers::ContainsSubstring(files.master_path().string()) && + Catch::Matchers::ContainsSubstring( + files.data_path(0, 1).string()) && + Catch::Matchers::ContainsSubstring("End of file")); +} + +TEST_CASE("RawFile bounds use the shortest subfile", + "[RawFile][read-errors][frame-count]") { + TemporaryRawFiles files(2); + const size_t short_module = GENERATE(0, 1); + std::filesystem::remove(files.data_path(short_module, 1)); + RawFile file(files.master_path()); + REQUIRE(file.total_frames() == 1); + REQUIRE(file.master().frames_in_file() == 2); + REQUIRE_NOTHROW(file.read_frame(0)); + const auto expected = Catch::Matchers::ContainsSubstring( + "Error reading frame index 1 from file '" + + files.master_path().string() + "':"); + + SECTION("indexed frame") { + REQUIRE_THROWS_WITH(file.read_frame(1), expected); + } + SECTION("batch") { + file.seek(0); + REQUIRE_THROWS_WITH(file.read_n(2), expected); + } + SECTION("ROI") { + file.seek(1); + REQUIRE_THROWS_WITH(file.read_roi(0), expected); + } + SECTION("generic File") { + File reader(files.master_path()); + REQUIRE_THROWS_WITH(reader.read_frame(1), expected); + } + SECTION("frame number") { + REQUIRE_THROWS_WITH(file.frame_number(1), expected); + } +} + +TEST_CASE("RawFile frame count comes from the complete subfile series", + "[RawFile][frame-count]") { + TemporaryRawFiles files; + const size_t master_frames = GENERATE(0, 1, 2, 5); + auto metadata = nlohmann::json::parse(std::ifstream(files.master_path())); + metadata["Frames in File"] = master_frames; + metadata["Total Frames"] = 1000; + std::ofstream(files.master_path()) << metadata; + + RawFile file(files.master_path()); + REQUIRE(file.total_frames() == 2); + REQUIRE(file.master().frames_in_file() == master_frames); + REQUIRE(file.master().total_frames_expected() == 1000); + REQUIRE(file.frame_number(1) == 101); + REQUIRE_NOTHROW(file.read_frame(1)); + REQUIRE_THROWS(file.frame_number(2)); + REQUIRE_THROWS(file.read_frame(2)); + + File reader(files.master_path()); + REQUIRE(reader.total_frames() == 2); + REQUIRE(reader.frame_number(1) == 101); + REQUIRE_NOTHROW(reader.read_frame(1)); +} + TEST_CASE("Read number of frames from a jungfrau raw file", "[.with-data][RawFile]") { @@ -263,8 +471,9 @@ TEST_CASE("check find_geometry", "[.with-data][RawFile]") { RawMasterFile master_file(fpath); auto geometry = DetectorGeometry( - master_file.geometry(), master_file.pixels_x(), master_file.pixels_y(), - master_file.udp_interfaces_per_module(), master_file.quad()); + master_file.detector_layout(), master_file.pixels_x(), + master_file.pixels_y(), master_file.udp_interfaces_per_module(), + master_file.quad()); CHECK(geometry.modules_x() == test_parameters.modules_x); CHECK(geometry.modules_y() == test_parameters.modules_y); @@ -308,8 +517,8 @@ TEST_CASE("Open multi module file with ROI", RawFile f(fpath, "r"); SECTION("read 2 frames") { - REQUIRE(f.master().roi().value().width() == 256); - REQUIRE(f.master().roi().value().height() == 256); + REQUIRE(f.master().roi().width() == 256); + REQUIRE(f.master().roi().height() == 256); CHECK(f.n_modules() == 2); @@ -400,8 +609,299 @@ TEST_CASE("Read Mythenframe", "[.with-data][RawFile]") { auto fpath = test_data_path() / "raw/newmythen03/run_2_master_1.json"; REQUIRE(std::filesystem::exists(fpath)); RawFile f(fpath); - REQUIRE(f.master().roi().value().width() == 2560); - REQUIRE(f.master().roi().value().height() == 1); + REQUIRE(f.master().roi().width() == 2560); + REQUIRE(f.master().roi().height() == 1); auto frame = f.read_frame(); REQUIRE(frame.cols() == 2560); -} \ No newline at end of file +} + +TEST_CASE("Read Jungfrau frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled top port") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_interfaces_top_disabled_master_6.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 256); + + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + } + + SECTION("disabled bottom port") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_interfaces_bottom_port_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 256, 512}); + } + SECTION("2 modules - top ports disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_modules_2_interfaces_top_ports_disabled_master_2.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); // cannot read frame + // because multiple rois + + auto frame = f.read_rois(); + + REQUIRE(frame.size() == 2); + REQUIRE(frame[0].cols() == 1024); + REQUIRE(frame[0].rows() == 256); + REQUIRE(frame[1].cols() == 1024); + REQUIRE(frame[1].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 512, 768}); + } + SECTION("2 modules - top ports disabled - bottom port disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "2_modules_2_interfaces_disabled_ports_master_1.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 256, 768}); + } + SECTION("4 modules- mixed ports disabled") { + auto fpath = test_data_path() / "raw/jungfrau" / + "4_modules_udp_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == + std::vector{1, 3, 4, 6}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); + auto frames = f.read_rois(); + REQUIRE(frames.size() == 4); + REQUIRE(frames[0].cols() == 1024); + REQUIRE(frames[0].rows() == 256); + REQUIRE(frames[1].cols() == 1024); + REQUIRE(frames[1].rows() == 256); + REQUIRE(frames[2].cols() == 1024); + REQUIRE(frames[2].rows() == 256); + REQUIRE(frames[3].cols() == 1024); + REQUIRE(frames[3].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 4); + REQUIRE(rois[0] == ROI{0, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 512, 768}); + REQUIRE(rois[2] == ROI{1024, 2048, 256, 512}); + REQUIRE(rois[3] == ROI{1024, 2048, 768, 1024}); + } +} + +TEST_CASE("Read Moench frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled top port") { + auto fpath = test_data_path() / "raw/moench" / + "2_interfaces_top_port_disabled_master_0.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 400); + REQUIRE(frame.rows() == 200); + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 400, 0, 200}); + } + + SECTION("disabled bottom port") { + auto fpath = test_data_path() / "raw/moench" / + "2_interfaces_bottom_port_disabled_master_6.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 400); + REQUIRE(frame.rows() == 200); + auto rois = f.master().rois(); + + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 400, 200, 400}); + } +} + +TEST_CASE("Read Eiger frame with disabled UDP ports", + "[.with-data][RawFile][disabled_udp_ports]") { + SECTION("disabled left port (bottom and top half module)") { + auto fpath = test_data_path() / "raw/eiger" / + "left_udp_port_disabled_master_2.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0, 2}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + auto frame = f.read_frame(); + + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 512); + + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{512, 1024, 0, 512}); + } + SECTION("disabled right port") { + auto fpath = test_data_path() / "raw/eiger" / + "right_udp_port_disabled_master_3.json"; + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1, 3}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 512, 0, 512}); + } + SECTION("2 full modules stacked vertically - right ports disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "2_modules_eiger_disabled_udp_port_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == + std::vector{1, 3, 5, 7}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() " + "instead")); + auto frames = f.read_rois(); + REQUIRE(frames.size() == 2); + REQUIRE(frames[0].cols() == 512); + REQUIRE(frames[0].rows() == 512); + REQUIRE(frames[1].cols() == 512); + REQUIRE(frames[1].rows() == 512); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + REQUIRE(rois[0] == ROI{0, 512, 0, 512}); + REQUIRE(rois[1] == ROI{1024, 1536, 0, 512}); + } + SECTION("quad module - bottom port disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "quad_eiger_disabled_bottom_port_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{1}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}); + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 512); + REQUIRE(frame.rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 512, 256, 512}); + } + SECTION("only bottom left port disabled") { + auto fpath = test_data_path() / "raw/eiger" / + "one_udp_port_disabled_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + REQUIRE(f.master().disabled_udp_ports() == std::vector{0}); + REQUIRE(f.master().udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + REQUIRE_THROWS_WITH( + f.read_frame(), + Catch::Matchers::ContainsSubstring( + "Multiple ROIs present in file. Use read_ROIs() instead")); + + auto frame = f.read_rois(); + REQUIRE(frame.size() == 2); + REQUIRE(frame[0].cols() == 512); + REQUIRE(frame[0].rows() == 256); + REQUIRE(frame[1].cols() == 1024); + REQUIRE(frame[1].rows() == 256); + auto rois = f.master().rois(); + REQUIRE(rois.size() == 2); + + REQUIRE(rois[0] == ROI{512, 1024, 0, 256}); + REQUIRE(rois[1] == ROI{0, 1024, 256, 512}); + } + SECTION("No udp ports disabled") { + auto fpath = test_data_path() / + "raw/eiger_virtual_500k_disabled_ports" / + "all_active_master_0.json"; + + REQUIRE(std::filesystem::exists(fpath)); + RawFile f(fpath); + + auto disabled_udp_ports = f.master().disabled_udp_ports(); + REQUIRE(disabled_udp_ports.empty()); + + auto udp_port_types = f.master().udp_port_types(); + REQUIRE(udp_port_types.has_value()); + REQUIRE(udp_port_types.value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + REQUIRE(f.total_frames() == 5); + + auto frame = f.read_frame(); + REQUIRE(frame.cols() == 1024); + REQUIRE(frame.rows() == 512); + + auto rois = f.master().rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 512}); + } +} diff --git a/src/RawMasterFile.cpp b/src/RawMasterFile.cpp index 9eff0d26..e0880ba5 100644 --- a/src/RawMasterFile.cpp +++ b/src/RawMasterFile.cpp @@ -4,7 +4,7 @@ #include "aare/logger.hpp" #include -#include "to_string.hpp" +#include "aare/to_string.hpp" namespace aare { @@ -115,6 +115,32 @@ RawMasterFile::RawMasterFile(const std::filesystem::path &fpath) } else { throw std::runtime_error(LOCATION + "Unsupported file type"); } + + m_geometry = DetectorGeometry(m_detector_layout, m_pixels_x, m_pixels_y, + m_udp_interfaces_per_module, m_quad); + + if (m_quad == 1 && m_udp_port_types.has_value()) { + m_udp_port_types.value() = {UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}; + } + + if (!m_disabled_udp_ports.empty()) { + // ROI takes precedence over disabled UDP ports, if both are defined in + // the master file + if (!complete_ROI(m_rois, m_geometry)) { + LOG(logWARNING) + << "ROI and disabled UDP ports defined in master file. ROI " + "will be used and disabled UDP ports will be ignored."; + } else { + update_rois_from_disabled_udp_ports(); + } + } + + m_ROI_geometries.reserve(m_rois.size()); + + for (const auto &roi : m_rois) { + m_ROI_geometries.push_back(ROIGeometry(roi, m_geometry)); + } } RawMasterFile::RawMasterFile(std::istream &is, const std::string &fname) @@ -164,10 +190,16 @@ std::optional RawMasterFile::counter_mask() const { return m_counter_mask; } -xy RawMasterFile::geometry() const { return m_geometry; } +xy RawMasterFile::detector_layout() const { return m_detector_layout; } + +const DetectorGeometry &RawMasterFile::geometry() const { return m_geometry; } + +const std::vector &RawMasterFile::roi_geometries() const { + return m_ROI_geometries; +} size_t RawMasterFile::n_modules() const { - return m_geometry.row * m_geometry.col; + return m_detector_layout.row * m_detector_layout.col; } xy RawMasterFile::udp_interfaces_per_module() const { @@ -193,26 +225,30 @@ ScanParameters RawMasterFile::scan_parameters() const { return m_scan_parameters; } -std::optional RawMasterFile::roi() const { - if (!m_rois) { - return std::nullopt; - } +std::optional> +RawMasterFile::udp_port_types() const { + return m_udp_port_types; +} - if (m_rois->empty()) { +std::vector RawMasterFile::disabled_udp_ports() const { + return m_disabled_udp_ports; +} + +ROI RawMasterFile::roi() const { + + if (m_rois.empty()) { throw std::runtime_error(LOCATION + "Zero ROIs in metadata."); } - if (m_rois.value().size() > 1) { + if (m_rois.size() > 1) { throw std::runtime_error(LOCATION + "Multiple ROIs present, use rois() method."); } else { - return m_rois.has_value() - ? std::optional(m_rois.value().at(0)) - : std::nullopt; // TODO: maybe throw if no roi exists + return m_rois.at(0); } } -std::optional> RawMasterFile::rois() const { return m_rois; } +std::vector RawMasterFile::rois() const { return m_rois; } ReadoutMode RawMasterFile::get_reading_mode() const { @@ -248,10 +284,11 @@ void RawMasterFile::parse_json(std::istream &is) { m_type = string_to(j["Detector Type"].get()); m_timing_mode = string_to(j["Timing Mode"].get()); - m_geometry = {j["Geometry"]["y"], - j["Geometry"]["x"]}; // TODO: isnt it only available for - // version > 7.1? - // - try block default should be 1x1 + m_detector_layout = { + j["Geometry"]["y"], + j["Geometry"]["x"]}; // TODO: isnt it only available for + // version > 7.1? + // - try block default should be 1x1 m_image_size_in_bytes = v < 8.0 ? j["Image Size in bytes"] : j["Image Size"]; @@ -301,6 +338,9 @@ void RawMasterFile::parse_json(std::istream &is) { if (j.contains("Number of rows") && j["Number of rows"].is_number()) { m_number_of_rows = j["Number of rows"]; } + if (j.contains("Number of Rows") && j["Number of Rows"].is_number()) { + m_number_of_rows = j["Number of Rows"]; // changed at some point + } // ---------------------------------------------------------------- // Special treatment of analog flag because of Moench03. @@ -377,7 +417,26 @@ void RawMasterFile::parse_json(std::istream &is) { } catch (const json::out_of_range &e) { // not a scan } - + try { + auto json_list_obj = j.at("UDP Ports Type"); + m_udp_port_types.emplace(); + for (auto &elem : json_list_obj) { + m_udp_port_types.value().push_back( + string_to(elem)); + } + } catch (const json::out_of_range &e) { + // leave the optional empty + } + try { + auto json_list_obj = j.at("UDP Ports Disabled"); + m_disabled_udp_ports.clear(); + for (auto &elem : json_list_obj) { + m_disabled_udp_ports.push_back(static_cast(elem)); + } + } catch (const json::out_of_range &e) { + m_disabled_udp_ports + .clear(); // empty list if not present in master file + } try { m_udp_interfaces_per_module = {j.at("Number of UDP Interfaces"), 1}; } catch (const json::out_of_range &e) { @@ -397,14 +456,21 @@ void RawMasterFile::parse_json(std::istream &is) { obj.at("ymin") = 0; obj.at("ymax") = 0; } - m_rois.emplace(); - m_rois.value().push_back(ROI{ - obj.at("xmin"), static_cast(obj.at("xmax")) + 1, - obj.at("ymin"), static_cast(obj.at("ymax")) + 1}); + m_rois.push_back({static_cast(obj.at("xmin")), + static_cast(obj.at("xmax")) + 1, + static_cast(obj.at("ymin")), + static_cast(obj.at("ymax")) + 1}); + } else { + // fill ROI with full detector size if not present in master + // file + m_rois.push_back( + {0, + m_detector_layout.col * static_cast(m_pixels_x), + 0, + m_detector_layout.row * static_cast(m_pixels_y)}); } } else { auto obj = j.at("Receiver Rois"); - m_rois.emplace(); for (auto &elem : obj) { // handle Mythen if (elem.at("ymin") == -1 && elem.at("ymax") == -1) { @@ -412,15 +478,18 @@ void RawMasterFile::parse_json(std::istream &is) { elem.at("ymax") = 0; } - m_rois.value().push_back(ROI{ - elem.at("xmin"), static_cast(elem.at("xmax")) + 1, - elem.at("ymin"), - static_cast(elem.at("ymax")) + 1}); + m_rois.push_back({static_cast(elem.at("xmin")), + static_cast(elem.at("xmax")) + 1, + static_cast(elem.at("ymin")), + static_cast(elem.at("ymax")) + 1}); } } } catch (const json::out_of_range &e) { - // leave the optional empty + // fill ROI with full detector size if not present in master file + m_rois.push_back( + {0, m_detector_layout.col * static_cast(m_pixels_x), 0, + m_detector_layout.row * static_cast(m_pixels_y)}); } if (j.contains("Counter Mask")) { @@ -477,9 +546,8 @@ void RawMasterFile::parse_raw(std::istream &is) { m_image_size_in_bytes = std::stoi(value); } else if (key == "Frame Padding") { m_frame_padding = std::stoi(value); - // } else if (key == "Frame Discard Policy"){ - // m_frame_discard_policy = - // StringTo(value); + } else if (key == "Frame Discard Policy") { + m_frame_discard_policy = string_to(value); // } else if (key == "Number of rows"){ // m_number_of_rows = std::stoi(value); } else if (key == "Analog Flag") { @@ -528,7 +596,7 @@ void RawMasterFile::parse_raw(std::istream &is) { m_max_frames_per_file = std::stoi(value); } else if (key == "Geometry") { pos = value.find(','); - m_geometry = { + m_detector_layout = { static_cast(std::stoi(value.substr(1, pos))), static_cast(std::stoi(value.substr(pos + 1)))}; } else if (key == "Number of UDP Interfaces") { @@ -548,13 +616,17 @@ void RawMasterFile::parse_raw(std::istream &is) { m_type = DetectorType::Moench03_old; } - if (m_geometry.col == 0 && m_geometry.row == 0) { + if (m_detector_layout.col == 0 && m_detector_layout.row == 0) { retrieve_geometry(); LOG(TLogLevel::logWARNING) << "No geometry found in master file. Retrieved geometry of " - << m_geometry.row << " x " << m_geometry.col << "\n "; + << m_detector_layout.row << " x " << m_detector_layout.col << "\n "; } + m_rois.push_back( + {0, m_detector_layout.col * static_cast(m_pixels_x), 0, + m_detector_layout.row * static_cast(m_pixels_y)}); + // TODO! Read files and find actual frames if (m_frames_in_file == 0) m_frames_in_file = m_total_frames_expected; @@ -578,7 +650,153 @@ void RawMasterFile::retrieve_geometry() { ++rows; ++cols; - m_geometry = {rows, cols}; + m_detector_layout = {rows, cols}; +} + +/** + * @brief Update ROIs from disabled UDP ports + */ +void RawMasterFile::update_rois_from_disabled_udp_ports() { + + const size_t num_udp_port_types = m_udp_port_types.value().size(); + + size_t first_port = m_disabled_udp_ports[0] % num_udp_port_types; + + bool all_ports_equal = + std::all_of(m_disabled_udp_ports.begin(), m_disabled_udp_ports.end(), + [&num_udp_port_types, first_port](size_t &port) { + return port % num_udp_port_types == first_port; + }); + + m_rois.clear(); // remove global roi + + const ssize_t udp_ports_per_module = + m_geometry.udp_interfaces_per_module().col * + m_geometry.udp_interfaces_per_module().row; + + bool port_disabled_for_all_modules = + m_disabled_udp_ports.size() == + m_geometry.modules_x() * m_geometry.modules_y() / udp_ports_per_module; + + if (all_ports_equal && port_disabled_for_all_modules) { + + if (m_udp_port_types.value()[first_port] == UDPPortPosition::LEFT) { + const size_t num_rois = + m_geometry.modules_x() / udp_ports_per_module; + m_rois.resize(num_rois); + + const ssize_t pixels_per_module_x = + m_geometry.pixels_x() / m_geometry.modules_x(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_x, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{ + idx * udp_ports_per_module * pixels_per_module_x + + pixels_per_module_x, + idx * udp_ports_per_module * pixels_per_module_x + + 2 * pixels_per_module_x, + 0, static_cast(m_geometry.pixels_y())}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::RIGHT) { + const size_t num_rois = + m_geometry.modules_x() / udp_ports_per_module; + m_rois.resize(num_rois); + + const ssize_t pixels_per_module_x = + m_geometry.pixels_x() / m_geometry.modules_x(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_x, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{idx * udp_ports_per_module * pixels_per_module_x, + idx * udp_ports_per_module * + pixels_per_module_x + + pixels_per_module_x, + 0, static_cast(m_geometry.pixels_y())}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::TOP) { + size_t num_rois = m_geometry.modules_y() / udp_ports_per_module; + m_rois.resize(num_rois); + // assumes euclidean coordinate system with origin at bottom + // left corner of the detector + const ssize_t pixels_per_module_y = + m_geometry.pixels_y() / m_geometry.modules_y(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_y, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{0, static_cast(m_geometry.pixels_x()), + idx * udp_ports_per_module * pixels_per_module_y, + idx * udp_ports_per_module * + pixels_per_module_y + + pixels_per_module_y}; + }); + } + if (m_udp_port_types.value()[first_port] == UDPPortPosition::BOTTOM) { + size_t num_rois = m_geometry.modules_y() / udp_ports_per_module; + m_rois.resize(num_rois); + const ssize_t pixels_per_module_y = + m_geometry.pixels_y() / m_geometry.modules_y(); + std::generate( + m_rois.begin(), m_rois.end(), + [n = 0, this, pixels_per_module_y, + udp_ports_per_module]() mutable { + ssize_t idx = n++; + return ROI{ + 0, static_cast(m_geometry.pixels_x()), + idx * udp_ports_per_module * pixels_per_module_y + + pixels_per_module_y, + idx * udp_ports_per_module * pixels_per_module_y + + 2 * pixels_per_module_y}; + }); + } + } else { + // iterate over all ports and create ROIs for each disabled port + LOG(logDEBUG) << "Creating ROIs from disabled UDP ports"; + // get the enabled ones: + std::vector enabled_ports(m_geometry.n_modules()); + std::iota(enabled_ports.begin(), enabled_ports.end(), 0); + std::for_each(m_disabled_udp_ports.begin(), m_disabled_udp_ports.end(), + [&enabled_ports](size_t &port) { + enabled_ports.erase(std::remove(enabled_ports.begin(), + enabled_ports.end(), + port), + enabled_ports.end()); + }); + + m_rois.reserve(enabled_ports.size()); + for (const auto enabled_port : enabled_ports) { + + auto module_geometry = + m_geometry.get_module_geometries(enabled_port); + m_rois.push_back( + ROI{module_geometry.origin_x, + module_geometry.origin_x + module_geometry.width, + module_geometry.origin_y, + module_geometry.origin_y + module_geometry.height}); + } + + if (m_udp_port_types == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}) { + m_rois = merge_consecutive_rois(m_rois); + } else if (m_udp_port_types == + std::vector{UDPPortPosition::BOTTOM, + UDPPortPosition::TOP} || + m_udp_port_types == + std::vector{UDPPortPosition::TOP, + UDPPortPosition::BOTTOM}) { + m_rois = merge_consecutive_rois(m_rois); + } else { + throw std::runtime_error(LOCATION + "Unsupported UDP port types"); + } + } } } // namespace aare diff --git a/src/RawMasterFile.test.cpp b/src/RawMasterFile.test.cpp index 59efed71..3838bfc2 100644 --- a/src/RawMasterFile.test.cpp +++ b/src/RawMasterFile.test.cpp @@ -3,11 +3,38 @@ #include "test_config.hpp" #include +#include #include #include using namespace aare; +TEST_CASE("Parse legacy master frame discard policies", + "[rawmasterfile][frame-policy]") { + const auto [policy, expected] = + GENERATE(table({ + {"nodiscard", FrameDiscardPolicy::NoDiscard}, + {"discard", FrameDiscardPolicy::Discard}, + {"discardpartial", FrameDiscardPolicy::DiscardPartial}, + {"", FrameDiscardPolicy::NoDiscard}, + })); + CAPTURE(policy); + std::ostringstream content; + content << "Version : 6.4\n" + "Detector Type : Jungfrau\n" + "Timing Mode : auto\n" + "Geometry : [1, 1]\n" + "Pixels : [3, 2]\n" + "Frame Padding : 0\n"; + if (!policy.empty()) { + content << "Frame Discard Policy : " << policy << '\n'; + } + std::istringstream input(content.str()); + RawMasterFile master(input, "run_master_0.raw"); + REQUIRE(master.frame_padding() == 0); + REQUIRE(master.frame_discard_policy() == expected); +} + TEST_CASE("Parse a master file fname") { RawFileNameComponents m("test_master_1.json"); REQUIRE(m.base_name() == "test"); @@ -68,7 +95,7 @@ TEST_CASE("A disabled scan") { REQUIRE(s.step() == 0); } -TEST_CASE("Parse a master file in .json format", "[.integration]") { +TEST_CASE("Parse a master file in .json format", "[.with-data]") { auto fpath = test_data_path() / "raw" / "jungfrau" / "jungfrau_single_master_0.json"; REQUIRE(std::filesystem::exists(fpath)); @@ -80,14 +107,14 @@ TEST_CASE("Parse a master file in .json format", "[.integration]") { // "Detector Type": "Jungfrau", REQUIRE(f.detector_type() == DetectorType::Jungfrau); // "Timing Mode": "auto", - REQUIRE(f.timing_mode() == TimingMode::Auto); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); // "Geometry": { // "x": 1, // "y": 1 // }, - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); // "Image Size in bytes": 1048576, REQUIRE(f.image_size_in_bytes() == 1048576); @@ -151,7 +178,7 @@ TEST_CASE("Parse a master file in .json format", "[.integration]") { } TEST_CASE("Parse a master file in old .raw format", - "[.integration][.with-data][.rawmasterfile]") { + "[.with-data][rawmasterfile]") { auto fpath = test_data_path() / "raw/jungfrau_2modules_version6.1.2/run_master_0.raw"; REQUIRE(std::filesystem::exists(fpath)); @@ -159,11 +186,11 @@ TEST_CASE("Parse a master file in old .raw format", CHECK(f.udp_interfaces_per_module() == xy{1, 1}); CHECK(f.n_modules() == 2); - CHECK(f.geometry().row == 2); - CHECK(f.geometry().col == 1); + CHECK(f.detector_layout().row == 2); + CHECK(f.detector_layout().col == 1); } -TEST_CASE("Parse a master file in .raw format", "[.integration]") { +TEST_CASE("Parse a master file in .raw format", "[.with-data]") { auto fpath = test_data_path() / @@ -179,10 +206,10 @@ TEST_CASE("Parse a master file in .raw format", "[.integration]") { // Detector Type : ChipTestBoard REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); // Timing Mode : auto - REQUIRE(f.timing_mode() == TimingMode::Auto); - // Geometry : [1, 1] - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + // Detector Layout : [1, 1] + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); // Image Size : 360000 bytes REQUIRE(f.image_size_in_bytes() == 360000); // Pixels : [96, 1] @@ -232,8 +259,7 @@ TEST_CASE("Parse a master file in .raw format", "[.integration]") { // Packets Caught Mask : 64 bytes } -TEST_CASE("Parse a master file in new .json format", - "[.integration][.with-data]") { +TEST_CASE("Parse a master file in new .json format", "[.with-data]") { auto file_path = test_data_path() / "raw" / "newmythen03" / "run_87_master_0.json"; @@ -246,10 +272,10 @@ TEST_CASE("Parse a master file in new .json format", REQUIRE(f.detector_type() == DetectorType::Mythen3); // Timing Mode : auto - REQUIRE(f.timing_mode() == TimingMode::Auto); - // Geometry : [2, 1] - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + // Detector Layout : [2, 1] + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 1); // Image Size : 5120 bytes REQUIRE(f.image_size_in_bytes() == 5120); @@ -260,14 +286,14 @@ TEST_CASE("Parse a master file in new .json format", REQUIRE(f.scan_parameters().step() == 0); REQUIRE(f.scan_parameters().settleTime() == 0); - auto roi = f.roi().value(); + auto roi = f.roi(); REQUIRE(roi.xmin == 0); REQUIRE(roi.xmax == 2560); REQUIRE(roi.ymin == 0); REQUIRE(roi.ymax == 1); } -TEST_CASE("Read eiger master file", "[.integration]") { +TEST_CASE("Read eiger master file", "[.with-data]") { auto fpath = test_data_path() / "raw/eiger/eiger_500k_32bit_master_0.json"; REQUIRE(std::filesystem::exists(fpath)); RawMasterFile f(fpath); @@ -279,7 +305,7 @@ TEST_CASE("Read eiger master file", "[.integration]") { // "Detector Type": "Eiger", REQUIRE(f.detector_type() == DetectorType::Eiger); // "Timing Mode": "auto", - REQUIRE(f.timing_mode() == TimingMode::Auto); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); // "Geometry": { // "x": 2, // "y": 2 @@ -399,9 +425,9 @@ TEST_CASE("Parse EIGER 7.2 master from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Eiger); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 2); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 2); REQUIRE(f.image_size_in_bytes() == 524288); REQUIRE(f.pixels_x() == 512); @@ -476,9 +502,9 @@ TEST_CASE("Parse JUNGFRAU 7.2 master from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Jungfrau); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 2); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 2); REQUIRE(f.n_modules() == 2); REQUIRE(f.image_size_in_bytes() == 524288); REQUIRE(f.pixels_x() == 1024); @@ -559,8 +585,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 192000); REQUIRE(f.pixels_x() == 32); REQUIRE(f.pixels_y() == 1); @@ -637,8 +663,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 16000); REQUIRE(f.pixels_x() == 64); REQUIRE(f.pixels_y() == 1); @@ -709,8 +735,8 @@ TEST_CASE("Parse Moench 7.2 master (SW 7.0.3) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Moench03_old); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -780,8 +806,8 @@ TEST_CASE("Parse Moench 7.2 master (SW 8.0.0) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::Moench03); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -862,8 +888,8 @@ TEST_CASE("Parse CTB 7.2 master (SW 8.0.0) from string stream") { REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 192000); REQUIRE(f.pixels_x() == 32); REQUIRE(f.pixels_y() == 1); @@ -943,8 +969,8 @@ TEST_CASE( REQUIRE(f.version() == "7.2"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry() == xy{1, 1}); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{1, 1}); REQUIRE(f.image_size_in_bytes() == 16000); REQUIRE(f.pixels_x() == 64); REQUIRE(f.pixels_y() == 1); @@ -1010,9 +1036,9 @@ TEST_CASE("Parse a CTB file from stream") { REQUIRE(f.version() == "8.0"); REQUIRE(f.detector_type() == DetectorType::ChipTestBoard); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 18432); REQUIRE(f.pixels_x() == 2); REQUIRE(f.pixels_y() == 1); @@ -1100,9 +1126,9 @@ TEST_CASE("Parse v8.0 MYTHEN3 from stream") { REQUIRE(f.version() == "8.0"); REQUIRE(f.detector_type() == DetectorType::Mythen3); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 2); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 2); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 5120); REQUIRE(f.pixels_x() == 1280); REQUIRE(f.pixels_y() == 1); @@ -1185,9 +1211,9 @@ TEST_CASE("Parse a v7.1 Mythen3 from stream") { REQUIRE(f.version() == "7.1"); REQUIRE(f.detector_type() == DetectorType::Mythen3); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 15360); REQUIRE(f.pixels_x() == 3840); REQUIRE(f.pixels_y() == 1); @@ -1267,9 +1293,9 @@ TEST_CASE("Parse old Moench03 from stream") { REQUIRE(f.version() == "7.1"); REQUIRE(f.detector_type() == DetectorType::Moench03_old); - REQUIRE(f.timing_mode() == TimingMode::Auto); - REQUIRE(f.geometry().col == 1); - REQUIRE(f.geometry().row == 1); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout().col == 1); + REQUIRE(f.detector_layout().row == 1); REQUIRE(f.image_size_in_bytes() == 320000); REQUIRE(f.pixels_x() == 400); REQUIRE(f.pixels_y() == 400); @@ -1286,3 +1312,111 @@ TEST_CASE("Parse old Moench03 from stream") { REQUIRE(f.period() == std::chrono::microseconds(600)); REQUIRE(f.analog_samples() == 5000); } + +TEST_CASE("Parse Eiger json v8.1 all ports active") { + std::string master_content = R"( + { + "Version": 8.1, + "Timestamp": "Mon Aug 10 17:44:46 2026", + "Detector Type": "Eiger", + "Timing Mode": "auto", + "Geometry": { + "x": 2, + "y": 2 + }, + "Image Size": 262144, + "Pixels": { + "x": 512, + "y": 256 + }, + "Max Frames Per File": 10000, + "Frame Discard Policy": "nodiscard", + "Frame Padding": 1, + "Scan Parameters": { + "enable": 0, + "dacInd": 0, + "start offset": 0, + "stop offset": 0, + "step size": 0, + "dac settle time ns": 0 + }, + "Total Frames": 5, + "Receiver Rois": [ + { + "xmin": 0, + "xmax": 1023, + "ymin": 0, + "ymax": 511 + } + ], + "Dynamic Range": 16, + "Ten Giga": 0, + "Exposure Time": "1s", + "Acquisition Period": "1s", + "Threshold Energy": -1, + "Sub Exposure Time": "2.62144ms", + "Sub Acquisition Period": "2.62144ms", + "Quad": 0, + "UDP Ports Type": [ + "left", + "right" + ], + "UDP Ports Disabled": [], + "Number of Rows": 256, + "Rate Corrections": [ + 0, + 0 + ], + "Readout Speed": "full_speed", + "Frames in File": 5, + "Additional JSON Header": {} +} + +)"; + + std::istringstream iss(master_content); + RawMasterFile f(iss, "test_master_0.json"); + + REQUIRE(f.version() == "8.1"); + REQUIRE(f.detector_type() == DetectorType::Eiger); + REQUIRE(f.timing_mode() == TimingMode::AUTO_TIMING); + REQUIRE(f.detector_layout() == xy{2, 2}); + REQUIRE(f.n_modules() == 4); + REQUIRE(f.image_size_in_bytes() == 262144); + REQUIRE(f.pixels_x() == 512); + REQUIRE(f.pixels_y() == 256); + REQUIRE(f.max_frames_per_file() == 10000); + REQUIRE(f.frame_discard_policy() == FrameDiscardPolicy::NoDiscard); + REQUIRE(f.frame_padding() == 1); + REQUIRE(f.total_frames_expected() == 5); + REQUIRE(f.frames_in_file() == 5); + REQUIRE(f.bitdepth() == 16); + REQUIRE(f.exptime() == std::chrono::seconds(1)); + REQUIRE(f.period() == std::chrono::seconds(1)); + REQUIRE(f.quad() == 0); + // rows return a std::optional, so we need to check if it has a value before + // using it + auto rows = f.number_of_rows(); + REQUIRE(rows.has_value()); + REQUIRE(rows.value() == 256); + REQUIRE(f.udp_interfaces_per_module() == xy{1, 2}); + + auto scan_parameters = f.scan_parameters(); + REQUIRE_FALSE(scan_parameters.enabled()); + REQUIRE(scan_parameters.dac() == DACIndex::DAC_0); + REQUIRE(scan_parameters.start() == 0); + REQUIRE(scan_parameters.stop() == 0); + REQUIRE(scan_parameters.step() == 0); + REQUIRE(scan_parameters.settleTime() == 0); + + REQUIRE(f.udp_port_types().has_value()); + REQUIRE(f.udp_port_types().value() == + std::vector{UDPPortPosition::LEFT, + UDPPortPosition::RIGHT}); + + REQUIRE(f.disabled_udp_ports().empty()); + + auto rois = f.rois(); + REQUIRE(rois.size() == 1); + REQUIRE(rois[0] == ROI{0, 1024, 0, 512}); +} diff --git a/src/RawSubFile.cpp b/src/RawSubFile.cpp index ed60a4a5..7a42bbba 100644 --- a/src/RawSubFile.cpp +++ b/src/RawSubFile.cpp @@ -12,6 +12,17 @@ namespace aare { +std::runtime_error RawSubFile::frame_error(size_t frame_index, + const std::string &message) const { + return std::runtime_error( + fmt::format("Error reading frame index {} from file '{}': {}", + frame_index, current_path().string(), message)); +} + +std::filesystem::path RawSubFile::current_path() const { + return fpath(m_current_file_index + m_offset); +} + RawSubFile::RawSubFile(const std::filesystem::path &fname, DetectorType detector, size_t rows, size_t cols, size_t bitdepth, uint32_t pos_row, uint32_t pos_col) @@ -34,8 +45,9 @@ RawSubFile::RawSubFile(const std::filesystem::path &fname, void RawSubFile::seek(size_t frame_index) { LOG(logDEBUG) << "RawSubFile::seek(" << frame_index << ")"; if (frame_index >= m_total_frames) { - throw std::runtime_error(LOCATION + " Frame index out of range: " + - std::to_string(frame_index)); + throw frame_error( + frame_index, + fmt::format("out of range (available frames: {})", m_total_frames)); } m_current_frame_index = frame_index; auto file_index = first_larger(m_last_frame_in_file, frame_index); @@ -49,6 +61,9 @@ void RawSubFile::seek(size_t frame_index) { auto byte_offset = frame_offset * (m_bytes_per_frame + sizeof(DetectorHeader)); m_file.seekg(byte_offset); + if (m_file.fail()) { + throw frame_error(frame_index, ifstream_error_msg(m_file)); + } } size_t RawSubFile::tell() { @@ -66,7 +81,7 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } if (m_file.fail()) { - throw std::runtime_error(LOCATION + ifstream_error_msg(m_file)); + throw frame_error(m_current_frame_index, ifstream_error_msg(m_file)); } // TODO! expand support for different bitdepths @@ -81,8 +96,8 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } else if (m_bitdepth == 32) { read_with_map(image_buf); } else { - throw std::runtime_error( - "Unsupported bitdepth for read with pixel map"); + throw frame_error(m_current_frame_index, + "Unsupported bitdepth for read with pixel map"); } } else { @@ -91,7 +106,7 @@ void RawSubFile::read_into(std::byte *image_buf, DetectorHeader *header) { } if (m_file.fail()) { - throw std::runtime_error(LOCATION + ifstream_error_msg(m_file)); + throw frame_error(m_current_frame_index, ifstream_error_msg(m_file)); } ++m_current_frame_index; @@ -135,6 +150,9 @@ size_t RawSubFile::frame_number(size_t frame_index) { seek(frame_index); DetectorHeader h{}; m_file.read(reinterpret_cast(&h), sizeof(DetectorHeader)); + if (m_file.fail()) { + throw frame_error(frame_index, ifstream_error_msg(m_file)); + } return h.frameNumber; } @@ -178,8 +196,8 @@ void RawSubFile::open_file(size_t file_index) { m_file.open(fname, std::ios::binary); if (!m_file.is_open()) { throw std::runtime_error( - LOCATION + - fmt::format("Could not open file {}", fpath(file_index).string())); + fmt::format("Could not open file '{}' for frame index {}", + fname.string(), m_current_frame_index)); } m_current_file_index = file_index; } @@ -208,4 +226,4 @@ void RawSubFile::scan_files() { } } -} // namespace aare \ No newline at end of file +} // namespace aare diff --git a/src/RawSubFile.test.cpp b/src/RawSubFile.test.cpp index 9550ed17..e09dff7e 100644 --- a/src/RawSubFile.test.cpp +++ b/src/RawSubFile.test.cpp @@ -2,11 +2,70 @@ #include "aare/RawSubFile.hpp" #include "aare/File.hpp" #include "aare/NDArray.hpp" +#include "raw_file_helpers.hpp" #include "test_config.hpp" #include +#include +#include using namespace aare; +TEST_CASE("RawSubFile errors identify the frame and actual data path", + "[RawSubFile][read-errors]") { + TemporaryRawFiles files; + const size_t offset = GENERATE(0, 5); + if (offset != 0) { + for (size_t index = 0; index < 2; ++index) { + std::filesystem::rename(files.data_path(0, index), + files.data_path(0, offset + index)); + } + } + RawSubFile file(files.data_path(0, offset), DetectorType::Jungfrau, 2, 3, + 16); + CHECK(file.current_path() == files.data_path(0, offset)); + std::vector buffer(file.bytes_per_frame() * 3); + DetectorHeader header{}; + const auto second_path = files.data_path(0, offset + 1); + const auto index_matcher = + Catch::Matchers::ContainsSubstring("frame index 1"); + const auto path_matcher = + Catch::Matchers::ContainsSubstring(second_path.string()); + const auto context = index_matcher && path_matcher; + + SECTION("truncated header") { + std::filesystem::resize_file(second_path, 0); + file.seek(1); + REQUIRE_THROWS_WITH(file.read_into(buffer.data(), &header), context); + } + SECTION("truncated pixels") { + std::filesystem::resize_file(second_path, sizeof(DetectorHeader)); + file.seek(1); + CHECK(file.current_path() == second_path); + REQUIRE_THROWS_WITH(file.read_into(buffer.data()), context); + } + SECTION("frame number from truncated header") { + std::filesystem::resize_file(second_path, 0); + REQUIRE_THROWS_WITH(file.frame_number(1), context); + } + SECTION("missing next file") { + std::filesystem::remove(second_path); + REQUIRE_THROWS_WITH(file.seek(1), context); + } + SECTION("batch beyond EOF") { + REQUIRE_THROWS_WITH( + file.read_into(buffer.data(), size_t{3}), + Catch::Matchers::ContainsSubstring("frame index 2") && + Catch::Matchers::ContainsSubstring(second_path.string())); + } + SECTION("seek beyond EOF") { + REQUIRE_THROWS_WITH( + file.seek(17), + Catch::Matchers::ContainsSubstring("frame index 17") && + Catch::Matchers::ContainsSubstring( + files.data_path(0, offset).string())); + } +} + TEST_CASE("Read frames directly from a RawSubFile", "[.with-data]") { auto fpath_raw = test_data_path() / "raw/jungfrau" / "jungfrau_single_d0_f0_0.raw"; @@ -77,4 +136,4 @@ TEST_CASE("Read frames directly from a RawSubFile starting at the second file", auto npy_frame = npy.read_frame(); CHECK((image.view() == npy_frame.view())); } -} \ No newline at end of file +} diff --git a/src/algorithm.test.cpp b/src/algorithm.test.cpp index daa0616d..e498c3b2 100644 --- a/src/algorithm.test.cpp +++ b/src/algorithm.test.cpp @@ -231,4 +231,4 @@ TEST_CASE("Bilinear interpolation", "[algorithm]") { 0.75); REQUIRE(interpolated_value == 5.25); } -} +} \ No newline at end of file diff --git a/src/decode.cpp b/src/decode.cpp index afe261f9..f522f5c1 100644 --- a/src/decode.cpp +++ b/src/decode.cpp @@ -1,6 +1,7 @@ // SPDX-License-Identifier: MPL-2.0 #include "aare/decode.hpp" #include +#include #include namespace aare { @@ -102,6 +103,57 @@ void adc_sar_04_decode64to16(NDView input, } } +namespace { + +constexpr ssize_t bytes_per_word = sizeof(uint64_t); + +// Shared loop for the byte-buffer overloads. Words are copied out with +// memcpy so the input does not need to be aligned to 8 bytes. +template +void decode64to16_from_bytes(NDView input, + NDView output) { + if (input.shape(1) % bytes_per_word != 0) { + throw std::invalid_argument( + fmt::format("{} input row length of {} bytes is not a multiple " + "of {}", + LOCATION, input.shape(1), bytes_per_word)); + } + if (input.shape(0) != output.shape(0) || + input.shape(1) / bytes_per_word != output.shape(1)) { + throw std::invalid_argument(fmt::format( + "{} output shape must be ({}, {}) for an input of " + "shape ({}, {})", + LOCATION, input.shape(0), input.shape(1) / bytes_per_word, + output.shape(0), output.shape(1))); + } + + for (ssize_t i = 0; i < output.shape(0); ++i) { + const uint8_t *row = input.data() + i * input.shape(1); + for (ssize_t j = 0; j < output.shape(1); ++j) { + uint64_t word{}; + std::memcpy(&word, row + j * bytes_per_word, sizeof(word)); + output(i, j) = decode_word(word); + } + } +} + +} // namespace + +void adc_sar_05_06_07_08decode64to16(NDView input, + NDView output) { + decode64to16_from_bytes(input, output); +} + +void adc_sar_05_decode64to16(NDView input, + NDView output) { + decode64to16_from_bytes(input, output); +} + +void adc_sar_04_decode64to16(NDView input, + NDView output) { + decode64to16_from_bytes(input, output); +} + double apply_custom_weights(uint16_t input, const NDView weights) { if (weights.size() > 16) { throw std::invalid_argument( diff --git a/src/decode.test.cpp b/src/decode.test.cpp index bab7c38b..09d6c61c 100644 --- a/src/decode.test.cpp +++ b/src/decode.test.cpp @@ -5,6 +5,8 @@ #include #include using Catch::Matchers::WithinAbs; +#include +#include #include using aare::BitOffset; @@ -49,6 +51,86 @@ TEST_CASE("test_adc_sar_05_decode64to16") { CHECK(output == 3072UL); } +TEST_CASE("Decode 64 bit ADC SAR samples from a byte buffer") { + // Two rows of two words each, viewed as raw bytes + aare::NDArray words({2, 2}); + words(0, 0) = 1UL << 29; + words(0, 1) = (1UL << 25) | (1UL << 22); + words(1, 0) = 0; + words(1, 1) = (1UL << 29) | (1UL << 19) | (1UL << 28); + + aare::NDView bytes( + reinterpret_cast(words.data()), {2, 16}); + aare::NDArray out({2, 2}); + aare::adc_sar_05_decode64to16(bytes, out.view()); + + CHECK(out(0, 0) == 1); + CHECK(out(0, 1) == 3072); + CHECK(out(1, 0) == 0); + CHECK(out(1, 1) == 7); + + // Matches the overload that takes the words directly + aare::NDArray reference({2, 2}); + aare::adc_sar_05_decode64to16(words.view(), reference.view()); + for (ssize_t i = 0; i < 2; ++i) { + for (ssize_t j = 0; j < 2; ++j) { + CHECK(out(i, j) == reference(i, j)); + } + } +} + +TEST_CASE("Each byte buffer decoder maps to its own bit order") { + // First two bits of each decoder's bit list + struct Case { + void (*decode)(aare::NDView, + aare::NDView); + int bit0; + int bit1; + }; + const Case cases[] = { + {aare::adc_sar_05_06_07_08decode64to16, 29, 17}, + {aare::adc_sar_05_decode64to16, 29, 19}, + {aare::adc_sar_04_decode64to16, 15, 17}, + }; + + for (const auto &c : cases) { + uint64_t word = (1UL << c.bit0) | (1UL << c.bit1); + std::vector buffer(8); + std::memcpy(buffer.data(), &word, sizeof(word)); + aare::NDView bytes(buffer.data(), {1, 8}); + aare::NDArray out({1, 1}); + c.decode(bytes, out.view()); + CHECK(out(0, 0) == 3); + } +} + +TEST_CASE("Byte buffer decoding does not require an aligned buffer") { + // Place the word three bytes into the buffer + std::vector buffer(3 + 8, 0); + uint64_t word = (1UL << 29) | (1UL << 19); + std::memcpy(buffer.data() + 3, &word, sizeof(word)); + + aare::NDView bytes(buffer.data() + 3, {1, 8}); + aare::NDArray out({1, 1}); + aare::adc_sar_05_decode64to16(bytes, out.view()); + CHECK(out(0, 0) == 3); +} + +TEST_CASE("Byte buffer decoding rejects invalid shapes") { + std::vector buffer(24, 0); + aare::NDArray out({1, 1}); + + // Row length that is not a multiple of 8 + aare::NDView odd_row(buffer.data(), {2, 12}); + REQUIRE_THROWS_AS(aare::adc_sar_05_decode64to16(odd_row, out.view()), + std::invalid_argument); + + // Output shape that does not match rows x (bytes / 8) + aare::NDView two_words(buffer.data(), {1, 16}); + REQUIRE_THROWS_AS(aare::adc_sar_05_decode64to16(two_words, out.view()), + std::invalid_argument); +} + TEST_CASE("test_apply_custom_weights") { uint16_t input = 1; diff --git a/src/fit/Chi2.hpp b/src/fit/Chi2.hpp new file mode 100644 index 00000000..27fcf1af --- /dev/null +++ b/src/fit/Chi2.hpp @@ -0,0 +1,253 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once +#include "aare/Models.hpp" +#include "aare/NDView.hpp" +#include +#include +#include +#include +#include +#include + +namespace aare { + +namespace func { + +/** + * @brief Chi-squared objective of a 1D model and its analytic derivatives. + * + * Shared by the Minuit2 FCN adapters below. + * + * chi2(par) = sum_i w_i * (y_i - f(x_i; par))^2 + * + * with w_i = 1/sigma_i^2 for weighted fits (points with sigma_i == 0 are + * skipped) and w_i = 1 otherwise. + * + * @tparam Model A model struct that satisfies: + * - static constexpr std::size_t npar; + * - static double eval(double x, const std::vector& par); + * - static void eval_and_grad(double x, const std::vector& par, + * double& f, std::array& g); + * - static bool is_valid(const std::vector& par); + * + * Derivatives, with df_i/dp_k taken from Model::eval_and_grad: + * + * d(chi2)/dp_k = -2 * sum_i w_i * (y_i - f_i) * df_i/dp_k + * d2(chi2)/dp_j dp_k ~= 2 * sum_i w_i * df_i/dp_j * df_i/dp_k + * + * The second line drops the residual-weighted second derivatives of the + * model, which is the Fumili (Gauss-Newton) approximation of the Hessian. + * + * Invalid model parameters do not throw; they return a large penalty (with + * a zero gradient and unit curvature) so the minimizer can remain in control. + * + * @throws std::invalid_argument if par.size() != Model::npar. + */ +template class Chi2Data1D { + public: + static constexpr std::size_t npar = Model::npar; + /** @brief Number of elements of the packed symmetric Hessian. */ + static constexpr std::size_t nhess = npar * (npar + 1) / 2; + static constexpr double invalid_penalty = 1e20; + + /** @brief An empty y_err view selects an unweighted fit. */ + Chi2Data1D(NDView x, NDView y, + NDView y_err = NDView{}) + : x_(x), y_(y), s_(y_err), weighted_(y_err.size() > 0) {} + + double value(const std::vector &par) const { + check_size(par); + if (!Model::is_valid(par)) + return invalid_penalty; + return weighted_ ? value_impl(par) : value_impl(par); + } + + std::vector gradient(const std::vector &par) const { + check_size(par); + std::vector grad(npar, 0.0); + if (!Model::is_valid(par)) + return grad; + if (weighted_) + gradient_impl(par, grad); + else + gradient_impl(par, grad); + return grad; + } + + /** + * @brief Compute chi2, its gradient and the Gauss-Newton Hessian in a + * single pass over the data. + * + * @param grad Receives d(chi2)/dp_k, resized to npar. + * @param hess Receives the packed symmetric Hessian in the layout used by + * ROOT::Minuit2::FumiliFCNBase: element (j, k) with j <= k is + * stored at index j + k * (k + 1) / 2. Resized to nhess. + * @return chi2 at par. + */ + double evaluate_all(const std::vector &par, + std::vector &grad, + std::vector &hess) const { + check_size(par); + grad.assign(npar, 0.0); + hess.assign(nhess, 0.0); + if (!Model::is_valid(par)) { + for (std::size_t k = 0; k < npar; ++k) + hess[k + k * (k + 1) / 2] = 1.0; + return invalid_penalty; + } + return weighted_ ? evaluate_all_impl(par, grad, hess) + : evaluate_all_impl(par, grad, hess); + } + + private: + static void check_size(const std::vector &par) { + if (par.size() != npar) { + throw std::invalid_argument( + "Chi2Data1D: wrong parameter vector size."); + } + } + + template + double value_impl(const std::vector &par) const { + double chi2 = 0.0; + for (ssize_t i = 0; i < x_.size(); ++i) { + if constexpr (Weighted) { + const double si = s_[i]; + if (si == 0.0) + continue; + const double r_i = y_[i] - Model::eval(x_[i], par); + chi2 += (r_i * r_i) / (si * si); + } else { + const double r_i = y_[i] - Model::eval(x_[i], par); + chi2 += r_i * r_i; + } + } + return chi2; + } + + template + void gradient_impl(const std::vector &par, + std::vector &grad) const { + std::array df{}; + double f_i = 0.0; + for (ssize_t i = 0; i < x_.size(); ++i) { + double c = 0.0; + if constexpr (Weighted) { + const double si = s_[i]; + if (si == 0.0) + continue; + Model::eval_and_grad(x_[i], par, f_i, df); + c = -2.0 * (y_[i] - f_i) / (si * si); + } else { + Model::eval_and_grad(x_[i], par, f_i, df); + c = -2.0 * (y_[i] - f_i); + } + for (std::size_t k = 0; k < npar; ++k) + grad[k] += c * df[k]; + } + } + + template + double evaluate_all_impl(const std::vector &par, + std::vector &grad, + std::vector &hess) const { + std::array df{}; + double f_i = 0.0; + double chi2 = 0.0; + for (ssize_t i = 0; i < x_.size(); ++i) { + double w = 1.0; + if constexpr (Weighted) { + const double si = s_[i]; + if (si == 0.0) + continue; + w = 1.0 / (si * si); + } + Model::eval_and_grad(x_[i], par, f_i, df); + const double r_i = y_[i] - f_i; + chi2 += w * r_i * r_i; + const double c = -2.0 * w * r_i; + for (std::size_t k = 0; k < npar; ++k) { + grad[k] += c * df[k]; + const double h_k = 2.0 * w * df[k]; + double *column = hess.data() + k * (k + 1) / 2; + for (std::size_t j = 0; j <= k; ++j) + column[j] += h_k * df[j]; + } + } + return chi2; + } + + NDView x_; + NDView y_; + NDView s_; + bool weighted_; +}; + +/** + * @brief Chi-squared FCN with analytic gradient, used by Migrad. + * + * Providing the analytic gradient avoids the 2*npar extra function + * evaluations per Minuit step that finite differences would need. + */ +template +class Chi2Model1DGrad : public ROOT::Minuit2::FCNGradientBase { + public: + /** @brief An empty y_err view selects an unweighted fit. */ + Chi2Model1DGrad(NDView x, NDView y, + NDView y_err = NDView{}) + : data_(x, y, y_err) {} + + ~Chi2Model1DGrad() override = default; + + double operator()(const std::vector &par) const override { + return data_.value(par); + } + + std::vector + Gradient(const std::vector &par) const override { + return data_.gradient(par); + } + + /** @brief Error definition: 1.0 for chi-squared (delta_chi2 = 1 -> + * 1-sigma). */ + double Up() const override { return 1.0; } + + private: + Chi2Data1D data_; +}; + +/** + * @brief Chi-squared FCN for Fumili. + * + * EvaluateAll fills the value, gradient and Gauss-Newton Hessian cached by + * FumiliFCNBase in one pass over the data, so Fumili never needs numerical + * derivatives. + */ +template +class Chi2Model1DFumili : public ROOT::Minuit2::FumiliFCNBase { + public: + /** @brief An empty y_err view selects an unweighted fit. */ + Chi2Model1DFumili(NDView x, NDView y, + NDView y_err = NDView{}) + : ROOT::Minuit2::FumiliFCNBase(static_cast(Model::npar)), + data_(x, y, y_err) {} + + double operator()(const std::vector &par) const override { + return data_.value(par); + } + + /** @brief Error definition: 1.0 for chi-squared (delta_chi2 = 1 -> + * 1-sigma). */ + double Up() const override { return 1.0; } + + void EvaluateAll(const std::vector &par) override { + SetFCNValue(data_.evaluate_all(par, Gradient(), Hessian())); + } + + private: + Chi2Data1D data_; +}; + +} // namespace func + +} // namespace aare diff --git a/src/fit/DampedGaussNewton.hpp b/src/fit/DampedGaussNewton.hpp new file mode 100644 index 00000000..fdf1670b --- /dev/null +++ b/src/fit/DampedGaussNewton.hpp @@ -0,0 +1,597 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include +#include +#include +#include +#include + +namespace aare::detail { + +/** @brief Fixed-size vector and square matrix of the solvers, on the stack. */ +template using Vec = std::array; +template using Mat = std::array, N>; + +/** @brief Outcome of one of the built-in minimizers. */ +struct MinimizerResult { + bool valid = false; + int calls = 0; + int iterations = 0; + double chi2 = 0.0; +}; + +/** + * @brief Cholesky factorisation of the leading m x m block of the symmetric + * positive-definite matrix A. Writes the lower triangle of L with + * A = L L^T. Returns false when the block is not positive definite. + * + * The helpers below clamp m to N. The callers never exceed it, but with the + * bound visible gcc's range analysis no longer reports subscripts of the + * one-parameter instantiations as out of range. + */ +template bool cholesky_factor(int m, const Mat &A, Mat &L) { + m = std::min(m, N); + for (int i = 0; i < m; ++i) { + for (int j = 0; j <= i; ++j) { + double sum = A[i][j]; + for (int k = 0; k < j; ++k) + sum -= L[i][k] * L[j][k]; + if (i == j) { + if (!(sum > 0.0)) + return false; + L[i][i] = std::sqrt(sum); + } else { + L[i][j] = sum / L[j][j]; + } + } + } + return true; +} + +/** @brief Solves L L^T z = b for the leading m x m block of the factor L. */ +template +void cholesky_substitute(int m, const Mat &L, const Vec &b, Vec &z) { + m = std::min(m, N); + Vec t{}; + for (int i = 0; i < m; ++i) { + double sum = b[i]; + for (int k = 0; k < i; ++k) + sum -= L[i][k] * t[k]; + t[i] = sum / L[i][i]; + } + for (int i = m - 1; i >= 0; --i) { + double sum = t[i]; + for (int k = i + 1; k < m; ++k) + sum -= L[k][i] * z[k]; + z[i] = sum / L[i][i]; + } +} + +/** @brief Solves A z = b for the leading m x m block of the SPD matrix A. */ +template +bool cholesky_solve(int m, const Mat &A, const Vec &b, Vec &z) { + Mat L{}; + if (!cholesky_factor(m, A, L)) + return false; + cholesky_substitute(m, L, b, z); + return true; +} + +/** @brief Inverse of the leading m x m block of the SPD matrix A. */ +template bool invert_spd(int m, const Mat &A, Mat &inv) { + m = std::min(m, N); + Vec e{}; + Vec col{}; + for (int j = 0; j < m; ++j) { + for (int i = 0; i < m; ++i) + e[i] = (i == j) ? 1.0 : 0.0; + if (!cholesky_solve(m, A, e, col)) + return false; + for (int i = 0; i < m; ++i) + inv[i][j] = col[i]; + } + return true; +} + +/** + * @brief Gauss-Newton parameter errors sqrt(diag(H^-1)) over the parameters + * that are not skipped, where H is the upper triangle of the Gauss-Newton + * Hessian of chi2 / 2 over all N parameters. Skipped parameters (fixed, or + * frozen on a limit) report 0. Returns false, leaving all errors 0, when the + * block cannot be inverted. + */ +template +bool gauss_newton_errors(const Mat &H, const std::array &skip, + double *err) { + std::fill(err, err + N, 0.0); + Mat A{}; + Mat inv{}; + int m = 0; + for (int a = 0; a < N; ++a) { + if (skip[a]) + continue; + int mb = 0; + for (int c = 0; c < N; ++c) { + if (skip[c]) + continue; + A[m][mb++] = a <= c ? H[a][c] : H[c][a]; + } + ++m; + } + if (m == 0 || !invert_spd(m, A, inv)) + return false; + int i = 0; + for (int a = 0; a < N; ++a) { + if (skip[a]) + continue; + err[a] = std::sqrt(std::max(inv[i][i], 0.0)); + ++i; + } + return true; +} + +/** + * @brief Damped Gauss-Newton iteration shared by the built-in minimizers. + * + * The problem is described by an evaluator called as + * `eval(const Vec &p, Normal &out)`: at the point p (all NP parameters, + * fixed ones at their values) it fills out.F = chi2 / 2, the gradient + * out.g = dF/dp and the upper triangle of the Gauss-Newton Hessian out.H, + * both over all NP parameters, and returns false when the point is not + * admissible. LevenbergMarquardt evaluates residuals and the Jacobian of + * the full model, VariableProjection the projected problem of the nonlinear + * parameters. The driver extracts the free parameters and implements + * + * - Marquardt damping u * diag(H), starting at u = damping0, with the + * update rule of Nielsen (Madsen, Nielsen and Tingleff, Methods for + * Non-Linear Least Squares Problems, 2004). The damping also bounds the + * accuracy of the last accepted step, so a solver that converges in few + * iterations should start with a small damping. + * - Limits: a trial step that leaves the allowed box is reflected at the + * limit. The step truncated at the limit is evaluated as well and the + * better of the two is kept. A parameter sitting on a limit whose gradient + * points outward is frozen for that iteration. + * - Convergence in Minuit's convention: the iteration stops when the + * estimated distance to the minimum of chi2 (EDM, from the Gauss-Newton + * Hessian) drops below 0.002 * tolerance. A start point that passes + * already still takes one undamped step, see converged(). When the damped + * step becomes negligible for every parameter before that, the iteration + * has stalled and the fit is valid only if the EDM test passes at that + * point. + * - max_calls bounds the number of evaluations: the initial point and every + * trial point count, so an iteration costs one evaluation, or two when a + * limit is crossed. + * - With reject_degenerate, a trial point at which the model has lost its + * sensitivity to a free parameter (its Hessian diagonal collapsed by more + * than eight orders of magnitude relative to the current point or to the + * start point) is rejected like an inadmissible point, so the damping + * grows and a shorter step is tried instead. + * - With polish, a converged fit takes one more undamped Gauss-Newton step + * and keeps it when it lowers chi2, see polish_step(). + * + * After fit() the final point, its Normal and the frozen parameters remain + * available for the error computation. + */ +template class DampedGaussNewton { + public: + static_assert(NP >= 1 && NP <= 16, + "DampedGaussNewton keeps NP x NP matrices on the stack"); + + struct Normal { + double F = 0.0; // chi2 / 2 + Vec g{}; // dF/dp + Mat H{}; // Gauss-Newton Hessian, upper triangle + std::array aux{}; // evaluator payload, kept with the point + }; + + struct Options { + double tolerance = 0.5; // EDM tolerance, Minuit's convention + int max_calls = 100; // budget of evaluations + double damping0 = 0.1; // initial Marquardt damping + bool reject_degenerate = false; + bool polish = false; + }; + + template + MinimizerResult fit(Eval &eval, const Vec &start, const Vec &lower, + const Vec &upper, const std::array &fixed, + const Options &opt) { + nfree_ = 0; + accepted_ = 0; + for (int k = 0; k < NP; ++k) { + lower_[k] = lower[k]; + upper_[k] = upper[k]; + p_[k] = start[k]; + fixed_[k] = fixed[k]; + free_pos_[k] = -1; + if (!fixed[k]) { + free_pos_[k] = nfree_; + free_[nfree_++] = k; + } + } + for (int a = 0; a < nfree_; ++a) { + q_[a] = p_[free_[a]]; + active_[a] = false; + } + + MinimizerResult res; + if (!evaluate(eval, q_, cur_)) + return res; + res.calls = 1; + for (int a = 0; a < nfree_; ++a) + start_diag_[a] = cur_.H[free_[a]][free_[a]]; + const double edm_target = 0.002 * opt.tolerance; + const int max_calls = opt.max_calls; + const bool reject_degenerate = opt.reject_degenerate; + double u = opt.damping0; // damping relative to diag(H) + double nu = 2.0; + + for (;; ++res.iterations) { + extract_free(); + mark_active(); + if (gmax_ < 1e-12) { + edm(); // the undamped step for converged() + res.valid = + converged(eval, res, max_calls, reject_degenerate, false); + break; + } + // Solve for the damped step first. The solve yields + // g^T (H + u D)^-1 g, a lower bound on the EDM, so the undamped + // factorisation behind edm() is only needed when that bound + // leaves room for convergence. + const bool have_step = solve_step(u); + if (!have_step || bound_ < edm_margin * edm_target) { + const double e = edm(); + if (e < edm_target) { + // A point already deep inside the tolerance needs no + // polishing. + res.valid = + converged(eval, res, max_calls, reject_degenerate, + opt.polish && e >= 0.01 * edm_target); + break; + } + } + if (res.calls >= max_calls) + break; + if (!have_step) { + u *= nu; + nu *= 2.0; + if (u > 1e32) + break; + continue; + } + // A stalled iteration is not a minimum: repeated rejections + // shrink the damped step until it is negligible anywhere. The + // test looks at the step before it is reflected or truncated at + // a limit, so a step that a limit folds back onto the current + // point still gets its truncated alternative evaluated. + if (negligible_step()) { + res.valid = edm() < edm_target; + break; + } + bool moved = false; + bool crossed = false; + for (int a = 0; a < nfree_; ++a) { + const int k = free_[a]; + const double v = q_[a] + delta_[a]; + const double truncated = std::clamp(v, lower_[k], upper_[k]); + double reflected = v; + if (truncated != v) { + crossed = true; + if (v < lower_[k]) + reflected = 2.0 * lower_[k] - v; + if (v > upper_[k]) + reflected = 2.0 * upper_[k] - v; + reflected = std::clamp(reflected, lower_[k], upper_[k]); + } + q_new_[a] = reflected; + q_alt_[a] = truncated; + moved = moved || reflected != q_[a]; + } + // A reflection that lands on the current point needs no + // evaluation; the truncated step below is the only candidate. + bool ok = false; + if (moved) { + ++res.calls; + ok = evaluate(eval, q_new_, trial_) && + !(reject_degenerate && degenerate(trial_)); + } + if (crossed && res.calls < max_calls) { + // Also try the step truncated at the limit, keep the better. + ++res.calls; + const bool ok_alt = evaluate(eval, q_alt_, alt_) && + !(reject_degenerate && degenerate(alt_)); + if (ok_alt && (!ok || alt_.F < trial_.F)) { + q_new_ = q_alt_; + std::swap(trial_, alt_); + ok = true; + } + } + const double rho = ok ? (cur_.F - trial_.F) / predicted_ : -1.0; + if (rho > 0.0) { + // The trial becomes the current point. + q_ = q_new_; + std::swap(cur_, trial_); + for (int a = 0; a < nfree_; ++a) + p_[free_[a]] = q_[a]; + ++accepted_; + const double t = 2.0 * rho - 1.0; + u *= std::max(1.0 / 3.0, 1.0 - t * t * t); + nu = 2.0; + } else { + u *= nu; + nu *= 2.0; + if (u > 1e32) + break; + } + } + if (!res.valid) + return res; + + // p_ tracks the last evaluated point, which may be a rejected trial. + for (int a = 0; a < nfree_; ++a) + p_[free_[a]] = q_[a]; + res.chi2 = 2.0 * cur_.F; + return res; + } + + /** @brief The final point, all NP parameters. */ + const Vec &point() const { return p_; } + + /** @brief F, gradient and Hessian at the final point. */ + const Normal &normal() const { return cur_; } + + /** + * @brief True for a parameter without an error: fixed, frozen on a limit, + * or ending exactly on one. The frozen flag alone misses the last case, + * because it needs the gradient to point outward and that gradient is + * zero when the unconstrained minimum coincides with the limit. + */ + bool skipped(int k) const { + return fixed_[k] || (free_pos_[k] >= 0 && active_[free_pos_[k]]) || + p_[k] <= lower_[k] || p_[k] >= upper_[k]; + } + + /** @brief Gauss-Newton errors at the final point, see + * gauss_newton_errors(). */ + void errors(double *err_out) const { + std::array skip; + for (int k = 0; k < NP; ++k) + skip[k] = skipped(k); + gauss_newton_errors(cur_.H, skip, err_out); + } + + private: + template + bool evaluate(Eval &eval, const Vec &q, Normal &out) { + for (int a = 0; a < nfree_; ++a) + p_[free_[a]] = q[a]; + return eval(p_, out); + } + + // Gradient and Hessian of the free parameters at the current point. + void extract_free() { + gmax_ = 0.0; + for (int a = 0; a < nfree_; ++a) { + const int ka = free_[a]; + g_[a] = cur_.g[ka]; + for (int b = 0; b < nfree_; ++b) { + const int kb = free_[b]; + jtj_[a][b] = ka <= kb ? cur_.H[ka][kb] : cur_.H[kb][ka]; + } + gmax_ = std::max(gmax_, std::abs(g_[a])); + } + } + + // A parameter sitting on a limit with the descent direction pointing + // outward is frozen for this iteration. Position and gradient are both + // taken at the current point q_, so a parameter stays frozen at the end + // only where the constrained minimum lies on its limit. An earlier + // version tested the last evaluated point p_, a rejected trial, so that a + // trial that came off a limit unfroze the parameter: testing q_ had left + // more FallingScurve and GaussianChargeSharing fits stuck with the width + // on its lower limit and a higher chi2. But pairing a trial position with + // the current gradient could freeze an interior parameter and stop the + // fit there. The width no longer reaches its limit that way, because + // LevenbergMarquardt now rejects degenerate trial points as + // VariableProjection does, see degenerate(). + void mark_active() { + for (int a = 0; a < nfree_; ++a) { + const int k = free_[a]; + active_[a] = (q_[a] <= lower_[k] && g_[a] > 0.0) || + (q_[a] >= upper_[k] && g_[a] < 0.0); + } + } + + // True when the damped step is negligible for every parameter that is + // not frozen: below 1e-8 of its value plus its own scale 1/sqrt(H_aa), + // the parameter change that raises chi2 by about one. Judging each + // parameter on its own scale keeps a large parameter (an amplitude of + // 1e10, a peak position at 1e9) from hiding the steps of the others. + bool negligible_step() const { + for (int a = 0; a < nfree_; ++a) { + if (active_[a]) + continue; + const double scale = + std::abs(q_[a]) + 1.0 / std::sqrt(std::max(jtj_[a][a], 1e-300)); + if (!(std::abs(delta_[a]) <= 1e-8 * scale)) + return false; + } + return true; + } + + // A trial point where the Hessian diagonal of a free parameter collapsed + // relative to the current point or to the start point: the model no + // longer responds to that parameter there, so the iteration could stop + // at a spurious stationary point (a step function with the width driven + // to zero, for example). The start point catches a collapse spread over + // many accepted steps, each of which passes the test against its + // predecessor. + bool degenerate(const Normal &trial) const { + for (int a = 0; a < nfree_; ++a) { + if (active_[a]) + continue; + const int k = free_[a]; + const double ref = std::max(cur_.H[k][k], start_diag_[a]); + if (!(trial.H[k][k] >= 1e-8 * ref)) + return true; + } + return false; + } + + // Minuit's estimated distance to the minimum of chi2, using the + // Gauss-Newton Hessian: g^T H^-1 g with g the gradient of chi2/2. + // Also keeps the undamped Gauss-Newton step for polish_step(). + double edm() { + Mat A{}; + Vec b{}; + Vec z{}; + const int m = pack(A, b, 0.0); + for (int a = 0; a < nfree_; ++a) + gn_step_[a] = 0.0; + if (m == 0) + return 0.0; + if (!cholesky_solve(m, A, b, z)) + return std::numeric_limits::infinity(); + double e = 0.0; + int i = 0; + for (int a = 0; a < nfree_; ++a) { + if (active_[a]) + continue; + gn_step_[a] = -z[i]; + e += b[i] * z[i]; + ++i; + } + return e; + } + + // The convergence test passed. A point that passes it before any step + // was accepted may merely lie within the tolerance of chi2's scale, as + // the start estimate of an unweighted fit of data of small magnitude + // does, so it still takes the undamped step of polish_step(), as Migrad + // always iterates at least once. Without the budget for that step the + // fit has not converged. With polish, a converged fit takes the same + // step when the budget allows. Returns whether the fit is valid. + template + bool converged(Eval &eval, MinimizerResult &res, int max_calls, + bool reject_degenerate, bool polish) { + const bool first = accepted_ == 0 && nfree_ > 0; + if (!first && !polish) + return true; + if (res.calls >= max_calls) + return !first; + polish_step(eval, res, reject_degenerate); + return true; + } + + // One undamped Gauss-Newton step from a point that passed the EDM test, + // kept when it lowers F. The test stops the iteration within the + // tolerance, but a solver that converges in few iterations stops while + // the damping is still noticeable, so its last accepted step fell short + // of the minimum by about the damping factor. The extra evaluation lands + // at the minimum to the accuracy of a full Gauss-Newton step. + template + void polish_step(Eval &eval, MinimizerResult &res, bool reject_degenerate) { + for (int a = 0; a < nfree_; ++a) { + const int k = free_[a]; + q_new_[a] = std::clamp(q_[a] + gn_step_[a], lower_[k], upper_[k]); + } + ++res.calls; + const bool ok = evaluate(eval, q_new_, trial_) && + !(reject_degenerate && degenerate(trial_)) && + trial_.F < cur_.F; + if (ok) { + q_ = q_new_; + std::swap(cur_, trial_); + extract_free(); + } + for (int a = 0; a < nfree_; ++a) + p_[free_[a]] = q_[a]; + if (ok) + mark_active(); + } + + // Copy the non-frozen block of H (+ u * diag) and g into A and b. + int pack(Mat &A, Vec &b, double u) const { + double mean_diag = 0.0; + for (int a = 0; a < nfree_; ++a) + mean_diag += jtj_[a][a]; + mean_diag = std::max(mean_diag / std::max(nfree_, 1), 1e-300); + int m = 0; + for (int a = 0; a < nfree_; ++a) { + if (active_[a]) + continue; + int mb = 0; + for (int c = 0; c < nfree_; ++c) { + if (active_[c]) + continue; + A[m][mb++] = jtj_[a][c]; + } + A[m][m] += u * std::max(jtj_[a][a], 1e-10 * mean_diag); + b[m] = g_[a]; + ++m; + } + return m; + } + + // Damped Gauss-Newton step (H + u D) delta = -g for the non-frozen + // parameters, the predicted decrease of F for that step, and + // bound_ = g^T (H + u D)^-1 g, which is at most the EDM because u D is + // positive semi-definite. + bool solve_step(double u) { + Mat A{}; + Vec b{}; + Vec z{}; + const int m = pack(A, b, u); + for (int a = 0; a < nfree_; ++a) + delta_[a] = 0.0; + predicted_ = 0.0; + bound_ = 0.0; + if (m == 0) + return true; + if (!cholesky_solve(m, A, b, z)) + return false; + int i = 0; + for (int a = 0; a < nfree_; ++a) { + if (active_[a]) + continue; + delta_[a] = -z[i]; + bound_ += b[i] * z[i]; + predicted_ += + 0.5 * delta_[a] * + (u * std::max(jtj_[a][a], 1e-300) * delta_[a] - g_[a]); + ++i; + } + return predicted_ > 0.0; + } + + int nfree_ = 0; + int accepted_ = 0; // trial points accepted in this fit + std::array free_{}; + std::array free_pos_{}; + std::array fixed_{}; + std::array active_{}; + Vec lower_{}; + Vec upper_{}; + Vec p_{}; + Vec q_{}; + Vec q_new_{}; + Vec q_alt_{}; + Vec g_{}; + Vec delta_{}; + Vec gn_step_{}; + Vec start_diag_{}; // Hessian diagonal at the start point + Mat jtj_{}; + // The two solves that bound the EDM differ by rounding only when the + // damping is negligible; the margin keeps the stop decision exact. + static constexpr double edm_margin = 1.0 + 1e-6; + double gmax_ = 0.0; + double predicted_ = 0.0; + double bound_ = 0.0; + Normal cur_; + Normal trial_; + Normal alt_; +}; + +} // namespace aare::detail diff --git a/src/fit/Fit.cpp b/src/fit/Fit.cpp new file mode 100644 index 00000000..8b34c177 --- /dev/null +++ b/src/fit/Fit.cpp @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/Fit.hpp" +#include "FitHelpers.hpp" +#include "FitMinuit2.hpp" +#include "LevenbergMarquardt.hpp" +#include "VariableProjection.hpp" +#include "aare/Models.hpp" +#include "aare/utils/par.hpp" +#include "aare/utils/task.hpp" +#include +#include +#include +#include +#include +#include +#include +#include + +namespace aare { +// ============================================================================ +// FitModel — method definitions +// ============================================================================ + +template +FitModel::FitModel(unsigned int strategy, unsigned int max_calls, + double tolerance, bool compute_errors, + Minimizer minimizer) + : strategy_(strategy), max_calls_(max_calls), tolerance_(tolerance), + compute_errors_(compute_errors), minimizer_(minimizer) { + for (std::size_t i = 0; i < npar; ++i) { + lower_[i] = Model::param_info[i].default_lo; + upper_[i] = Model::param_info[i].default_hi; + } +} + +template +void FitModel::check_index(unsigned int idx) const { + if (idx >= npar) + throw std::out_of_range("FitModel: parameter index " + + std::to_string(idx) + " out of range"); +} + +template +unsigned int FitModel::checked_index(const std::string &name) const { + for (std::size_t i = 0; i < npar; ++i) { + if (name == Model::param_info[i].name) + return static_cast(i); + } + throw std::runtime_error("FitModel: unknown parameter name '" + name + "'"); +} + +template +void FitModel::SetParLimits(unsigned int idx, double lo, double hi) { + check_index(idx); + if (!(lo < hi)) + throw std::runtime_error( + "FitModel: lower limit must be below upper limit for parameter '" + + GetParName(idx) + "'"); + lower_[idx] = lo; + upper_[idx] = hi; +} + +template +void FitModel::FixParameter(unsigned int idx, double val) { + SetParameter(idx, val); + user_fixed_[idx] = true; +} + +template +void FitModel::ReleaseParameter(unsigned int idx) { + check_index(idx); + user_fixed_[idx] = false; +} + +template +void FitModel::ReleaseParameter(const std::string &name) { + ReleaseParameter(checked_index(name)); +} + +template +void FitModel::SetParameter(unsigned int idx, double val) { + check_index(idx); + value_[idx] = val; + user_start_[idx] = true; +} + +template +void FitModel::SetParameter(const std::string &name, double val) { + SetParameter(checked_index(name), val); +} + +template +void FitModel::FixParameter(const std::string &name, double val) { + FixParameter(checked_index(name), val); +} + +template +void FitModel::SetParLimits(const std::string &name, double lo, + double hi) { + SetParLimits(checked_index(name), lo, hi); +} + +template +std::string FitModel::GetParName(unsigned int idx) const { + check_index(idx); + return Model::param_info[idx].name; +} + +template +std::vector FitModel::GetParNames() const { + std::vector names; + names.reserve(npar); + for (std::size_t i = 0; i < npar; ++i) + names.emplace_back(Model::param_info[i].name); + return names; +} + +// ============================================================================ +// fit_pixel / fit_3d +// ============================================================================ + +namespace detail { + +struct NotSeparable {}; + +/** + * @brief Fits pixels with the minimizer selected by the model. + * + * Keep one instance per thread: the buffers of the built-in minimizers are + * reused between pixels so that a data cube is fitted without per-pixel + * allocations. The Minuit2 minimizers create their state per pixel. + */ +template class PixelFitter { + public: + /** + * @brief Fit one pixel. + * + * @param par_out Receives npar fitted values; zeros on failure. + * @param err_out Receives npar errors when non-null; zeros on failure. + * @param chi2 Receives the chi-squared at the minimum; 0 on failure. + * @return true for a valid minimum. + */ + bool fit(const FitModel &model, NDView x, + NDView y, NDView y_err, double *par_out, + double *err_out, double &chi2) { + // Samples with a zero error are ignored. They are dropped here, + // before the start estimate and the minimizer: a masked sample may + // hold NaN, and a zero weight does not remove a NaN from a sum. + if (y_err.size() > 0 && + std::find(y_err.begin(), y_err.end(), 0.0) != y_err.end()) { + keep_x_.clear(); + keep_y_.clear(); + keep_err_.clear(); + for (ssize_t i = 0; i < y.size(); ++i) { + if (y_err[i] == 0.0) + continue; + keep_x_.push_back(x[i]); + keep_y_.push_back(y[i]); + keep_err_.push_back(y_err[i]); + } + const auto n = static_cast(keep_x_.size()); + x = NDView(keep_x_.data(), {n}); + y = NDView(keep_y_.data(), {n}); + y_err = NDView(keep_err_.data(), {n}); + } + // Fewer samples than free parameters leave the fit undetermined, + // and the start estimates read at least two samples. + ssize_t n_free = 0; + for (std::size_t k = 0; k < Model::npar; ++k) + n_free += model.is_user_fixed(static_cast(k)) ? 0 : 1; + if (y.size() < std::max(n_free, 2)) + return fail(par_out, err_out, chi2); + + const auto start = start_values(model, x, y); + + if (model.minimizer() == Minimizer::VarPro) { + // A model without the separable structure, a pixel that does + // not converge and a pixel whose linear solution violates a + // limit fall back to the full solver. + if constexpr (model::is_separable::value) { + const auto res = + vp_.fit(model, x, y, y_err, start, par_out, err_out); + if (res.valid && finite(par_out, err_out, res.chi2)) { + chi2 = res.chi2; + return true; + } + } + return fit_lm(model, x, y, y_err, start, par_out, err_out, chi2); + } + if (model.minimizer() == Minimizer::LevenbergMarquardt) + return fit_lm(model, x, y, y_err, start, par_out, err_out, chi2); + return fit_pixel_minuit2(model, x, y, y_err, start, par_out, err_out, + chi2); + } + + private: + // The built-in solvers do not guard every operation against NaN or + // infinity, so a result is checked before it is reported as a valid fit. + static bool finite(const double *par, const double *err, double chi2) { + bool ok = std::isfinite(chi2); + for (std::size_t k = 0; k < Model::npar; ++k) + ok = ok && std::isfinite(par[k]) && (!err || std::isfinite(err[k])); + return ok; + } + + bool fit_lm(const FitModel &model, NDView x, + NDView y, NDView y_err, + const std::array &start, double *par_out, + double *err_out, double &chi2) { + const auto res = lm_.fit(model, x, y, y_err, start, par_out, err_out); + if (res.valid && finite(par_out, err_out, res.chi2)) { + chi2 = res.chi2; + return true; + } + return fail(par_out, err_out, chi2); + } + + // A failed fit reports zeros. + static bool fail(double *par_out, double *err_out, double &chi2) { + std::fill(par_out, par_out + Model::npar, 0.0); + if (err_out) + std::fill(err_out, err_out + Model::npar, 0.0); + chi2 = 0.0; + return false; + } + + // The samples of the current pixel that have a nonzero error, see fit(). + std::vector keep_x_; + std::vector keep_y_; + std::vector keep_err_; + LevenbergMarquardt lm_; + std::conditional_t::value, + VariableProjection, NotSeparable> + vp_; +}; + +} // namespace detail + +template +NDArray fit_pixel(const FitModel &model, NDView x, + NDView y, NDView y_err) { + constexpr std::size_t npar = Model::npar; + detail::check_fit_pixel_shapes(x, y, y_err); + const bool want_errors = model.compute_errors(); + const auto result_size = + static_cast(want_errors ? (2 * npar + 1) : (npar + 1)); + NDArray result({result_size}, 0.0); + + detail::PixelFitter fitter; + double chi2 = 0.0; + double *par = result.data(); + double *err = want_errors ? result.data() + npar : nullptr; + if (!fitter.fit(model, x, y, y_err, par, err, chi2)) + return result; // all zeros + + result(result_size - 1) = chi2; + return result; +} + +template +NDArray fit_pixel(const FitModel &model, NDView x, + NDView y) { + return fit_pixel(model, x, y, NDView{}); +} + +template +void fit_3d(const FitModel &model, NDView x, + NDView y, NDView y_err, + NDView par_out, NDView err_out, + NDView chi2_out, int n_threads) { + constexpr std::size_t npar = Model::npar; + detail::check_fit_3d_shapes(x, y, y_err, par_out, err_out, chi2_out); + + const bool has_errors = (y_err.size() > 0); + const bool want_par_errors = (err_out.size() > 0) && model.compute_errors(); + + auto process = [&](ssize_t first_row, ssize_t last_row) { + detail::PixelFitter fitter; // buffers reused within the task + std::array par{}; + std::array err{}; + for (ssize_t row = first_row; row < last_row; row++) { + for (ssize_t col = 0; col < y.shape(1); col++) { + NDView values(&y(row, col, 0), {y.shape(2)}); + NDView errors = + has_errors ? NDView(&y_err(row, col, 0), + {y_err.shape(2)}) + : NDView{}; + + double chi2 = 0.0; + fitter.fit(model, x, values, errors, par.data(), + want_par_errors ? err.data() : nullptr, chi2); + + for (std::size_t k = 0; k < npar; ++k) + par_out(row, col, static_cast(k)) = par[k]; + if (want_par_errors) { + for (std::size_t k = 0; k < npar; ++k) + err_out(row, col, static_cast(k)) = err[k]; + } + chi2_out(row, col) = chi2; + } + } + }; + + auto tasks = split_task(0, static_cast(y.shape(0)), n_threads); + RunInParallel(process, tasks); +} + +// ============================================================================ +// Explicit instantiations for all supported model types +// ============================================================================ + +// NOLINTBEGIN +#define AARE_INSTANTIATE_FIT(Model) \ + template class FitModel; \ + template NDArray fit_pixel( \ + const FitModel &, NDView, NDView, \ + NDView); \ + template NDArray fit_pixel( \ + const FitModel &, NDView, NDView); \ + template void fit_3d(const FitModel &, NDView, \ + NDView, NDView, \ + NDView, NDView, \ + NDView, int); + +AARE_INSTANTIATE_FIT(model::Gaussian) +AARE_INSTANTIATE_FIT(model::GaussianErfcPlateau) +AARE_INSTANTIATE_FIT(model::GaussianChargeSharing) +AARE_INSTANTIATE_FIT(model::GaussianChargeSharingKb) +AARE_INSTANTIATE_FIT(model::Pol1) +AARE_INSTANTIATE_FIT(model::Pol2) +AARE_INSTANTIATE_FIT(model::RisingScurve) +AARE_INSTANTIATE_FIT(model::FallingScurve) + +#undef AARE_INSTANTIATE_FIT +// NOLINTEND + +} // namespace aare diff --git a/src/fit/Fit.test.cpp b/src/fit/Fit.test.cpp new file mode 100644 index 00000000..992500e8 --- /dev/null +++ b/src/fit/Fit.test.cpp @@ -0,0 +1,1308 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/Fit.hpp" +#include "aare/FitModel.hpp" +#include "aare/Models.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using aare::Minimizer; +using aare::NDArray; +using aare::NDView; +using Catch::Approx; + +namespace { + +constexpr ssize_t n_points = 61; + +struct Data { + NDArray x; + NDArray y; + NDArray y_err; +}; + +void fill_gaussian(NDArray &x, NDArray &y, + double amplitude, double mean, double sigma) { + for (ssize_t i = 0; i < n_points; ++i) { + x(i) = -6.0 + 0.2 * static_cast(i); + const double z = (x(i) - mean) / sigma; + y(i) = amplitude * std::exp(-0.5 * z * z); + } +} + +// Gaussian with A = 120, mu = 0.8, sigma = 1.3 and unit errors, optionally +// with deterministic pseudo-noise. +Data gaussian_data(bool add_noise = false) { + Data d{NDArray({n_points}), NDArray({n_points}), + NDArray({n_points}, 1.0)}; + fill_gaussian(d.x, d.y, 120.0, 0.8, 1.3); + if (add_noise) { + for (ssize_t i = 0; i < n_points; ++i) + d.y(i) += 3.0 * std::sin(7.3 * static_cast(i)); + } + return d; +} + +// S-curve on 100 points in [0, 100] with unit errors, optionally with +// deterministic pseudo-noise and errors of 30. +template +Data scurve_data(const std::vector &par, bool add_noise = false) { + constexpr ssize_t n = 100; + Data d{NDArray({n}), NDArray({n}), + NDArray({n}, 1.0)}; + for (ssize_t i = 0; i < n; ++i) { + d.x(i) = 100.0 * static_cast(i) / static_cast(n - 1); + d.y(i) = Model::eval(d.x(i), par); + if (add_noise) { + d.y(i) += 30.0 * std::sin(7.3 * static_cast(i)); + d.y_err(i) = 30.0; + } + } + return d; +} + +const char *name(Minimizer m) { + switch (m) { + case Minimizer::Migrad: + return "Migrad"; + case Minimizer::Fumili: + return "Fumili"; + case Minimizer::LevenbergMarquardt: + return "LevenbergMarquardt"; + case Minimizer::VarPro: + return "VarPro"; + } + return "unknown"; +} + +bool uses_minuit2(Minimizer m) { + return m != Minimizer::LevenbergMarquardt && m != Minimizer::VarPro; +} + +template +aare::FitModel make_model(Minimizer minimizer, bool errors = false, + unsigned int max_calls = 100) { + return aare::FitModel(0, max_calls, 0.5, errors, minimizer); +} + +// Minuit2 reaches an active limit through its parameter transformation; +// the built-in solver lands on it exactly. +double limit_margin(Minimizer m) { return uses_minuit2(m) ? 1e-6 : 1e-9; } + +// Relative agreement with the true parameters: the Minuit2 minimizers stop +// at Minuit's EDM tolerance, the built-in solver lands tighter. +double truth_tolerance(Minimizer m) { return uses_minuit2(m) ? 1e-3 : 1e-4; } + +// Hesse errors at a point forced away from the free minimum differ from the +// Gauss-Newton estimates of Fumili and the built-in solver. +double misfit_error_tolerance(Minimizer m) { + return uses_minuit2(m) ? 0.1 : 5e-2; +} + +template +bool on_limit(const aare::FitModel &model, unsigned int idx, + double value) { + const double margin = 1e-6 * std::max(1.0, std::abs(value)); + return std::abs(value - model.lower_limit(idx)) <= margin || + std::abs(value - model.upper_limit(idx)) <= margin; +} + +/** + * Run one configuration through every minimizer and compare Fumili and the + * built-in solver against Migrad: parameters and chi2 within tol (relative), + * unless the candidate found a clearly lower chi2. Errors are compared for + * free parameters that do not end on a limit with tolerance err_tol. + */ +template +void check_minimizers_agree(const std::string &label, Data d, bool weighted, + bool errors, unsigned int max_calls, double tol, + Setup setup, double err_tol = 0.0) { + if (err_tol == 0.0) + err_tol = 10.0 * tol; + constexpr std::size_t npar = Model::npar; + + const auto run = [&](Minimizer minimizer) { + aare::FitModel model(0, max_calls, 0.5, errors, minimizer); + setup(model); + return weighted ? aare::fit_pixel(model, d.x.view(), d.y.view(), + d.y_err.view()) + : aare::fit_pixel(model, d.x.view(), d.y.view()); + }; + aare::FitModel reference_model(0, max_calls, 0.5, errors, + Minimizer::Migrad); + setup(reference_model); + const auto ref = run(Minimizer::Migrad); + + for (const auto candidate : + {Minimizer::Fumili, Minimizer::LevenbergMarquardt, + Minimizer::VarPro}) { + INFO(label << ": " << name(candidate) << " vs Migrad"); + const auto res = run(candidate); + REQUIRE(res.size() == ref.size()); + + const auto ichi = static_cast(errors ? 2 * npar : npar); + const double chi_ref = ref(ichi); + const double chi_res = res(ichi); + if (chi_ref > 0.0 && chi_res > 0.0 && + chi_ref - chi_res > tol * std::max(chi_ref, 1.0)) { + WARN(label + ": " + name(candidate) + " found a lower chi2 (" + + std::to_string(chi_res) + " vs " + std::to_string(chi_ref) + + ")"); + continue; + } + double scale = 0.0; + for (std::size_t k = 0; k < npar; ++k) + scale = std::max(scale, std::abs(ref(static_cast(k)))); + for (std::size_t k = 0; k < npar; ++k) { + const auto i = static_cast(k); + const double denom = + std::max({std::abs(ref(i)), 1e-2 * scale, 1e-12}); + INFO("parameter " + << reference_model.GetParName(static_cast(k))); + CHECK(std::abs(res(i) - ref(i)) / denom <= tol); + } + CHECK(std::abs(chi_res - chi_ref) / std::max(std::abs(chi_ref), 1.0) <= + tol); + if (!errors) + continue; + for (std::size_t k = 0; k < npar; ++k) { + const auto idx = static_cast(k); + const auto i = static_cast(k); + if (reference_model.is_user_fixed(idx) || + on_limit(reference_model, idx, ref(i))) + continue; + const auto ie = static_cast(npar + k); + INFO("error of " << reference_model.GetParName(idx)); + CHECK(std::abs(res(ie) - ref(ie)) / + std::max(std::abs(ref(ie)), 1e-12) <= + err_tol); + } + } +} + +} // namespace + +TEST_CASE("Fit unweighted and weighted noise-free Gaussian data", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + + const auto unweighted_model = make_model(minimizer); + const auto unweighted = + aare::fit_pixel(unweighted_model, d.x.view(), d.y.view()); + + REQUIRE(unweighted.size() == 4); + CHECK(unweighted(0) == Approx(120.0).epsilon(1e-4).margin(1e-6)); + CHECK(unweighted(1) == Approx(0.8).epsilon(1e-4).margin(1e-6)); + CHECK(unweighted(2) == Approx(1.3).epsilon(1e-4).margin(1e-6)); + CHECK(unweighted(3) == Approx(0.0).margin(1e-3)); + + const auto weighted_model = + make_model(minimizer, true); + const auto weighted = + aare::fit_pixel(weighted_model, d.x.view(), d.y.view(), d.y_err.view()); + + REQUIRE(weighted.size() == 7); + CHECK(weighted(0) == Approx(120.0).epsilon(1e-4).margin(1e-6)); + CHECK(weighted(1) == Approx(0.8).epsilon(1e-4).margin(1e-6)); + CHECK(weighted(2) == Approx(1.3).epsilon(1e-4).margin(1e-6)); + // At zero residual the Hesse, Fumili and Gauss-Newton errors coincide. + CHECK(weighted(3) == Approx(0.361).epsilon(2e-2)); + CHECK(weighted(4) == Approx(0.00451).epsilon(2e-2)); + CHECK(weighted(5) == Approx(0.00451).epsilon(2e-2)); + CHECK(weighted(6) == Approx(0.0).margin(1e-3)); +} + +TEST_CASE("Fit weighted Pol1 errors match the analytic formula", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t n = 20; + NDArray x({n}); + NDArray y({n}); + NDArray y_err({n}); + double S = 0.0, Sx = 0.0, Sxx = 0.0; + for (ssize_t i = 0; i < n; ++i) { + x(i) = static_cast(i); + y(i) = 2.0 + 0.5 * x(i); + y_err(i) = 1.0 + 0.1 * static_cast(i); + const double w = 1.0 / (y_err(i) * y_err(i)); + S += w; + Sx += w * x(i); + Sxx += w * x(i) * x(i); + } + const double delta = S * Sxx - Sx * Sx; + + const auto model = make_model(minimizer, true); + const auto res = aare::fit_pixel(model, x.view(), y.view(), y_err.view()); + + REQUIRE(res.size() == 5); + CHECK(res(0) == Approx(2.0).epsilon(1e-6)); + CHECK(res(1) == Approx(0.5).epsilon(1e-6)); + CHECK(res(2) == Approx(std::sqrt(Sxx / delta)).epsilon(1e-6)); + CHECK(res(3) == Approx(std::sqrt(S / delta)).epsilon(1e-6)); + CHECK(res(4) == Approx(0.0).margin(1e-8)); +} + +TEST_CASE("Fit with a fixed parameter", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + + SECTION("fixed at a wrong value") { + auto model = make_model(minimizer, true); + model.FixParameter("sigma", 1.5); + REQUIRE(model.is_user_fixed(2)); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + REQUIRE(res.size() == 7); + CHECK(res(2) == 1.5); + CHECK(res(0) == Approx(111.15).epsilon(1e-3)); + CHECK(res(1) == Approx(0.8).epsilon(1e-3)); + CHECK(res(3) == + Approx(0.274).epsilon(misfit_error_tolerance(minimizer))); + CHECK(res(5) == 0.0); // fixed parameters report no error + CHECK(res(6) == Approx(1684.25).epsilon(1e-3)); + } + + SECTION("fixed at the true value, then released") { + auto model = make_model(minimizer, true); + model.FixParameter(2, 1.3); + auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(0) == Approx(120.0).epsilon(1e-4).margin(1e-6)); + CHECK(res(1) == Approx(0.8).epsilon(1e-4).margin(1e-6)); + CHECK(res(2) == 1.3); + CHECK(res(5) == 0.0); + CHECK(res(6) == Approx(0.0).margin(1e-3)); + + model.ReleaseParameter("sigma"); + REQUIRE_FALSE(model.is_user_fixed(2)); + res = aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(2) == Approx(1.3).epsilon(1e-4).margin(1e-6)); + CHECK(res(5) == Approx(0.00451).epsilon(2e-2)); + } +} + +TEST_CASE("Fit with parameter limits", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + const double margin = limit_margin(minimizer); + const double err_tol = misfit_error_tolerance(minimizer); + + SECTION("active upper limit on mu") { + auto model = make_model(minimizer); + model.SetParLimits("mu", -5.0, 0.5); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + CHECK(res(1) == Approx(0.5).margin(margin)); + CHECK(res(0) == Approx(116.88).epsilon(1e-3)); + CHECK(res(2) == Approx(1.335).epsilon(1e-3)); + CHECK(res(3) == Approx(4301.95).epsilon(1e-3)); + } + + SECTION("active upper limit on sigma, with errors") { + auto model = make_model(minimizer, true); + model.SetParLimits(2, 0.5, 1.0); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(2) == Approx(1.0).margin(margin)); + CHECK(res(1) == Approx(0.8).epsilon(1e-3)); + CHECK(res(3) == Approx(0.336).epsilon(err_tol)); + CHECK(res(5) == 0.0); // on a limit: no error reported + CHECK(res(6) == Approx(5550.61).epsilon(1e-3)); + } + + SECTION("fixed amplitude and an active limit on mu") { + auto model = make_model(minimizer, true); + model.FixParameter("A", 100.0); + model.SetParLimits("mu", 0.6, 0.7); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(0) == 100.0); + CHECK(res(1) == Approx(0.7).margin(margin)); + CHECK(res(2) == Approx(1.4603).epsilon(1e-3)); + CHECK(res(3) == 0.0); + CHECK(res(4) == 0.0); + CHECK(res(5) == Approx(0.00469).epsilon(err_tol)); + CHECK(res(6) == Approx(3631.01).epsilon(1e-3)); + } + + SECTION("inactive limits do not change the result") { + const auto free_model = make_model(minimizer); + const auto reference = + aare::fit_pixel(free_model, d.x.view(), d.y.view()); + + auto model = make_model(minimizer); + model.SetParLimits("mu", -5.0, 5.0); + model.SetParLimits("sigma", 0.5, + std::numeric_limits::infinity()); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + for (ssize_t k = 0; k < 3; ++k) + CHECK(res(k) == Approx(reference(k)).epsilon(1e-4).margin(1e-6)); + } +} + +TEST_CASE("Fit honours user start values", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + + SECTION("start value for mu") { + auto model = make_model(minimizer); + model.SetParameter("mu", 0.0); + REQUIRE(model.is_user_start(1)); + CHECK(model.value(1) == 0.0); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + CHECK(res(0) == Approx(120.0).epsilon(1e-4).margin(1e-6)); + CHECK(res(1) == Approx(0.8).epsilon(1e-4).margin(1e-6)); + CHECK(res(2) == Approx(1.3).epsilon(1e-4).margin(1e-6)); + } + + SECTION("negative Gaussian with all start values given") { + Data n{NDArray({n_points}), NDArray({n_points}), + NDArray({n_points}, 1.0)}; + fill_gaussian(n.x, n.y, -120.0, 0.8, 1.3); + auto model = make_model(minimizer); + model.SetParameter("A", -100.0); + model.SetParameter("mu", 0.5); + model.SetParameter("sigma", 1.0); + const auto res = aare::fit_pixel(model, n.x.view(), n.y.view()); + CHECK(res(0) == Approx(-120.0).epsilon(1e-4).margin(1e-6)); + CHECK(res(1) == Approx(0.8).epsilon(1e-4).margin(1e-6)); + CHECK(res(2) == Approx(1.3).epsilon(1e-4).margin(1e-6)); + } +} + +TEST_CASE("Fit reports failure as zeros when max_calls is exhausted", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + + // Start far from the minimum so that no minimizer converges in 3 calls. + auto model = make_model(minimizer, false, 3); + model.SetParameter("mu", -3.0); + model.SetParameter("sigma", 3.0); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + REQUIRE(res.size() == 4); + for (ssize_t k = 0; k < res.size(); ++k) + CHECK(res(k) == 0.0); + + auto model_err = make_model(minimizer, true, 3); + model_err.SetParameter("mu", -3.0); + model_err.SetParameter("sigma", 3.0); + const auto res_err = + aare::fit_pixel(model_err, d.x.view(), d.y.view(), d.y_err.view()); + REQUIRE(res_err.size() == 7); + for (ssize_t k = 0; k < res_err.size(); ++k) + CHECK(res_err(k) == 0.0); +} + +TEST_CASE("FitModel validates parameter names, indices and limits", "[fit]") { + aare::FitModel model; + CHECK(model.GetParNames() == std::vector{"A", "mu", "sigma"}); + CHECK(model.GetParName(2) == "sigma"); + CHECK(model.GetNpar() == 3); + CHECK(model.strategy() == 0); + CHECK(model.minimizer() == Minimizer::Migrad); + CHECK(model.lower_limit(2) == 1e-12); + CHECK(std::isinf(model.upper_limit(2))); + + CHECK_THROWS_AS(model.SetParameter("nope", 1.0), std::runtime_error); + CHECK_THROWS_AS(model.SetParameter(3, 1.0), std::out_of_range); + CHECK_THROWS_AS(model.GetParName(5), std::out_of_range); + CHECK_THROWS_AS(model.SetParLimits(0, 1.0, 1.0), std::runtime_error); + CHECK_THROWS_AS(model.FixParameter("nope", 1.0), std::runtime_error); + CHECK_NOTHROW(model.SetParLimits("mu", -1.0, 1.0)); + CHECK(model.lower_limit(1) == -1.0); + CHECK(model.upper_limit(1) == 1.0); + + model.SetMinimizer(Minimizer::LevenbergMarquardt); + CHECK(model.minimizer() == Minimizer::LevenbergMarquardt); +} + +TEST_CASE("fit_pixel rejects arrays of different sizes", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + const auto model = make_model(minimizer); + NDView short_y(d.y.data(), {n_points - 1}); + NDView short_err(d.y_err.data(), {n_points - 1}); + CHECK_THROWS_AS(aare::fit_pixel(model, d.x.view(), short_y), + std::runtime_error); + CHECK_THROWS_AS(aare::fit_pixel(model, d.x.view(), d.y.view(), short_err), + std::runtime_error); +} + +TEST_CASE("Copying a FitModel keeps the minimizer", "[fit]") { + const aare::FitModel original( + 0, 100, 0.5, false, Minimizer::LevenbergMarquardt); + const aare::FitModel copied(original); + aare::FitModel assigned; + assigned = original; + + CHECK(copied.minimizer() == Minimizer::LevenbergMarquardt); + CHECK(assigned.minimizer() == Minimizer::LevenbergMarquardt); +} + +TEST_CASE("Fit rising and falling S-curves", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + const std::vector truth{10.0, 0.1, 50.0, 3.0, 1000.0, 2.0}; + + SECTION("rising, free fit with errors") { + auto d = scurve_data(truth); + const auto model = + make_model(minimizer, true, 500); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + REQUIRE(res.size() == 13); + for (std::size_t k = 0; k < truth.size(); ++k) { + CHECK(res(static_cast(k)) == + Approx(truth[k]).epsilon(1e-3)); + CHECK(res(static_cast(6 + k)) > 0.0); + } + CHECK(res(12) == Approx(0.0).margin(1e-3)); + } + + SECTION("rising, fixed slope and limited width") { + auto d = scurve_data(truth); + auto model = + make_model(minimizer, true, 500); + model.FixParameter("p1", 0.1); + model.SetParLimits("sigma", 1.0, 4.0); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(1) == 0.1); + CHECK(res(7) == 0.0); + CHECK(res(2) == Approx(50.0).epsilon(1e-3)); + CHECK(res(3) == Approx(3.0).epsilon(1e-3)); + CHECK(res(9) > 0.0); + } + + SECTION("rising, active limit on the threshold") { + auto d = scurve_data(truth); + auto model = + make_model(minimizer, false, 500); + model.SetParLimits("mu", 40.0, 48.0); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + CHECK(res(2) == Approx(48.0).margin(limit_margin(minimizer))); + CHECK(res(6) > 0.0); // chi2 of the constrained fit + } + + SECTION("falling") { + auto d = scurve_data(truth); + const auto model = + make_model(minimizer, false, 500); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + REQUIRE(res.size() == 7); + for (std::size_t k = 0; k < truth.size(); ++k) + CHECK(res(static_cast(k)) == + Approx(truth[k]).epsilon(1e-3)); + } +} + +TEST_CASE("Fit a Gaussian data cube in parallel", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + NDArray x({n_points}); + NDArray values({n_points}); + fill_gaussian(x, values, 80.0, -0.6, 0.9); + + NDArray y({2, 2, n_points}); + for (ssize_t row = 0; row < 2; ++row) { + for (ssize_t col = 0; col < 2; ++col) { + for (ssize_t i = 0; i < n_points; ++i) + y(row, col, i) = values(i); + } + } + + NDArray par({2, 2, 3}); + NDArray chi2({2, 2}); + const auto model = make_model(minimizer); + + aare::fit_3d(model, x.view(), y.view(), NDView{}, par.view(), + NDView{}, chi2.view(), 2); + + for (ssize_t row = 0; row < 2; ++row) { + for (ssize_t col = 0; col < 2; ++col) { + CHECK(par(row, col, 0) == Approx(80.0).epsilon(1e-4).margin(1e-6)); + CHECK(par(row, col, 1) == Approx(-0.6).epsilon(1e-4).margin(1e-6)); + CHECK(par(row, col, 2) == Approx(0.9).epsilon(1e-4).margin(1e-6)); + CHECK(chi2(row, col) == Approx(0.0).margin(1e-3)); + } + } + + CHECK_THROWS_AS(aare::fit_3d(model, NDView(x.data(), {10}), + y.view(), NDView{}, par.view(), + NDView{}, chi2.view(), 2), + std::runtime_error); +} + +TEST_CASE("Fit a data cube with errors matches per-pixel fits", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t rows = 4; + constexpr ssize_t cols = 3; + NDArray x({n_points}); + NDArray y({rows, cols, n_points}); + NDArray y_err({rows, cols, n_points}, 1.0); + NDArray tmp({n_points}); + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + fill_gaussian(x, tmp, 80.0 + 10.0 * static_cast(row), + -0.6 + 0.3 * static_cast(col), + 0.9 + 0.1 * static_cast(row)); + for (ssize_t i = 0; i < n_points; ++i) + y(row, col, i) = tmp(i); + } + } + + NDArray par({rows, cols, 3}); + NDArray err({rows, cols, 3}); + NDArray chi2({rows, cols}); + const auto model = make_model(minimizer, true); + + aare::fit_3d(model, x.view(), y.view(), y_err.view(), par.view(), + err.view(), chi2.view(), 3); + + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + NDView yv(&y(row, col, 0), {n_points}); + NDView ev(&y_err(row, col, 0), {n_points}); + const auto single = aare::fit_pixel(model, x.view(), yv, ev); + // reused solver buffers must not leak state between pixels + for (ssize_t k = 0; k < 3; ++k) { + CHECK(par(row, col, k) == Approx(single(k)).margin(1e-12)); + CHECK(err(row, col, k) == Approx(single(3 + k)).margin(1e-12)); + CHECK(err(row, col, k) > 0.0); + } + CHECK(chi2(row, col) == Approx(single(6)).margin(1e-12)); + const double tol = truth_tolerance(minimizer); + CHECK(par(row, col, 0) == + Approx(80.0 + 10.0 * static_cast(row)) + .epsilon(tol) + .margin(1e-6)); + CHECK(par(row, col, 1) == + Approx(-0.6 + 0.3 * static_cast(col)) + .epsilon(tol) + .margin(1e-6)); + CHECK(par(row, col, 2) == + Approx(0.9 + 0.1 * static_cast(row)) + .epsilon(tol) + .margin(1e-6)); + } + } +} + +TEST_CASE("Fitting with every parameter fixed returns the fixed values", + "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + auto d = gaussian_data(); + + auto model = make_model(minimizer, true); + model.FixParameter("A", 120.0); + model.FixParameter("mu", 0.8); + model.FixParameter("sigma", 1.3); + + const auto result = aare::fit_pixel(model, d.x.view(), d.y.view()); + + REQUIRE(result.size() == 7); + CHECK(result(0) == 120.0); + CHECK(result(1) == 0.8); + CHECK(result(2) == 1.3); + for (ssize_t k = 3; k < 6; ++k) + CHECK(result(k) == 0.0); + CHECK(result(6) == Approx(0.0).margin(1e-12)); +} + +TEST_CASE("All minimizers agree on Gaussian fits", "[fit]") { + using aare::model::Gaussian; + const auto none = [](auto &) {}; + check_minimizers_agree("exact, unweighted", gaussian_data(), + false, false, 100, 1e-4, none); + check_minimizers_agree("exact, weighted with errors", + gaussian_data(), true, true, 100, 1e-4, + none); + check_minimizers_agree( + "sigma fixed at a wrong value", gaussian_data(), true, true, 100, 1e-3, + [](auto &m) { m.FixParameter("sigma", 1.5); }, 0.2); + check_minimizers_agree( + "sigma fixed at the true value", gaussian_data(), true, true, 100, 1e-3, + [](auto &m) { m.FixParameter("sigma", 1.3); }); + check_minimizers_agree( + "active upper limit on mu", gaussian_data(), false, false, 100, 1e-3, + [](auto &m) { m.SetParLimits("mu", -5.0, 0.5); }); + check_minimizers_agree( + "inactive limits", gaussian_data(), false, false, 100, 1e-4, + [](auto &m) { m.SetParLimits("mu", -5.0, 5.0); }); + check_minimizers_agree( + "active upper limit on sigma", gaussian_data(), true, true, 100, 1e-3, + [](auto &m) { m.SetParLimits("sigma", 0.5, 1.0); }, 0.2); + check_minimizers_agree( + "user start value", gaussian_data(), false, false, 100, 1e-4, + [](auto &m) { m.SetParameter("mu", 0.0); }); + check_minimizers_agree( + "fixed amplitude and active limit on mu", gaussian_data(), true, true, + 100, 1e-3, + [](auto &m) { + m.FixParameter("A", 100.0); + m.SetParLimits("mu", 0.6, 0.7); + }, + 0.2); + check_minimizers_agree("noisy, weighted with errors", + gaussian_data(true), true, true, 500, 1e-3, + none); + check_minimizers_agree( + "noisy, fixed mu and active limit on sigma", gaussian_data(true), true, + true, 500, 1e-3, + [](auto &m) { + m.FixParameter("mu", 0.8); + m.SetParLimits("sigma", 0.5, 1.1); + }, + 0.2); +} + +TEST_CASE("All minimizers agree on S-curve fits", "[fit]") { + using aare::model::RisingScurve; + const std::vector truth{10.0, 0.1, 50.0, 3.0, 1000.0, 2.0}; + const auto none = [](auto &) {}; + check_minimizers_agree("noisy, weighted with errors", + scurve_data(truth, true), + true, true, 500, 2e-2, none); + check_minimizers_agree("noisy, fixed slope and limited width", + scurve_data(truth, true), + true, true, 500, 1e-2, [](auto &m) { + m.FixParameter("p1", 0.1); + m.SetParLimits("sigma", 1.0, 4.0); + }); + check_minimizers_agree( + "noisy, active limit on the threshold", + scurve_data(truth, true), false, false, 500, 1e-2, + [](auto &m) { m.SetParLimits("mu", 40.0, 48.0); }); + check_minimizers_agree( + "noisy, two fixed, limited amplitude, start value", + scurve_data(truth, true), true, true, 500, 1e-2, + [](auto &m) { + m.FixParameter("sigma", 3.0); + m.FixParameter("p0", 10.0); + m.SetParLimits("A", 900.0, 1100.0); + m.SetParameter("C", 0.0); + }); +} + +TEST_CASE("All minimizers agree on a noisy data cube", "[fit]") { + constexpr ssize_t rows = 3; + constexpr ssize_t cols = 2; + NDArray x({n_points}); + NDArray y({rows, cols, n_points}); + NDArray y_err({rows, cols, n_points}, 1.0); + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + const double mu = -0.6 + 0.3 * static_cast(col); + const double sigma = 0.9 + 0.1 * static_cast(row); + for (ssize_t i = 0; i < n_points; ++i) { + x(i) = -6.0 + 0.2 * static_cast(i); + const double z = (x(i) - mu) / sigma; + y(row, col, i) = 80.0 * std::exp(-0.5 * z * z) + + 2.0 * std::sin(7.3 * static_cast(i)); + } + } + } + + struct Cube { + NDArray par{{rows, cols, 3}}; + NDArray err{{rows, cols, 3}}; + NDArray chi2{{rows, cols}}; + }; + const auto fit_cube = [&](Minimizer minimizer) { + Cube c; + const auto model = + make_model(minimizer, true, 500); + aare::fit_3d(model, x.view(), y.view(), y_err.view(), c.par.view(), + c.err.view(), c.chi2.view(), 2); + return c; + }; + const auto ref = fit_cube(Minimizer::Migrad); + + for (const auto candidate : + {Minimizer::Fumili, Minimizer::LevenbergMarquardt, + Minimizer::VarPro}) { + INFO(name(candidate) << " vs Migrad"); + const auto res = fit_cube(candidate); + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + for (ssize_t k = 0; k < 3; ++k) { + CHECK(res.par(row, col, k) == + Approx(ref.par(row, col, k)).epsilon(1e-3)); + CHECK(res.err(row, col, k) == + Approx(ref.err(row, col, k)).epsilon(2e-2)); + } + CHECK(res.chi2(row, col) == + Approx(ref.chi2(row, col)).epsilon(1e-4)); + } + } + } +} + +namespace { + +// The separable structure of a model must reproduce eval and eval_and_grad: +// f = sum_j par[linear_par[j]] * phi[j], df/dpar[linear_par[j]] = phi[j] and +// df/dpar[nonlinear[k]] = sum_j par[linear_par[j]] * dphi[k][j]. +template +void check_separable(const std::vector &par, + const std::vector &xs) { + using Traits = aare::model::separable_traits; + static_assert(aare::model::is_separable::value); + typename Traits::Basis phi{}; + typename Traits::BasisGrad dphi{}; + std::array g{}; + double f = 0.0; + for (const double x : xs) { + INFO("x = " << x); + Model::basis_and_grad(x, par, phi, dphi); + Model::eval_and_grad(x, par, f, g); + double sum = 0.0; + for (std::size_t j = 0; j < Traits::nlin; ++j) { + const auto idx = Model::linear_par[j]; + sum += par[idx] * phi[j]; + CHECK(g[idx] == Approx(phi[j]).epsilon(1e-12).margin(1e-14)); + } + CHECK(Model::eval(x, par) == Approx(sum).epsilon(1e-12).margin(1e-14)); + CHECK(f == Approx(sum).epsilon(1e-12).margin(1e-14)); + for (std::size_t k = 0; k < Traits::nnl; ++k) { + double d = 0.0; + for (std::size_t j = 0; j < Traits::nlin; ++j) + d += par[Model::linear_par[j]] * dphi[k][j]; + CHECK(g[Traits::nonlinear_par[k]] == + Approx(d).epsilon(1e-12).margin(1e-14)); + } + } + // The bulk basis pass must agree with the per-point one. + if constexpr (aare::model::has_basis_columns::value) { + const auto n = static_cast(xs.size()); + std::vector cols(Traits::nlin * xs.size()); + std::vector dcols(Traits::nnl * Traits::nlin * xs.size()); + Model::basis_columns(xs.data(), n, par, cols.data(), dcols.data()); + for (ssize_t i = 0; i < n; ++i) { + INFO("x = " << xs[static_cast(i)]); + Model::basis_and_grad(xs[static_cast(i)], par, phi, + dphi); + for (std::size_t j = 0; j < Traits::nlin; ++j) + CHECK(cols[j * xs.size() + static_cast(i)] == + Approx(phi[j]).epsilon(1e-12).margin(1e-14)); + for (std::size_t k = 0; k < Traits::nnl; ++k) + for (std::size_t j = 0; j < Traits::nlin; ++j) + CHECK(dcols[(k * Traits::nlin + j) * xs.size() + + static_cast(i)] == + Approx(dphi[k][j]).epsilon(1e-12).margin(1e-14)); + } + } +} + +} // namespace + +TEST_CASE("Separable models are consistent with eval and eval_and_grad", + "[fit]") { + using namespace aare::model; + const std::vector xs = {-4.0, -1.5, 0.3, 2.0, + 7.5, 40.0, 55.0, 120.0}; + check_separable({2.0, -0.5}, xs); + check_separable({2.0, -0.5, 0.05}, xs); + check_separable({120.0, 0.8, 1.3}, xs); + check_separable({80.0, 15.0, 2.0, 1.5}, xs); + check_separable({20.0, 0.05, 50.0, 4.0, 1500.0, 0.3}, + xs); + check_separable( + {20.0, 0.05, 50.0, 4.0, 1500.0, 0.3, 1.1, 0.1}, xs); + check_separable({10.0, 0.1, 50.0, 3.0, 1000.0, 2.0}, xs); + check_separable({10.0, 0.1, 50.0, 3.0, 1000.0, 2.0}, xs); +} + +TEST_CASE("VarPro falls back to LevenbergMarquardt for a limit " + "on a linear parameter", + "[fit]") { + using aare::model::FallingScurve; + const std::vector truth{10.0, 0.1, 50.0, 3.0, 1000.0, 2.0}; + auto d = scurve_data(truth); + + auto vp = make_model(Minimizer::VarPro, true, 500); + auto lm = + make_model(Minimizer::LevenbergMarquardt, true, 500); + // The unconstrained amplitude is 1000, so the linear solve cannot + // honour this limit and the pixel is refitted with the full solver. + vp.SetParLimits("A", 0.0, 500.0); + lm.SetParLimits("A", 0.0, 500.0); + + const auto res_vp = + aare::fit_pixel(vp, d.x.view(), d.y.view(), d.y_err.view()); + const auto res_lm = + aare::fit_pixel(lm, d.x.view(), d.y.view(), d.y_err.view()); + REQUIRE(res_vp.size() == res_lm.size()); + for (ssize_t k = 0; k < res_vp.size(); ++k) + CHECK(res_vp(k) == res_lm(k)); + CHECK(res_vp(4) == Approx(500.0).margin(1e-9)); + CHECK(res_vp(12) > 0.0); +} + +TEST_CASE("fast_exp matches std::exp", "[fit]") { + double max_rel = 0.0; + for (int i = 0; i <= 14000; ++i) { + const double x = -700.0 + 1400.0 * static_cast(i) / 14000.0; + const double reference = std::exp(x); + max_rel = + std::max(max_rel, std::abs(aare::model::fast_exp(x) - reference) / + reference); + } + CHECK(max_rel < 1e-15); + CHECK(aare::model::fast_exp(0.0) == 1.0); + CHECK(aare::model::fast_exp(-1000.0) == aare::model::fast_exp(-700.0)); +} + +TEST_CASE("Fit with a one-sided limit far from the start value", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t n = 20; + NDArray x({n}); + NDArray y({n}); + for (ssize_t i = 0; i < n; ++i) { + x(i) = static_cast(i); + y(i) = 2.0 + 0.5 * x(i); + } + const double inf = std::numeric_limits::infinity(); + const bool upper = GENERATE(true, false); + auto model = make_model(minimizer); + if (upper) + model.SetParLimits(0, -inf, 1e9); + else + model.SetParLimits(0, -1e9, inf); + + const auto res = aare::fit_pixel(model, x.view(), y.view()); + CHECK(res(0) == Approx(2.0).epsilon(1e-6)); + CHECK(res(1) == Approx(0.5).epsilon(1e-6)); +} + +TEST_CASE("A parameter ending exactly on a limit reports an error of 0", + "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t n = 3; + NDArray x({n}); + NDArray y({n}, 2.0); + for (ssize_t i = 0; i < n; ++i) + x(i) = static_cast(i); + // The unconstrained minimum p0 = 2 coincides with the limit. + const bool lower = GENERATE(true, false); + auto model = make_model(minimizer, true); + model.FixParameter(1, 0.0); + if (lower) + model.SetParLimits(0, 2.0, 3.0); + else + model.SetParLimits(0, 1.0, 2.0); + + const auto res = aare::fit_pixel(model, x.view(), y.view()); + REQUIRE(res.size() == 5); + CHECK(res(0) == Approx(2.0).epsilon(1e-6)); + CHECK(res(2) == 0.0); + CHECK(res(3) == 0.0); +} + +// --------------------------------------------------------------------------- +// Regressions of the built-in solvers: fits that used to be reported as valid +// although they had stopped away from the minimum or returned garbage. +// --------------------------------------------------------------------------- + +namespace { + +// Standard normal noise from std::mt19937, whose output the standard fixes, +// and a Box-Muller transform written out here: std::normal_distribution +// differs between standard libraries. +std::vector normal_noise(std::size_t n, unsigned seed) { + std::mt19937 rng(seed); + const auto uniform = [&rng] { + return (static_cast(rng()) + 0.5) / 4294967296.0; + }; + std::vector out(n); + for (std::size_t i = 0; i < n; i += 2) { + const double r = std::sqrt(-2.0 * std::log(uniform())); + const double phi = 2.0 * std::acos(-1.0) * uniform(); + out[i] = r * std::cos(phi); + if (i + 1 < n) + out[i + 1] = r * std::sin(phi); + } + return out; +} + +} // namespace + +TEST_CASE("A Gaussian fit to pure noise fails or stays finite and bounded", + "[fit]") { + // Fitting noise, a narrow width can collapse between two scan points; + // the amplitude that fits a single point then diverges. Such a fit must + // be reported as failed, not as a valid result with |A| up to 1e159. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t rows = 20; + constexpr ssize_t cols = 20; + NDArray x({n_points}); + for (ssize_t i = 0; i < n_points; ++i) + x(i) = -6.0 + 0.2 * static_cast(i); + NDArray y({rows, cols, n_points}); + const auto noise = normal_noise(static_cast(y.size()), 1); + std::copy(noise.begin(), noise.end(), y.begin()); + + NDArray par({rows, cols, 3}); + NDArray chi2({rows, cols}); + const auto model = make_model(minimizer); + aare::fit_3d(model, x.view(), y.view(), NDView{}, par.view(), + NDView{}, chi2.view(), 1); + + int absurd = 0; + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + bool ok = std::isfinite(chi2(row, col)); + for (ssize_t k = 0; k < 3; ++k) + ok = ok && std::isfinite(par(row, col, k)) && + std::abs(par(row, col, k)) < 1e6; + absurd += ok ? 0 : 1; + } + } + CHECK(absurd == 0); +} + +TEST_CASE("A step reflected back onto the start point is not convergence", + "[fit]") { + // p0 starts at 0 with limits [-1, 1]; the damped step of +2 reflects at + // the upper limit back onto 0. The step truncated at the limit must still + // be tried: the constrained minimum is p0 = 1. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + NDArray x({4}); + NDArray y({4}, 2.2); + for (ssize_t i = 0; i < 4; ++i) + x(i) = static_cast(i); + + auto model = make_model(minimizer); + model.SetParameter("p0", 0.0); + model.FixParameter("p1", 0.0); + model.SetParLimits("p0", -1.0, 1.0); + const auto res = aare::fit_pixel(model, x.view(), y.view()); + CHECK(res(0) == Approx(1.0).margin(limit_margin(minimizer))); + CHECK(res(2) == Approx(5.76).epsilon(1e-6)); +} + +TEST_CASE("A large parameter does not hide the steps of the others", "[fit]") { + // The negligible-step test used one relative norm over all parameters, + // so an amplitude of 1e10 or a position of 1e9 stopped the fit at the + // start estimate of the width. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + const auto [amplitude, offset] = GENERATE( + std::pair{1e8, 0.0}, std::pair{1e10, 0.0}, std::pair{120.0, 1e9}); + INFO("amplitude " << amplitude << ", offset " << offset); + + Data d{NDArray({n_points}), NDArray({n_points}), + NDArray({n_points})}; + fill_gaussian(d.x, d.y, amplitude, 0.8, 1.3); + for (ssize_t i = 0; i < n_points; ++i) { + d.x(i) += offset; + d.y_err(i) = std::sqrt(d.y(i)) + 1.0; + } + const auto model = make_model(minimizer, false, 500); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + // The EDM tolerance leaves a few 1e-4 relative, far below the errors; + // the stuck fits were 0.7 % (A = 1e8) and 5 % (start estimate) off. + CHECK(res(0) == Approx(amplitude).epsilon(1e-3)); + CHECK(res(1) - offset == Approx(0.8).margin(1e-3)); + CHECK(res(2) == Approx(1.3).epsilon(1e-3)); +} + +TEST_CASE("A rejected trial on a limit does not freeze an interior parameter", + "[fit]") { + // mu starts at 3 inside [0, 5]. A trial truncated onto the lower limit is + // rejected; testing that trial's position against the gradient of the + // accepted point froze mu, and the fit stopped at the start value. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t n = 11; + const double mu = std::sqrt(2.0 * std::log(2.0)); + NDArray x({n}); + NDArray y({n}); + for (ssize_t i = 0; i < n; ++i) { + x(i) = -0.1 + 0.02 * static_cast(i); + y(i) = 10.0 * std::exp(-0.5 * (x(i) - mu) * (x(i) - mu)); + } + auto model = make_model(minimizer); + model.FixParameter("A", 10.0); + model.FixParameter("sigma", 1.0); + model.SetParameter("mu", 3.0); + model.SetParLimits("mu", 0.0, 5.0); + const auto res = aare::fit_pixel(model, x.view(), y.view()); + CHECK(res(1) == Approx(mu).margin(1e-3)); + CHECK(res(3) < 1e-4); +} + +// --------------------------------------------------------------------------- +// Samples with a zero error are ignored. They may hold NaN (a masked dead +// frame), which a zero weight does not remove from a sum, so they must not +// reach the start estimate or the minimizer at all. +// --------------------------------------------------------------------------- + +TEST_CASE("Samples with a zero error are ignored even when NaN", "[fit]") { + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + const double nan = std::numeric_limits::quiet_NaN(); + + constexpr ssize_t n = 10; + NDArray x({n}); + NDArray y({n}); + NDArray y_err({n}, 1.0); + for (ssize_t i = 0; i < n; ++i) { + x(i) = static_cast(i); + y(i) = 2.0 + 0.5 * x(i); + } + const auto line = make_model(minimizer, true); + + // The start estimate of Pol1 uses the first and the last sample. + const auto masked = GENERATE(as{}, 4, 0, n - 1); + SECTION("masked NaN on a line") { + INFO("masked sample " << masked); + y(masked) = nan; + y_err(masked) = 0.0; + const auto res = + aare::fit_pixel(line, x.view(), y.view(), y_err.view()); + CHECK(res(0) == Approx(2.0).epsilon(1e-5).margin(1e-6)); + CHECK(res(1) == Approx(0.5).epsilon(1e-5).margin(1e-6)); + CHECK(res(2) > 0.0); + CHECK(res(4) == Approx(0.0).margin(1e-6)); + } + + SECTION("masked NaN at the peak of a Gaussian") { + auto d = gaussian_data(); + constexpr ssize_t peak = 34; // x = 0.8 + d.y(peak) = nan; + d.y_err(peak) = 0.0; + const auto model = make_model(minimizer); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + const double tol = truth_tolerance(minimizer); + CHECK(res(0) == Approx(120.0).epsilon(tol)); + CHECK(res(1) == Approx(0.8).epsilon(tol)); + CHECK(res(2) == Approx(1.3).epsilon(tol)); + } + + SECTION("too few samples left fail cleanly") { + // One sample left for two free parameters, then none at all. + for (ssize_t i = 1; i < n; ++i) { + y(i) = nan; + y_err(i) = 0.0; + } + auto res = aare::fit_pixel(line, x.view(), y.view(), y_err.view()); + for (ssize_t k = 0; k < res.size(); ++k) + CHECK(res(k) == 0.0); + y_err(0) = 0.0; + res = aare::fit_pixel(line, x.view(), y.view(), y_err.view()); + for (ssize_t k = 0; k < res.size(); ++k) + CHECK(res(k) == 0.0); + } +} + +TEST_CASE("Masked samples in a data cube match per-pixel fits", "[fit]") { + // The kept samples of each pixel go through reused buffers, which must + // not carry anything over from one pixel to the next. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + constexpr ssize_t rows = 2; + constexpr ssize_t cols = 3; + // masked sample per pixel, -1 for none + constexpr ssize_t masked[rows][cols] = {{-1, 10, 0}, + {34, -1, n_points - 1}}; + NDArray x({n_points}); + NDArray tmp({n_points}); + NDArray y({rows, cols, n_points}); + NDArray y_err({rows, cols, n_points}, 1.0); + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + fill_gaussian(x, tmp, 80.0 + 10.0 * static_cast(col), 0.8, + 1.0 + 0.2 * static_cast(row)); + for (ssize_t i = 0; i < n_points; ++i) + y(row, col, i) = tmp(i); + if (masked[row][col] >= 0) { + y(row, col, masked[row][col]) = + std::numeric_limits::quiet_NaN(); + y_err(row, col, masked[row][col]) = 0.0; + } + } + } + + NDArray par({rows, cols, 3}); + NDArray err({rows, cols, 3}); + NDArray chi2({rows, cols}); + const auto model = make_model(minimizer, true); + aare::fit_3d(model, x.view(), y.view(), y_err.view(), par.view(), + err.view(), chi2.view(), 1); + + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + INFO("pixel " << row << ", " << col); + NDView yv(&y(row, col, 0), {n_points}); + NDView ev(&y_err(row, col, 0), {n_points}); + const auto single = aare::fit_pixel(model, x.view(), yv, ev); + for (ssize_t k = 0; k < 3; ++k) { + CHECK(par(row, col, k) == Approx(single(k)).margin(1e-12)); + CHECK(err(row, col, k) == Approx(single(3 + k)).margin(1e-12)); + } + CHECK(chi2(row, col) == Approx(single(6)).margin(1e-12)); + // Close to the truth within the EDM tolerance; the exact check + // is the agreement with the per-pixel fit above. + const double tol = 1e-3; + CHECK(par(row, col, 0) == + Approx(80.0 + 10.0 * static_cast(col)).epsilon(tol)); + CHECK(par(row, col, 1) == Approx(0.8).epsilon(tol)); + CHECK(par(row, col, 2) == + Approx(1.0 + 0.2 * static_cast(row)).epsilon(tol)); + } + } +} + +TEST_CASE("A parameter whose minimum lies on its limit reports no error", + "[fit]") { + // At a minimum that lies exactly on a limit the gradient is zero, so a + // test for an outward gradient does not see the parameter as bound. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + + SECTION("linear parameter") { + NDArray x({4}); + NDArray y({4}); + const double xs[4] = {-1.0, -1.0, 1.0, 1.0}; + for (ssize_t i = 0; i < 4; ++i) { + x(i) = xs[i]; + y(i) = 1.0 + 2.0 * xs[i]; + } + auto model = make_model(minimizer, true); + model.SetParLimits("p0", 0.0, 1.0); + const auto res = aare::fit_pixel(model, x.view(), y.view()); + CHECK(res(0) == Approx(1.0).margin(limit_margin(minimizer))); + CHECK(res(2) == 0.0); + CHECK(res(3) == Approx(0.5).epsilon(1e-6)); // 1 / sqrt(sum x^2) + } + + SECTION("nonlinear parameter") { + auto d = gaussian_data(); + auto model = make_model(minimizer, true); + model.SetParLimits("mu", -5.0, 0.8); + const auto res = + aare::fit_pixel(model, d.x.view(), d.y.view(), d.y_err.view()); + CHECK(res(1) == Approx(0.8).margin(1e-6)); + CHECK(res(4) == 0.0); + CHECK(res(3) > 0.0); + CHECK(res(5) > 0.0); + } +} + +TEST_CASE("A fit does not stop at its start point without taking a step", + "[fit]") { + // The EDM tolerance is absolute in chi2 units. An unweighted fit of data + // of small magnitude passes it at the start estimate already; the fit + // must still take a step, as Migrad always iterates at least once. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + const double amplitude = GENERATE(0.1, 1e-4); + INFO("amplitude " << amplitude); + Data d{NDArray({n_points}), NDArray({n_points}), + NDArray({n_points}, 1.0)}; + fill_gaussian(d.x, d.y, amplitude, 0.8, 1.3); + + // The start estimate of sigma is 1.3617; Migrad ends at 1.3205. + const auto model = make_model(minimizer); + const auto res = aare::fit_pixel(model, d.x.view(), d.y.view()); + CHECK(std::abs(res(2) - 1.3) < 0.03); + + // Without the budget for a single step the fit has not converged. + const auto one_call = + make_model(minimizer, false, 1); + const auto none = aare::fit_pixel(one_call, d.x.view(), d.y.view()); + for (ssize_t k = 0; k < none.size(); ++k) + CHECK(none(k) == 0.0); +} + +TEST_CASE("A polynomial far from x = 0 keeps its precision", "[fit]") { + // On a narrow range far from 0 the columns 1, x, x^2 are nearly + // collinear. VarPro solves them from the normal equations and lost six + // digits at x = 20000; from about x = 60000 it falls back to LM. + const auto minimizer = + GENERATE(Minimizer::Migrad, Minimizer::Fumili, + Minimizer::LevenbergMarquardt, Minimizer::VarPro); + INFO("minimizer " << name(minimizer)); + const auto [lo, span] = + GENERATE(std::pair{8000.0, 1000.0}, std::pair{20000.0, 500.0}, + std::pair{60000.0, 200.0}); + INFO("x in [" << lo << ", " << lo + span << "]"); + // 1.5 + 2e-3 (x - lo) + 3e-7 (x - lo)^2 expanded in powers of x. + const double c[3] = {1.5 - 2e-3 * lo + 3e-7 * lo * lo, 2e-3 - 6e-7 * lo, + 3e-7}; + constexpr ssize_t n = 40; + NDArray x({n}); + NDArray y({n}); + for (ssize_t i = 0; i < n; ++i) { + x(i) = lo + span * static_cast(i) / static_cast(n - 1); + y(i) = c[0] + c[1] * x(i) + c[2] * x(i) * x(i); + } + const auto model = make_model(minimizer); + const auto res = aare::fit_pixel(model, x.view(), y.view()); + for (ssize_t k = 0; k < 3; ++k) + CHECK(res(k) == Approx(c[k]).epsilon(1e-9)); +} diff --git a/src/fit/FitHelpers.hpp b/src/fit/FitHelpers.hpp new file mode 100644 index 00000000..a82599cf --- /dev/null +++ b/src/fit/FitHelpers.hpp @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "aare/FitModel.hpp" +#include "aare/NDView.hpp" +#include +#include +#include + +// Helpers shared by the built-in solver and the optional Minuit2 backend so +// that both apply the same start-value precedence and shape checks. + +namespace aare::detail { + +/** + * @brief Starting values for one pixel. + * + * Model::estimate_par supplies data-driven estimates. User start values and + * fixed values take precedence, and free parameters are clamped into their + * limits. + */ +template +std::array start_values(const FitModel &model, + NDView x, + NDView y) { + auto start = Model::estimate_par(x, y); + for (std::size_t k = 0; k < Model::npar; ++k) { + const auto idx = static_cast(k); + if (model.is_user_fixed(idx) || model.is_user_start(idx)) + start[k] = model.value(idx); + if (!model.is_user_fixed(idx)) + start[k] = std::clamp(start[k], model.lower_limit(idx), + model.upper_limit(idx)); + } + return start; +} + +/** @brief Validate the array sizes passed to fit_pixel. */ +inline void check_fit_pixel_shapes(NDView x, NDView y, + NDView y_err) { + if (x.size() != y.size()) + throw std::runtime_error("fit_pixel: x.size() must match y.size()."); + if (y_err.size() > 0 && y_err.size() != y.size()) + throw std::runtime_error( + "fit_pixel: y and y_err must have identical size."); +} + +/** @brief Validate the array shapes passed to fit_3d. */ +template +void check_fit_3d_shapes(NDView x, NDView y, + NDView y_err, NDView par_out, + NDView err_out, + NDView chi2_out) { + constexpr auto npar = static_cast(Model::npar); + + if (x.size() != y.shape(2)) + throw std::runtime_error("fit_3d: x.size() must match y.shape(2)."); + + if (par_out.shape(0) != y.shape(0) || par_out.shape(1) != y.shape(1) || + par_out.shape(2) != npar) + throw std::runtime_error("par_out must have shape [rows, cols, npar]."); + + if (chi2_out.shape(0) != y.shape(0) || chi2_out.shape(1) != y.shape(1)) + throw std::runtime_error("chi2_out must have shape [rows, cols]."); + + const bool err_out_matches = err_out.shape(0) == y.shape(0) && + err_out.shape(1) == y.shape(1) && + err_out.shape(2) == npar; + + if (y_err.size() > 0) { + if (y.shape(0) != y_err.shape(0) || y.shape(1) != y_err.shape(1) || + y.shape(2) != y_err.shape(2)) + throw std::runtime_error( + "fit_3d: y and y_err must have identical shape."); + if (!err_out_matches) + throw std::runtime_error( + "err_out must have shape [rows, cols, npar]."); + } else if (err_out.size() > 0 && !err_out_matches) { + throw std::runtime_error("err_out must have shape [rows, cols, npar]."); + } +} + +} // namespace aare::detail diff --git a/src/fit/FitMinuit2.cpp b/src/fit/FitMinuit2.cpp new file mode 100644 index 00000000..326c4b11 --- /dev/null +++ b/src/fit/FitMinuit2.cpp @@ -0,0 +1,168 @@ +// SPDX-License-Identifier: MPL-2.0 +// Minuit2 backends of the pixel fit: Migrad (MnHesse for the errors) and +// Fumili. This is the only translation unit that includes Minuit2 headers. +#include "FitMinuit2.hpp" +#include "Chi2.hpp" +#include "Minuit2/FunctionMinimum.h" +#include "Minuit2/MnFumiliMinimize.h" +#include "Minuit2/MnHesse.h" +#include "Minuit2/MnMigrad.h" +#include "Minuit2/MnStrategy.h" +#include "Minuit2/MnUserParameterState.h" +#include "Minuit2/MnUserParameters.h" +#include "MinuitSteps.hpp" +#include "aare/Models.hpp" +#include +#include +#include +#include +#include + +namespace aare::detail { + +namespace { + +// Translate the FitModel state into Minuit2 user parameters for one pixel. +template +ROOT::Minuit2::MnUserParameters +user_parameters(const FitModel &model, NDView x, + NDView y, + const std::array &start) { + constexpr std::size_t npar = Model::npar; + + double x_range = 0.0; + double y_range = 0.0; + double slope_scale = 0.0; + minuit2::compute_ranges(x, y, x_range, y_range, slope_scale); + std::array steps{}; + minuit2::Steps::compute(start, x_range, y_range, slope_scale, steps); + + ROOT::Minuit2::MnUserParameters upar; + for (std::size_t i = 0; i < npar; ++i) { + const auto idx = static_cast(i); + upar.Add(Model::param_info[i].name, start[i], steps[i]); + const double lo = model.lower_limit(idx); + const double hi = model.upper_limit(idx); + if (std::isfinite(lo) && std::isfinite(hi)) { + upar.SetLimits(idx, lo, hi); + } else if (std::isfinite(lo) || std::isfinite(hi)) { + // Minuit2's transformation for one-sided limits converges markedly + // slower than the two-sided one, so an open side gets a wide bound. + // It is placed relative to the start value, which then always + // lies inside the range however far it is from the finite limit. + const double span = std::max(1e6, 1e3 * std::abs(start[i])); + if (std::isfinite(lo)) + upar.SetLimits(idx, lo, std::max(start[i], lo) + span); + else + upar.SetLimits(idx, std::min(start[i], hi) - span, hi); + } + if (model.is_user_fixed(idx)) + upar.Fix(idx); + } + return upar; +} + +// Minuit2 approaches an active limit asymptotically through its parameter +// transformation, so "on a limit" is decided with a small relative margin. +template +bool on_limit(const FitModel &model, unsigned int idx, double value) { + const double margin = 1e-6 * std::max(1.0, std::abs(value)); + return std::abs(value - model.lower_limit(idx)) <= margin || + std::abs(value - model.upper_limit(idx)) <= margin; +} + +// Copy the minimum into the flat result. Fixed parameters and parameters on +// a limit report an error of 0. +template +void unpack_minimum(const FitModel &model, + const ROOT::Minuit2::FunctionMinimum &min, double *par_out, + double *err_out, double &chi2) { + constexpr std::size_t npar = Model::npar; + const auto &state = min.UserState(); + const std::vector values = state.Params(); + for (std::size_t k = 0; k < npar; ++k) + par_out[k] = values[k]; + + if (err_out) { + const std::vector errors = state.Errors(); + for (std::size_t k = 0; k < npar; ++k) { + const auto idx = static_cast(k); + const bool no_error = + model.is_user_fixed(idx) || on_limit(model, idx, values[k]); + err_out[k] = no_error ? 0.0 : errors[k]; + } + } + chi2 = min.Fval(); +} + +} // namespace + +template +bool fit_pixel_minuit2(const FitModel &model, NDView x, + NDView y, NDView y_err, + const std::array &start, + double *par_out, double *err_out, double &chi2) { + constexpr std::size_t npar = Model::npar; + std::fill(par_out, par_out + npar, 0.0); + if (err_out) + std::fill(err_out, err_out + npar, 0.0); + chi2 = 0.0; + + if (!Model::is_valid(std::vector(start.begin(), start.end()))) + return false; + + const auto upar = user_parameters(model, x, y, start); + const ROOT::Minuit2::MnStrategy strategy(model.strategy()); + + if (model.minimizer() == Minimizer::Fumili) { + // Fumili takes the gradient and the Gauss-Newton Hessian from the FCN, + // so the covariance of the minimum is available without MnHesse. + func::Chi2Model1DFumili fcn(x, y, y_err); + ROOT::Minuit2::MnFumiliMinimize fumili( + fcn, ROOT::Minuit2::MnUserParameterState(upar), strategy); + const ROOT::Minuit2::FunctionMinimum min = + fumili(model.max_calls(), model.tolerance()); + if (min.IsValid()) { + unpack_minimum(model, min, par_out, err_out, chi2); + return true; + } + // Minuit2's Fumili cannot invert its linearised Hessian once a + // two-sided limit becomes active. Refit such pixels with Migrad. + } + + func::Chi2Model1DGrad fcn(x, y, y_err); + ROOT::Minuit2::MnMigrad migrad(fcn, upar, strategy); + ROOT::Minuit2::FunctionMinimum min = + migrad(model.max_calls(), model.tolerance()); + if (!min.IsValid()) + return false; + + // MnHesse cannot handle a fit in which every parameter is fixed. + if (err_out && upar.VariableParameters() > 0) { + ROOT::Minuit2::MnHesse hesse; + hesse(fcn, min); + } + unpack_minimum(model, min, par_out, err_out, chi2); + return true; +} + +// NOLINTBEGIN +#define AARE_INSTANTIATE_MINUIT2_FIT(Model) \ + template bool fit_pixel_minuit2( \ + const FitModel &, NDView, NDView, \ + NDView, const std::array &, double *, \ + double *, double &); + +AARE_INSTANTIATE_MINUIT2_FIT(model::Gaussian) +AARE_INSTANTIATE_MINUIT2_FIT(model::GaussianErfcPlateau) +AARE_INSTANTIATE_MINUIT2_FIT(model::GaussianChargeSharing) +AARE_INSTANTIATE_MINUIT2_FIT(model::GaussianChargeSharingKb) +AARE_INSTANTIATE_MINUIT2_FIT(model::Pol1) +AARE_INSTANTIATE_MINUIT2_FIT(model::Pol2) +AARE_INSTANTIATE_MINUIT2_FIT(model::RisingScurve) +AARE_INSTANTIATE_MINUIT2_FIT(model::FallingScurve) + +#undef AARE_INSTANTIATE_MINUIT2_FIT +// NOLINTEND + +} // namespace aare::detail diff --git a/src/fit/FitMinuit2.hpp b/src/fit/FitMinuit2.hpp new file mode 100644 index 00000000..c0240be8 --- /dev/null +++ b/src/fit/FitMinuit2.hpp @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "aare/FitModel.hpp" +#include "aare/NDView.hpp" +#include + +// Minuit2 backends (Migrad and Fumili) of the pixel fit. FitMinuit2.cpp is +// the only translation unit that includes Minuit2 headers; it defines and +// explicitly instantiates the function below for every supported model. + +namespace aare::detail { + +/** + * @brief Fit one pixel with the Minuit2 minimizer selected by the model. + * + * @param start Starting values, see start_values() in FitHelpers.hpp. + * @param par_out Receives npar fitted values; zeros on failure. + * @param err_out Receives npar errors when non-null; zeros on failure. + * Fixed parameters and parameters on a limit report 0. + * @param chi2 Receives the chi-squared at the minimum; 0 on failure. + * @return true when Minuit2 reports a valid minimum. + */ +template +bool fit_pixel_minuit2(const FitModel &model, NDView x, + NDView y, NDView y_err, + const std::array &start, + double *par_out, double *err_out, double &chi2); + +} // namespace aare::detail diff --git a/src/fit/LevenbergMarquardt.hpp b/src/fit/LevenbergMarquardt.hpp new file mode 100644 index 00000000..8a6a7213 --- /dev/null +++ b/src/fit/LevenbergMarquardt.hpp @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "DampedGaussNewton.hpp" +#include "aare/FitModel.hpp" +#include "aare/NDView.hpp" + +#include +#include +#include +#include +#include + +namespace aare::detail { + +/** + * @brief Levenberg-Marquardt least-squares minimiser for one FitModel. + * + * Minimises chi2 = sum_i ((y_i - f(x_i; p)) / s_i)^2 using the analytic + * gradient of the model (s_i = 1 for unweighted fits; points with s_i == 0 + * are ignored). Keep one instance per thread: the buffers are reused between + * pixels so that after the first fit no memory is allocated. + * + * Every evaluation computes the residuals r = (y - f) / s and the Jacobian + * J = dr/dp in one pass over the data and reduces them to the normal + * equations g = J^T r and J^T J, which DampedGaussNewton iterates on: it + * handles fixed parameters, limits, damping and the convergence test, see + * there. Rejected trials are rare with its damping rule, and a rejected one + * only wastes a single pass. Trial points at which the model loses its + * sensitivity to a free parameter are rejected. As in Minuit2, max_calls = 0 + * selects the default budget 200 + 100 * npar + 5 * npar^2. + * + * Errors are Gauss-Newton estimates, sqrt(diag((J^T J)^-1)), over the free + * parameters that do not sit on a limit. Fixed and limit-bound parameters + * report 0. + */ +template class LevenbergMarquardt { + public: + static constexpr int npar = static_cast(Model::npar); + static_assert(Model::npar >= 1 && Model::npar <= 16, + "LevenbergMarquardt keeps npar x npar matrices on the stack"); + + using Result = MinimizerResult; + using Driver = DampedGaussNewton; + using Normal = typename Driver::Normal; + + /** + * @brief Fit one pixel. + * + * @param model Limits, fixed flags and minimiser settings. + * @param x Scan points. + * @param y Measured values. + * @param s Per-point uncertainties, empty view for an unweighted fit. + * @param start Starting values, see start_values() in FitHelpers.hpp. + * @param par_out Receives npar fitted values; zeros on failure. + * @param err_out Receives npar errors when non-null; zeros on failure. + */ + Result fit(const FitModel &model, NDView x, + NDView y, NDView s, + const std::array &start, double *par_out, + double *err_out) { + x_ = x; + y_ = y; + s_ = s; + weighted_ = s.size() > 0; + const auto n = static_cast(x.size()); + Vec lower{}; + Vec upper{}; + std::array fixed{}; + for (int k = 0; k < npar; ++k) { + const auto idx = static_cast(k); + lower[k] = model.lower_limit(idx); + upper[k] = model.upper_limit(idx); + fixed[k] = model.is_user_fixed(idx); + } + r_.resize(n); + J_.resize(n * static_cast(npar)); + if (weighted_) { + // 1 / s_i once per pixel rather than once per evaluation. + w_.resize(n); + for (std::size_t i = 0; i < n; ++i) + w_[i] = s_[static_cast(i)] != 0.0 + ? 1.0 / s_[static_cast(i)] + : 0.0; + } + const int max_calls = model.max_calls() > 0 + ? static_cast(model.max_calls()) + : 200 + 100 * npar + 5 * npar * npar; + + auto eval = [this](const Vec &p, Normal &out) { + return evaluate(p, out); + }; + typename Driver::Options opt; + opt.tolerance = model.tolerance(); + opt.max_calls = max_calls; + // Start cautiously: the estimated start values of the linear + // parameters can be far off, and the first steps are then wild. + opt.damping0 = 0.1; + // A width driven onto its lower limit turns a model into a step + // function whose chi2 no longer responds to the width and jumps + // with the position; the iteration then stalls there. Such trial + // points are rejected, as in VariableProjection. + opt.reject_degenerate = true; + const Result res = driver_.fit(eval, start, lower, upper, fixed, opt); + if (!res.valid) { + std::fill(par_out, par_out + npar, 0.0); + if (err_out) + std::fill(err_out, err_out + npar, 0.0); + return res; + } + std::copy(driver_.point().begin(), driver_.point().end(), par_out); + if (err_out) + driver_.errors(err_out); + return res; + } + + private: + double weight(ssize_t i) const { + return weighted_ ? w_[static_cast(i)] : 1.0; + } + + // Residuals r = (y - f) / s and the Jacobian J = dr/dp at the point p, + // reduced to F = chi2 / 2, g = J^T r and the upper triangle of J^T J. + // The sums run over all parameters with fixed-size loops, so the + // compiler can unroll them and keep the accumulators in registers. + // Returns false when the model rejects the parameters. + bool evaluate(const Vec &p, Normal &out) { + pvec_.assign(p.begin(), p.end()); + if (!Model::is_valid(pvec_)) + return false; + const ssize_t n = x_.size(); + double F = 0.0; + std::array g{}; + double f = 0.0; + for (ssize_t i = 0; i < n; ++i) { + const double w = weight(i); + Model::eval_and_grad(x_[i], pvec_, f, g); + r_[i] = (y_[i] - f) * w; + F += 0.5 * r_[i] * r_[i]; + double *Ji = &J_[static_cast(i) * npar]; + for (int k = 0; k < npar; ++k) + Ji[k] = -g[k] * w; + } + Vec G{}; + Mat H{}; + for (ssize_t i = 0; i < n; ++i) { + const double *Ji = &J_[static_cast(i) * npar]; + const double ri = r_[i]; + for (int a = 0; a < npar; ++a) { + G[a] += Ji[a] * ri; + for (int b = a; b < npar; ++b) + H[a][b] += Ji[a] * Ji[b]; + } + } + out.F = F; + for (int a = 0; a < npar; ++a) { + out.g[a] = G[a]; + for (int b = a; b < npar; ++b) + out.H[a][b] = H[a][b]; + } + return true; + } + + NDView x_, y_, s_; + bool weighted_ = false; + Driver driver_; + std::vector r_; + std::vector J_; + std::vector w_; // 1 / s_i for weighted fits + std::vector pvec_; +}; + +} // namespace aare::detail diff --git a/src/fit/MinuitSteps.hpp b/src/fit/MinuitSteps.hpp new file mode 100644 index 00000000..d5aa8bbe --- /dev/null +++ b/src/fit/MinuitSteps.hpp @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +// Data-driven Minuit step sizes for the optional Minuit2 backend. These +// estimators used to live in aare/Models.hpp; the built-in solver does not +// need them. + +#include "aare/Models.hpp" +#include "aare/NDView.hpp" +#include +#include +#include + +namespace aare::minuit2 { + +/** + * @brief Compute data-range statistics used by the step-size estimators. + * + * Model-independent, called once per pixel. + */ +inline void compute_ranges(NDView x, NDView y, + double &x_range, double &y_range, + double &slope_scale) { + const auto [x_min, x_max] = std::minmax_element(x.begin(), x.end()); + const auto [y_min, y_max] = std::minmax_element(y.begin(), y.end()); + + x_range = std::max(*x_max - *x_min, 1e-9); + y_range = std::max(*y_max - *y_min, 1e-9); + + slope_scale = std::max(y_range / x_range, 1e-9); +} + +/** @brief Per-model Minuit step sizes, specialised for every fit model. */ +template struct Steps; + +template <> struct Steps { + static void compute(const std::array &start, + [[maybe_unused]] double x_range, double y_range, + double slope_scale, std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.1 * slope_scale; + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double slope_scale, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.1 * slope_scale; + steps[2] = 0.1 * slope_scale / std::max(x_range, 1e-12); + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double /*slope_scale*/, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.05 * x_range; + steps[2] = 0.05 * x_range; + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double /*slope_scale*/, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = std::max(0.1 * std::abs(start[1]), 0.1 * y_range); + steps[2] = 0.05 * x_range; + steps[3] = 0.05 * x_range; + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double slope_scale, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.1 * slope_scale; + steps[2] = 0.05 * x_range; + steps[3] = 0.05 * x_range; + steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); + steps[5] = std::max(0.1 * std::abs(start[5]), 0.01); + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double slope_scale, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.1 * slope_scale; + steps[2] = 0.05 * x_range; + steps[3] = 0.05 * x_range; + steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); + steps[5] = std::max(0.1 * std::abs(start[5]), 0.01); + steps[6] = std::max(0.01 * std::abs(start[6]), 1e-3); + steps[7] = std::max(0.1 * std::abs(start[7]), 0.01); + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double slope_scale, + std::array &steps) { + steps[0] = std::max(0.1 * std::abs(start[0]), 0.1 * y_range); + steps[1] = 0.1 * slope_scale; + steps[2] = 0.05 * x_range; + steps[3] = 0.05 * x_range; + steps[4] = std::max(0.1 * std::abs(start[4]), 0.1 * y_range); + steps[5] = 0.1 * slope_scale; + } +}; + +template <> struct Steps { + static void compute(const std::array &start, double x_range, + double y_range, double slope_scale, + std::array &steps) { + Steps::compute(start, x_range, y_range, + slope_scale, steps); + } +}; + +} // namespace aare::minuit2 diff --git a/src/fit/VariableProjection.hpp b/src/fit/VariableProjection.hpp new file mode 100644 index 00000000..0ea0cdbc --- /dev/null +++ b/src/fit/VariableProjection.hpp @@ -0,0 +1,511 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "DampedGaussNewton.hpp" +#include "aare/FitModel.hpp" +#include "aare/Models.hpp" +#include "aare/NDView.hpp" + +#include +#include +#include +#include +#include + +namespace aare::detail { + +/** + * @brief Variable projection least-squares minimiser for a separable + * FitModel (Golub and Pereyra, 1973). + * + * The model declares the parameters it is linear in, see + * model::is_separable. For a given point of the nonlinear parameters the + * linear ones are solved exactly from their normal equations, and the + * damped Gauss-Newton iteration of DampedGaussNewton runs over the + * nonlinear parameters only, on the projected residual + * r(alpha) = y - Phi(alpha) beta(alpha). Its Jacobian uses Kaufman's + * approximation, the derivatives of the model at fixed beta projected out + * of the span of the basis functions, and the reduced Gauss-Newton Hessian + * is the Schur complement of the full one. Compared with + * LevenbergMarquardt the iteration needs fewer evaluations, cannot be misled + * by poor start values of the linear parameters, and each evaluation + * accumulates far fewer sums. The driver rejects trial points at which the + * model loses its sensitivity to a nonlinear parameter and polishes the + * converged point with one undamped step. + * + * One evaluation is a pass over the data that writes the basis functions + * and their derivatives as columns (the model's vectorisable basis_columns + * when it has one, otherwise basis_and_grad per point), a reduction to the + * normal equations of the linear parameters, the small solve, and a second + * reduction over the columns for the projected residual. + * + * Keep one instance per thread: the buffers are reused between pixels so + * that after the first fit no memory is allocated. + * + * - Fixed nonlinear parameters are left out of the iteration. A fixed + * linear parameter keeps its value and moves to the data side of the + * linear solve. + * - Limits on nonlinear parameters are handled by the driver. A limit on a + * linear parameter cannot be enforced by the linear solve: when the + * solution violates one, the fit reports failure so that the caller can + * fall back to LevenbergMarquardt. + * - Start values of free linear parameters are ignored. + * - max_calls counts evaluations of the projected problem, as in + * LevenbergMarquardt; 0 selects 200 + 100 * npar + 5 * npar^2. + * - Errors are the Gauss-Newton estimates of the full model at the + * solution, sqrt(diag((J^T J)^-1)) over the free parameters that do not + * sit on a limit, the same as LevenbergMarquardt reports. + */ +template class VariableProjection { + public: + static_assert(model::is_separable::value, + "VariableProjection needs a model with linear_par and " + "basis_and_grad"); + using Traits = model::separable_traits; + static constexpr int npar = static_cast(Model::npar); + static constexpr int nlin = static_cast(Traits::nlin); + static constexpr int nnl = static_cast(Traits::nnl); + static_assert(Model::npar >= 1 && Model::npar <= 16, + "VariableProjection keeps npar x npar matrices on the stack"); + static_assert(nlin >= 1, "a separable model has at least one linear " + "parameter"); + + using Result = MinimizerResult; + // The driver iterates over the nonlinear parameters. A fully linear + // model has none and is solved without iterating. + static constexpr int ndrv = nnl > 0 ? nnl : 1; + static constexpr int ncol = nnl * nlin > 0 ? nnl *nlin : 1; + using Driver = DampedGaussNewton; + using Normal = typename Driver::Normal; + + /** + * @brief Fit one pixel, see LevenbergMarquardt::fit for the arguments. + */ + Result fit(const FitModel &model, NDView x, + NDView y, NDView s, + const std::array &start, double *par_out, + double *err_out) { + x_ = x; + y_ = y; + const auto n = static_cast(x.size()); + phi_.resize(n * nlin); + dphi_.resize(n * nlin * nnl); + weighted_ = s.size() > 0; + if (weighted_) { + // 1 / s_i^2 once per pixel; points with s_i == 0 are ignored. + w2_.resize(n); + for (std::size_t i = 0; i < n; ++i) { + const double si = s[static_cast(i)]; + w2_[i] = si != 0.0 ? 1.0 / (si * si) : 0.0; + } + } + pvec_.assign(start.begin(), start.end()); + have_last_ = false; + have_ref_ = false; + nfree_lin_ = 0; + for (int j = 0; j < nlin; ++j) { + const auto k = static_cast(Traits::linear_index(j)); + lin_fixed_[j] = model.is_user_fixed(k); + if (!lin_fixed_[j]) + free_lin_[nfree_lin_++] = j; + } + Vec alpha{}; + Vec lower{}; + Vec upper{}; + std::array fixed{}; + for (int k = 0; k < nnl; ++k) { + const auto idx = + static_cast(Traits::nonlinear_par[k]); + alpha[k] = start[idx]; + lower[k] = model.lower_limit(idx); + upper[k] = model.upper_limit(idx); + fixed[k] = model.is_user_fixed(idx); + } + const int max_calls = model.max_calls() > 0 + ? static_cast(model.max_calls()) + : 200 + 100 * npar + 5 * npar * npar; + + Result res; + auto fail = [&]() { + res.valid = false; + res.chi2 = 0.0; + std::fill(par_out, par_out + npar, 0.0); + if (err_out) + std::fill(err_out, err_out + npar, 0.0); + return res; + }; + + if constexpr (nnl == 0) { + Normal nrm; + if (!evaluate(alpha, nrm)) + return fail(); + res.calls = 1; + res.valid = true; + res.chi2 = 2.0 * nrm.F; + for (int j = 0; j < nlin; ++j) + pvec_[Traits::linear_index(j)] = + nrm.aux[static_cast(j)]; + } else { + auto eval = [this](const Vec &a, Normal &out) { + return evaluate(a, out); + }; + typename Driver::Options opt; + opt.tolerance = model.tolerance(); + opt.max_calls = max_calls; + // The same cautious start as LevenbergMarquardt: a large first + // step can still drive a width to zero. Rejecting degenerate + // trial points guards against that, and the polish step makes + // up for the few iterations, which stop while the damping still + // shortens the steps. + opt.damping0 = 0.1; + opt.reject_degenerate = true; + opt.polish = true; + res = driver_.fit(eval, alpha, lower, upper, fixed, opt); + if (!res.valid) + return fail(); + for (int k = 0; k < nnl; ++k) + pvec_[Traits::nonlinear_par[k]] = driver_.point()[k]; + for (int j = 0; j < nlin; ++j) + pvec_[Traits::linear_index(j)] = + driver_.normal().aux[static_cast(j)]; + } + + // The linear solve knows nothing about limits. A solution beyond a + // limit fails; one within rounding of a limit is moved onto it, so + // that it counts as on the limit whichever way the rounding went. + for (int j = 0; j < nlin; ++j) { + if (lin_fixed_[j]) + continue; + const auto k = static_cast(Traits::linear_index(j)); + const double lo = model.lower_limit(k); + const double hi = model.upper_limit(k); + const double tol = 1e-12 * std::max(1.0, std::abs(pvec_[k])); + if (pvec_[k] < lo - tol || pvec_[k] > hi + tol) + return fail(); + if (std::abs(pvec_[k] - lo) <= tol) + pvec_[k] = lo; + else if (std::abs(pvec_[k] - hi) <= tol) + pvec_[k] = hi; + } + std::copy(pvec_.begin(), pvec_.end(), par_out); + if (err_out) + errors(model, err_out); + return res; + } + + private: + // F = chi2 / 2, gradient and Kaufman Hessian of the projected residual + // at the nonlinear point alpha, and the linear solution in out.aux. + // Returns false when the model rejects the point, or when the basis + // functions are numerically linearly dependent there or one of them has + // vanished on the scan points, see reduce(). + bool evaluate(const Vec &alpha, Normal &out) { + for (int k = 0; k < nnl; ++k) + pvec_[Traits::nonlinear_par[k]] = alpha[k]; + if (!Model::is_valid(pvec_)) + return false; + const ssize_t n = x_.size(); + + // Basis functions and their derivatives, one column per function: + // phi_[j * n + i] and dphi_[(k * nlin + j) * n + i]. + if constexpr (model::has_basis_columns::value) { + Model::basis_columns(x_.data(), n, pvec_, phi_.data(), + dphi_.data()); + } else { + typename Traits::Basis phi{}; + typename Traits::BasisGrad dphi{}; + double *pc = phi_.data(); + double *dc = dphi_.data(); + const auto un = static_cast(n); + for (ssize_t i = 0; i < n; ++i) { + Model::basis_and_grad(x_[i], pvec_, phi, dphi); + const auto ui = static_cast(i); + for (int j = 0; j < nlin; ++j) + pc[static_cast(j) * un + ui] = + phi[static_cast(j)]; + for (int k = 0; k < nnl; ++k) + for (int j = 0; j < nlin; ++j) + dc[static_cast(k * nlin + j) * un + ui] = + dphi[static_cast(k)] + [static_cast(j)]; + } + } + return weighted_ ? reduce(alpha, out) : reduce(alpha, out); + } + + // The reductions over the columns: normal equations and solve of the + // linear parameters, then the projected residual. All loops over the + // linear and nonlinear parameters have compile-time bounds; the free + // linear block is packed out of the full sums afterwards. + template bool reduce(const Vec &alpha, Normal &out) { + const ssize_t n = x_.size(); + const auto un = static_cast(n); + std::array pc; + for (int j = 0; j < nlin; ++j) + pc[j] = phi_.data() + static_cast(j) * un; + std::array dc; + for (int c = 0; c < nnl * nlin; ++c) + dc[c] = dphi_.data() + static_cast(c) * un; + const double *yd = y_.data(); + const double *w2 = Weighted ? w2_.data() : nullptr; + + // Pass 1: normal equations M = Phi^T W Phi, t = Phi^T W y of all + // linear parameters. + Mat M{}; + Vec t{}; + for (ssize_t i = 0; i < n; ++i) { + const double wi = Weighted ? w2[i] : 1.0; + const double yi = yd[i]; + Vec f; + for (int j = 0; j < nlin; ++j) + f[j] = pc[j][i]; + for (int a = 0; a < nlin; ++a) { + const double fa = f[a] * wi; + t[a] += fa * yi; + for (int b = a; b < nlin; ++b) + M[a][b] += fa * f[b]; + } + } + + // Free linear parameters: their block of M, with the fixed ones + // moved to the right-hand side, v_a = t_a - sum_fixed c_j M_aj. + // The lower triangle of M is filled in so that the run-time indices + // below need no branch: gcc's range analysis derives an impossible + // subscript from the branch not taken and warns about it. The loops + // run over the compile-time bound and write zeros past nfree_lin_ + // for the same reason, gcc otherwise reports v and qf as possibly + // uninitialised when they are passed to cholesky_substitute. Both + // are false positives; neither change affects the results. + for (int a = 1; a < nlin; ++a) + for (int b = 0; b < a; ++b) + M[a][b] = M[b][a]; + Mat A{}; + Vec v{}; + for (int a = 0; a < nlin; ++a) { + if (a >= nfree_lin_) { + v[a] = 0.0; + continue; + } + const int ja = free_lin_[a]; + double va = t[ja]; + for (int j = 0; j < nlin; ++j) + if (lin_fixed_[j]) + va -= pvec_[Traits::linear_index(j)] * M[ja][j]; + v[a] = va; + for (int b = 0; b < nfree_lin_; ++b) + A[a][b] = M[ja][free_lin_[b]]; + } + // A free basis function that has all but vanished on the scan points + // (a Gaussian narrower than the point spacing, centred between two + // points) leaves its coefficient undetermined, and the solve returns + // an absurd one that fits a single point. Its squared norm is compared + // with that at the pixel's first evaluation, the start point. + if (!have_ref_) { + for (int j = 0; j < nlin; ++j) + ref_norm2_[j] = M[j][j]; + have_ref_ = true; + } + for (int a = 0; a < nfree_lin_; ++a) { + const int j = free_lin_[a]; + if (!(M[j][j] >= vanished_basis * ref_norm2_[j])) + return false; + } + Mat L{}; + if (!cholesky_factor(nfree_lin_, A, L)) + return false; + // Nearly collinear basis functions: a pivot of the factorisation far + // below its diagonal element means that the linear solution loses + // about log10(diagonal / pivot) digits, beyond which it is noise. + bool refine = false; + for (int a = 0; a < nfree_lin_; ++a) { + if (!(L[a][a] * L[a][a] >= collinear_basis * A[a][a])) + return false; + refine = refine || L[a][a] * L[a][a] < refine_basis * A[a][a]; + } + Vec beta_free{}; + cholesky_substitute(nfree_lin_, L, v, beta_free); + for (int a = 0; a < nfree_lin_; ++a) + pvec_[Traits::linear_index(free_lin_[a])] = beta_free[a]; + // Short of that, the digits lost to the normal equations are + // recovered by one step of iterative refinement on the data (the + // corrected semi-normal equations): the correction solves the same + // equations for Phi^T W r of the residual of the first solution. A + // polynomial fitted far from x = 0 needs it; x in [20000, 20500] + // otherwise costs a Pol2 six digits. + if (refine) { + Vec beta; + for (int j = 0; j < nlin; ++j) + beta[j] = pvec_[Traits::linear_index(j)]; + Vec s{}; + for (ssize_t i = 0; i < n; ++i) { + double r = yd[i]; + for (int j = 0; j < nlin; ++j) + r -= beta[j] * pc[j][i]; + const double rw = Weighted ? r * w2[i] : r; + for (int j = 0; j < nlin; ++j) + s[j] += pc[j][i] * rw; + } + Vec sf{}; + for (int a = 0; a < nlin; ++a) + sf[a] = a < nfree_lin_ ? s[free_lin_[a]] : 0.0; + Vec delta{}; + cholesky_substitute(nfree_lin_, L, sf, delta); + for (int a = 0; a < nfree_lin_; ++a) + pvec_[Traits::linear_index(free_lin_[a])] += delta[a]; + } + Vec beta; + for (int j = 0; j < nlin; ++j) { + beta[j] = pvec_[Traits::linear_index(j)]; + out.aux[static_cast(j)] = beta[j]; + } + + // Pass 2: residual r = y - Phi beta and u_k = df/dalpha_k at fixed + // beta; F, b = U^T W r, N = U^T W U and Q = Phi^T W U. + double F = 0.0; + Vec b{}; + Mat N{}; + std::array, ndrv> Q{}; + for (ssize_t i = 0; i < n; ++i) { + const double wi = Weighted ? w2[i] : 1.0; + Vec f; + for (int j = 0; j < nlin; ++j) + f[j] = pc[j][i]; + double r = yd[i]; + for (int j = 0; j < nlin; ++j) + r -= beta[j] * f[j]; + Vec u{}; + for (int k = 0; k < nnl; ++k) + for (int j = 0; j < nlin; ++j) + u[k] += beta[j] * dc[k * nlin + j][i]; + const double rw = r * wi; + F += 0.5 * r * rw; + for (int k = 0; k < nnl; ++k) { + const double uw = u[k] * wi; + b[k] += uw * r; + for (int l = k; l < nnl; ++l) + N[k][l] += uw * u[l]; + for (int j = 0; j < nlin; ++j) + Q[k][j] += uw * f[j]; + } + } + + // The full Gauss-Newton Hessian of the model at this point is + // [[M, Q^T], [Q, N]]; kept for the errors of the final point. + for (int k = 0; k < nnl; ++k) + last_alpha_[k] = alpha[k]; + for (int a = 0; a < nlin; ++a) + for (int c = a; c < nlin; ++c) + last_M_[a][c] = M[a][c]; + for (int k = 0; k < nnl; ++k) { + for (int j = 0; j < nlin; ++j) + last_Q_[k][j] = Q[k][j]; + for (int l = k; l < nnl; ++l) + last_N_[k][l] = N[k][l]; + } + have_last_ = true; + + // Kaufman's Jacobian: U projected out of the span of the free basis, + // H = N - Q_free^T A^-1 Q_free. + std::array, ndrv> qf{}; + std::array, ndrv> z{}; + for (int k = 0; k < nnl; ++k) { + for (int a = 0; a < nlin; ++a) + qf[k][a] = a < nfree_lin_ ? Q[k][free_lin_[a]] : 0.0; + cholesky_substitute(nfree_lin_, L, qf[k], z[k]); + } + out.F = F; + for (int k = 0; k < nnl; ++k) { + out.g[k] = -b[k]; + for (int l = k; l < nnl; ++l) { + double h = N[k][l]; + for (int a = 0; a < nfree_lin_; ++a) + h -= qf[k][a] * z[l][a]; + out.H[k][l] = h; + } + } + return true; + } + + // Gauss-Newton errors from the Jacobian of the full model at the + // solution, the estimate LevenbergMarquardt reports as well. The + // Jacobian's normal matrix is assembled from the sums of the last + // evaluation when that was the final point, which it is unless the last + // trial was rejected; otherwise one pass over the data computes it. + void errors(const FitModel &model, double *err_out) { + Mat H{}; + bool at_solution = have_last_; + for (int k = 0; k < nnl; ++k) + at_solution = at_solution && + pvec_[Traits::nonlinear_par[k]] == last_alpha_[k]; + if (at_solution) { + for (int a = 0; a < nlin; ++a) + for (int c = a; c < nlin; ++c) + H[Traits::linear_index(a)][Traits::linear_index(c)] = + last_M_[a][c]; + for (int k = 0; k < nnl; ++k) { + const auto m = static_cast(Traits::nonlinear_par[k]); + for (int j = 0; j < nlin; ++j) { + const auto i = static_cast(Traits::linear_index(j)); + H[std::min(i, m)][std::max(i, m)] = last_Q_[k][j]; + } + for (int l = k; l < nnl; ++l) + H[m][Traits::nonlinear_par[l]] = last_N_[k][l]; + } + } else { + const ssize_t n = x_.size(); + std::array g{}; + double f = 0.0; + for (ssize_t i = 0; i < n; ++i) { + Model::eval_and_grad(x_[i], pvec_, f, g); + const double wi = + weighted_ ? w2_[static_cast(i)] : 1.0; + for (int a = 0; a < npar; ++a) + for (int c = a; c < npar; ++c) + H[a][c] += g[a] * g[c] * wi; + } + } + // Fixed parameters and parameters that end on a limit have no error. + // The driver knows the nonlinear ones; the linear solve can land a + // linear parameter exactly on a limit without violating it. + std::array skip; + for (int k = 0; k < npar; ++k) { + const auto idx = static_cast(k); + const double v = pvec_[static_cast(k)]; + skip[k] = model.is_user_fixed(idx) || v <= model.lower_limit(idx) || + v >= model.upper_limit(idx); + } + if constexpr (nnl > 0) { + for (int k = 0; k < nnl; ++k) + skip[Traits::nonlinear_par[k]] = + skip[Traits::nonlinear_par[k]] || driver_.skipped(k); + } + gauss_newton_errors(H, skip, err_out); + } + + NDView x_, y_; + Driver driver_; + bool weighted_ = false; + // Sums of the last evaluation, see errors(). + bool have_last_ = false; + Vec last_alpha_{}; + Mat last_M_{}; + std::array, ndrv> last_Q_{}; + Mat last_N_{}; + // Squared norms of the basis functions at the start point, see reduce(). + bool have_ref_ = false; + Vec ref_norm2_{}; + static constexpr double vanished_basis = 1e-8; + static constexpr double collinear_basis = 1e-10; + // Pivot ratio below which the linear solution is refined, about four + // lost digits. + static constexpr double refine_basis = 1e-4; + int nfree_lin_ = 0; + std::array free_lin_{}; + std::array lin_fixed_{}; + std::vector phi_; // nlin columns of n basis values + std::vector dphi_; // nnl * nlin columns of their derivatives + std::vector w2_; // 1 / s_i^2 for weighted fits + std::vector pvec_; +}; + +} // namespace aare::detail diff --git a/src/hist/PedestalTrackingPixelHistogram.cpp b/src/hist/PedestalTrackingPixelHistogram.cpp index 932b480e..61b416ef 100644 --- a/src/hist/PedestalTrackingPixelHistogram.cpp +++ b/src/hist/PedestalTrackingPixelHistogram.cpp @@ -1,11 +1,17 @@ +// SPDX-License-Identifier: MPL-2.0 #include "aare/hist/PedestalTrackingPixelHistogram.hpp" #include "aare/File.hpp" +#include "aare/MultiThreadedFileReader.hpp" #include +#include #include #include #include +#include +#include #include +#include #include #include @@ -207,20 +213,10 @@ void PedestalTrackingPixelHistogram::worker_loop(int thread_id) { switch (kind) { case WorkKind::PushPedestal: { - // Accumulate raw frame values into this thread's pedestal - // shard. Uses the pixel-level push_no_update which only - // touches m_sum/m_sum2/m_cur_samples (no m_mean writes). - for (int local_row = 0; local_row < local_rows; ++local_row) { - const auto row = static_cast(first_row + local_row); - for (ssize_t col = 0; col < image->shape(1); ++col) { - if (has_mask && mask_.value()(row, col)) { - continue; // Skip masked pixels - } - my_pedestal.template push_no_update( - static_cast(local_row), - static_cast(col), (*image)(row, col)); - } - } + auto frame = image->sub_view(first_row, first_row + local_rows); + my_pedestal.add_init_frame( + frame); // dont bother adding bad pixels in pedestal - not + // needed for calculation break; } case WorkKind::UpdateMean: { @@ -228,7 +224,6 @@ void PedestalTrackingPixelHistogram::worker_loop(int thread_id) { // thread's shard. Also refresh the cached per-pixel std so // FillWithThreshold can read it without recomputing on the // hot path. - my_pedestal.update_mean(); auto &my_std = partial_std_[thread_id]; for (int local_row = 0; local_row < local_rows; ++local_row) { for (int col = 0; col < cols_; ++col) { @@ -248,65 +243,70 @@ void PedestalTrackingPixelHistogram::worker_loop(int thread_id) { // tracking gate entirely. The [xmin, xmax) histogram gate // lives inside PixelHistogramImpl::fill. const auto n_sigma = n_sigma_.load(std::memory_order_relaxed); - if (n_sigma <= AxisType{0.0}) { - // Fill without pedestal tracking. - for (const auto &frame : *images) { - const auto cols = frame.shape(1); - for (int local_row = 0; local_row < local_rows; - ++local_row) { - const auto row = - static_cast(first_row + local_row); - for (ssize_t col = 0; col < cols; ++col) { - if (has_mask && mask_.value()(row, col)) { - continue; // Skip masked pixels - } - const FrameType raw = frame(row, col); - const AxisType val = - static_cast(raw) - - static_cast(my_pedestal.mean( - static_cast(local_row), - static_cast(col))); - my_hist.fill_unchecked(local_row, - static_cast(col), val); - } - } - } - break; - } else { - // Do pedestal tracking. Duplicated code for clean hot path. + constexpr std::size_t pixel_tile_size = 256; + const auto local_pixels = static_cast(local_rows) * + static_cast(cols_); + const auto global_pixel_begin = + static_cast(first_row) * + static_cast(cols_); + const auto &frames = *images; - auto &my_std = partial_std_[thread_id]; - for (const auto &frame : *images) { - const auto cols = frame.shape(1); - for (int local_row = 0; local_row < local_rows; - ++local_row) { - const auto row = - static_cast(first_row + local_row); - for (ssize_t col = 0; col < cols; ++col) { - if (has_mask && mask_.value()(row, col)) { - continue; // Skip masked pixels + // Compile the shared traversal once for each mode. if constexpr + // removes the tracking gate entirely from the histogram-only hot + // path, while keeping the runtime mode decision outside the + // per-pixel loops. + const auto fill_tiles = [&](auto tracking, + const AxisType *std_data) { + constexpr bool track_pedestal = decltype(tracking)::value; + + // Each worker retains exclusive ownership of its row shard. + // Pixel tiling keeps a bounded part of that shard's + // [pixel x bin] storage hot while input remains contiguous. + for (std::size_t tile_begin = 0; tile_begin < local_pixels; + tile_begin += pixel_tile_size) { + const auto tile_end = + std::min(tile_begin + pixel_tile_size, local_pixels); + // const bool ptr_to_mask = has_mask ? mask_.value().data() + // + global_pixel_begin + tile_begin : nullptr; + for (const auto &frame : frames) { + const auto *input = + frame.data() + global_pixel_begin + tile_begin; + for (std::size_t local_pixel = tile_begin; + local_pixel < tile_end; ++local_pixel, ++input) { + const FrameType raw = *input; + + if (has_mask && mask_.value()[local_pixel]) { + continue; // Skip masked pixels entirely } - const FrameType raw = frame(row, col); + const AxisType val = static_cast(raw) - - static_cast(my_pedestal.mean( - static_cast(local_row), - static_cast(col))); - my_hist.fill_unchecked(local_row, - static_cast(col), val); - const AxisType sigma = my_std(local_row, col); - if (sigma > AxisType{0.0} && - std::abs(static_cast(val)) < - n_sigma * sigma) { - my_pedestal.template push( - static_cast(local_row), - static_cast(col), raw); + my_pedestal.mean_unchecked( + static_cast(local_pixel)); + my_hist.fill_flat_unchecked(local_pixel, val); + + if constexpr (track_pedestal) { + const AxisType sigma = std_data[local_pixel]; + if (sigma > AxisType{0.0} && + std::abs(val) < n_sigma * sigma) { + my_pedestal.push_ema_unchecked( + local_pixel, raw); + } } } } } - break; + }; + + if (n_sigma <= AxisType{0.0}) { + fill_tiles(std::false_type{}, nullptr); + } else { + // Frames stay chronological for every pixel. Accepted raw + // values update that pixel's sums and cached mean immediately; + // pixels are otherwise independent. + fill_tiles(std::true_type{}, partial_std_[thread_id].data()); } + break; } } @@ -440,6 +440,11 @@ void PedestalTrackingPixelHistogram::fill_async(NDArray &&image) { "constructor shape"); } + // ProducerConsumerQueue is SPSC. Serialising this short producer-side + // operation also prevents fill_async from racing fill_from_file's direct + // batch dispatch. + std::lock_guard ingestion_lock(ingestion_mutex_); + // SPSC backpressure: spin with a short sleep until a slot frees up. // The std::move only consumes `image` on the iteration that succeeds // (placement-new inside write() runs only when the slot is free). @@ -512,35 +517,42 @@ PedestalTrackingPixelHistogram::bin_edges() const { } void PedestalTrackingPixelHistogram::fill_from_file( - const std::filesystem::path &fname, ssize_t max_frames, bool verbose) { + const std::filesystem::path &fname, ssize_t max_frames, bool verbose, + std::size_t reader_threads, std::size_t reader_chunk_size) { constexpr std::size_t progress_interval = 66; auto last = std::chrono::steady_clock::now(); - const auto completed_start = - completed_async_fills_.load(std::memory_order_acquire); std::size_t last_reported = 0; - const auto completed_for_this_file = [&]() { - return completed_async_fills_.load(std::memory_order_acquire) - - completed_start; - }; - const auto wait_for_completed = [&](std::size_t target) { - while (completed_for_this_file() < target) { - std::this_thread::sleep_for(async_wait_); - } - }; + if (max_frames < -1) { + throw std::invalid_argument( + "PedestalTrackingPixelHistogram max_frames must be -1 or " + "non-negative"); + } - File f(fname); - // check that row col matches constructor - if (f.rows() != static_cast(rows_) || - f.cols() != static_cast(cols_)) { + // Preserve the old max_frames behaviour: values beyond EOF are clamped + // rather than rejected by MultiThreadedFileReader. + const auto source_total_frames = File(fname).total_frames(); + const auto n_frames = max_frames == -1 + ? source_total_frames + : std::min(static_cast(max_frames), + source_total_frames); + + experimental::MultiThreadedFileReader reader(fname, reader_threads, + reader_chunk_size, n_frames); + if (reader.rows() != static_cast(rows_) || + reader.cols() != static_cast(cols_)) { throw std::invalid_argument("PedestalTrackingPixelHistogram: Frame in " "file {} has shape ({}, {}) does not match " "constructor shape"); } + if (reader.dtype() != Dtype::UINT16 || + reader.bytes_per_frame() != static_cast(rows_) * + static_cast(cols_) * + sizeof(FrameType)) { + throw std::invalid_argument( + "PedestalTrackingPixelHistogram requires uint16 file frames"); + } - const ssize_t total_frames = f.total_frames(); - const ssize_t n_frames = - max_frames == -1 ? total_frames : std::min(max_frames, total_frames); const auto print_progress = [&](std::size_t done) { const auto now = std::chrono::steady_clock::now(); const double dt = std::chrono::duration(now - last).count(); @@ -548,30 +560,89 @@ void PedestalTrackingPixelHistogram::fill_from_file( const double fps = dt > 0.0 ? static_cast(done_in_interval) / dt : 0.0; - fmt::print( - "\rProgress: {}/{} ({:.1f}%) {:.1f} FPS ", done, n_frames, - 100.0 * static_cast(done) / static_cast(n_frames), - fps); + fmt::print("\rProgress: {}/{} ({:.1f}%) {:.1f} FPS ", done, + n_frames, + n_frames == 0 ? 100.0 + : 100.0 * static_cast(done) / + static_cast(n_frames), + fps); std::fflush(stdout); last = now; last_reported = done; }; - for (ssize_t i = 0; i < n_frames; ++i) { - aare::NDArray frame({rows_, cols_}); - f.read_into(reinterpret_cast(frame.data())); - fill_async(std::move(frame)); + using ReadBuffer = NDArray; + const auto read_into_buffer = [&reader](ReadBuffer &buffer) { + const auto frame_count = reader.next_read_frames(); + const auto frames_read = reader.read_into(buffer.buffer()); + if (frames_read != frame_count) { + throw std::runtime_error( + "MultiThreadedFileReader returned an incomplete batch"); + } + return frames_read; + }; - if (verbose && (i + 1) % progress_interval == 0) { - wait_for_completed(static_cast(i + 1)); - print_progress(static_cast(i + 1)); + const auto fill_batch = [this](ReadBuffer &buffer, + std::size_t frame_count) { + std::vector> views; + views.reserve(frame_count); + auto batch_view = buffer.view(); + for (std::size_t i = 0; i < frame_count; ++i) { + views.push_back(batch_view(i)); + } + std::lock_guard fill_lock(fill_mutex_); + dispatch_fill_batch_(views); + }; + + // Exclude other queue producers for the duration. Drain anything already + // submitted before bypassing the coordinator with direct batch dispatch. + std::lock_guard ingestion_lock(ingestion_mutex_); + flush(); + + std::size_t completed = 0; + if (n_frames != 0) { + // Allocate the maximum wave size once per buffer. The last read may + // contain fewer frames; its returned count limits the views passed to + // the histogram workers, leaving the unused tail untouched. + const auto buffer_capacity = reader.next_read_frames(); + std::array buffers{ + ReadBuffer({static_cast(buffer_capacity), rows_, cols_}), + ReadBuffer({static_cast(buffer_capacity), rows_, cols_})}; + + std::size_t current_index = 0; + auto current_frames = read_into_buffer(buffers[current_index]); + while (true) { + const bool has_next = completed + current_frames < n_frames; + const std::size_t next_index = current_index ^ std::size_t{1}; + + // MultiThreadedFileReader::read_into is blocking. Run the next + // read on a dedicated prefetch task while the current batch is + // processed by the histogram worker pool. + std::future next; + if (has_next) { + next = std::async(std::launch::async, read_into_buffer, + std::ref(buffers[next_index])); + } + + fill_batch(buffers[current_index], current_frames); + completed += current_frames; + + if (verbose && (completed - last_reported >= progress_interval || + completed == n_frames)) { + print_progress(completed); + } + + if (!has_next) { + break; + } + current_frames = next.get(); + current_index = next_index; } } - flush(); + if (verbose) { - const auto done = completed_for_this_file(); - if (done > last_reported) { - print_progress(done); + if (completed > last_reported || n_frames == 0) { + print_progress(completed); } fmt::print("\n\n"); std::fflush(stdout); diff --git a/src/hist/PedestalTrackingPixelHistogram.test.cpp b/src/hist/PedestalTrackingPixelHistogram.test.cpp new file mode 100644 index 00000000..2c9f1590 --- /dev/null +++ b/src/hist/PedestalTrackingPixelHistogram.test.cpp @@ -0,0 +1,289 @@ +// SPDX-License-Identifier: MPL-2.0 +#include "aare/hist/PedestalTrackingPixelHistogram.hpp" + +#include "aare/File.hpp" +#include "aare/Frame.hpp" +#include "aare/NumpyFile.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +using aare::FileConfig; +using aare::Frame; +using aare::NumpyFile; +using aare::PedestalTrackingPixelHistogram; + +namespace { + +class TemporaryHistogramFile { + public: + using Generator = + std::function; + + TemporaryHistogramFile(std::size_t rows = 2, std::size_t cols = 3, + std::size_t frames = 10, Generator generator = {}) { + const auto unique = + std::chrono::steady_clock::now().time_since_epoch().count(); + path_ = std::filesystem::temp_directory_path() / + ("aare-pedestal-hist-" + std::to_string(unique) + ".npy"); + + FileConfig config; + config.dtype = aare::Dtype::UINT16; + config.rows = rows; + config.cols = cols; + NumpyFile file(path_, "w", config); + for (std::size_t frame_index = 0; frame_index < frames; ++frame_index) { + Frame frame(rows, cols, config.dtype); + auto image = frame.view(); + for (ssize_t row = 0; row < image.shape(0); ++row) { + for (ssize_t col = 0; col < image.shape(1); ++col) { + image(row, col) = + generator ? generator(frame_index, + static_cast(row), + static_cast(col)) + : static_cast(frame_index); + } + } + file.write(frame); + } + } + + ~TemporaryHistogramFile() { std::filesystem::remove(path_); } + + TemporaryHistogramFile(const TemporaryHistogramFile &) = delete; + TemporaryHistogramFile &operator=(const TemporaryHistogramFile &) = delete; + + const std::filesystem::path &path() const { return path_; } + + private: + std::filesystem::path path_; +}; + +} // namespace + +TEST_CASE("Pedestal tracking histogram fills ordered multi-reader batches", + "[PedestalTrackingPixelHistogram]") { + TemporaryHistogramFile file; + PedestalTrackingPixelHistogram histogram(2, 3, 10, 0.0f, 10.0f, 2, 4, 0.0f); + + // Two reader workers claiming two frames each produces a full batch of + // four followed by a partial batch of three. + histogram.fill_from_file(file.path(), 7, false, 2, 2); + const auto values = histogram.values(); + + for (ssize_t row = 0; row < 2; ++row) { + for (ssize_t col = 0; col < 3; ++col) { + for (ssize_t bin = 0; bin < 10; ++bin) { + CHECK(values(row, col, bin) == (bin < 7 ? 1 : 0)); + } + } + } +} + +TEST_CASE("Pedestal tracking file fill handles limits and reader options", + "[PedestalTrackingPixelHistogram]") { + TemporaryHistogramFile file; + PedestalTrackingPixelHistogram histogram(2, 3, 10, 0.0f, 10.0f, 2, 4, 0.0f); + + CHECK_NOTHROW(histogram.fill_from_file(file.path(), 0, false, 2, 2)); + CHECK_THROWS_AS(histogram.fill_from_file(file.path(), -2, false, 2, 2), + std::invalid_argument); + CHECK_THROWS_AS(histogram.fill_from_file(file.path(), -1, false, 0, 2), + std::invalid_argument); + CHECK_THROWS_AS(histogram.fill_from_file(file.path(), -1, false, 2, 0), + std::invalid_argument); + + // Preserve the previous API's clamp-at-EOF behaviour. + histogram.fill_from_file(file.path(), 100, false, 2, 3); + const auto values = histogram.values(); + for (ssize_t row = 0; row < 2; ++row) { + for (ssize_t col = 0; col < 3; ++col) { + for (ssize_t bin = 0; bin < 10; ++bin) { + CHECK(values(row, col, bin) == 1); + } + } + } +} + +TEST_CASE("Pedestal tracking file fill validates frame metadata", + "[PedestalTrackingPixelHistogram]") { + TemporaryHistogramFile wrong_shape(3, 3, 1); + PedestalTrackingPixelHistogram histogram(2, 3, 10, 0.0f, 10.0f, 1, 4, 0.0f); + + CHECK_THROWS_AS( + histogram.fill_from_file(wrong_shape.path(), -1, false, 2, 1), + std::invalid_argument); +} + +TEST_CASE("Histogram-only fill crosses a pixel tile without changing pedestal", + "[PedestalTrackingPixelHistogram]") { + constexpr int rows = 1; + constexpr int cols = 513; + constexpr std::size_t pedestal_samples = 1000; + + const auto baseline = [](std::size_t col) { + return static_cast(100 + col % 17); + }; + TemporaryHistogramFile file( + rows, cols, 2, + [baseline](std::size_t frame, std::size_t, std::size_t col) { + return static_cast(baseline(col) + frame + 1); + }); + + // Negative n_sigma exercises the complete histogram-only condition; zero + // is covered by the ordered batch tests above. + PedestalTrackingPixelHistogram histogram(rows, cols, 4, 0.0f, 4.0f, 1, 6, + -1.0f); + + for (std::size_t seed = 0; seed < pedestal_samples; ++seed) { + aare::NDArray frame({rows, cols}); + const int offset = seed % 2 == 0 ? -1 : 1; + for (ssize_t col = 0; col < cols; ++col) { + frame(0, col) = static_cast( + static_cast(baseline(static_cast(col))) + + offset); + } + histogram.push_pedestal_no_update(frame.view()); + } + histogram.update_mean(); + const auto mean_before = histogram.pedestal_mean(); + + // One two-frame batch traverses the 512-pixel tile and its one-pixel tail. + histogram.fill_from_file(file.path(), -1, false, 1, 2); + + const auto values = histogram.values(); + for (ssize_t col = 0; col < cols; ++col) { + for (ssize_t bin = 0; bin < 4; ++bin) { + CHECK(values(0, col, bin) == ((bin == 1 || bin == 2) ? 1 : 0)); + } + } + + const auto mean_after = histogram.pedestal_mean(); + CHECK( + std::equal(mean_before.begin(), mean_before.end(), mean_after.begin())); +} + +TEST_CASE("Pedestal tracking threshold is strict and rejects zero sigma", + "[PedestalTrackingPixelHistogram]") { + constexpr std::size_t pedestal_samples = 1000; + PedestalTrackingPixelHistogram histogram(1, 3, 8, -4.0f, 4.0f, 1, 4, 2.0f); + + // Pixels 0 and 1 have mean 100 and population sigma 1. Pixel 2 has the + // same mean and zero sigma. + for (std::size_t seed = 0; seed < pedestal_samples; ++seed) { + aare::NDArray frame({1, 3}); + const auto noisy = static_cast(seed % 2 == 0 ? 99 : 101); + frame(0, 0) = noisy; + frame(0, 1) = noisy; + frame(0, 2) = 100; + histogram.push_pedestal_no_update(frame.view()); + } + histogram.update_mean(); + + const auto mean_before = histogram.pedestal_mean(); + REQUIRE(mean_before(0, 0) == 100.0f); + REQUIRE(mean_before(0, 1) == 100.0f); + REQUIRE(mean_before(0, 2) == 100.0f); + + aare::NDArray frame({1, 3}); + frame(0, 0) = 102; // residual == 2 * sigma: excluded + frame(0, 1) = 101; // residual < 2 * sigma: included + frame(0, 2) = 101; // sigma == 0: excluded + histogram.fill_async(std::move(frame)); + histogram.flush(); + + const auto mean_after = histogram.pedestal_mean(); + CHECK(mean_after(0, 0) == 100.0f); + CHECK(mean_after(0, 1) == static_cast(100001.0 * (1.0 / 1000.0))); + CHECK(mean_after(0, 2) == 100.0f); + + // Histogramming uses each residual before a possible EMA update. + const auto values = histogram.values(); + for (ssize_t bin = 0; bin < 8; ++bin) { + CHECK(values(0, 0, bin) == (bin == 6 ? 1 : 0)); + CHECK(values(0, 1, bin) == (bin == 5 ? 1 : 0)); + CHECK(values(0, 2, bin) == (bin == 5 ? 1 : 0)); + } +} + +TEST_CASE("Tiled pedestal tracking matches chronological single-frame fills", + "[PedestalTrackingPixelHistogram]") { + constexpr int rows = 5; + constexpr int cols = 257; + constexpr std::size_t frames = 40; + constexpr int bins = 32; + constexpr float xmin = -16.0f; + constexpr float xmax = 16.0f; + + const auto baseline = [](std::size_t row, std::size_t col) { + return static_cast(100 + (row + col) % 5); + }; + const auto sample = [baseline](std::size_t frame, std::size_t row, + std::size_t col) { + const int offsets[] = {1, -1, 8, 0}; + return static_cast(static_cast(baseline(row, col)) + + offsets[frame % 4]); + }; + + TemporaryHistogramFile file(rows, cols, frames, sample); + PedestalTrackingPixelHistogram tiled(rows, cols, bins, xmin, xmax, 2, 16, + 2.0f); + PedestalTrackingPixelHistogram chronological(rows, cols, bins, xmin, xmax, + 2, 16, 2.0f); + + // Seed a non-zero cached standard deviation. Five rows split over two + // workers make both row shards cross the 512-pixel tile boundary. Forty + // data frames exercise chronological processing across a sizable batch. + for (std::size_t seed = 0; seed < 1000; ++seed) { + aare::NDArray frame({rows, cols}); + const int offset = seed % 2 == 0 ? -2 : 2; + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + frame(row, col) = static_cast( + static_cast(baseline(row, col)) + offset); + } + } + tiled.push_pedestal_no_update(frame.view()); + chronological.push_pedestal_no_update(frame.view()); + } + tiled.update_mean(); + chronological.update_mean(); + + // One forty-frame reader wave exercises tiled batch processing. + tiled.fill_from_file(file.path(), -1, false, 2, 20); + + // The reference path establishes the same result one chronological frame + // at a time, without any cross-frame traversal reordering. + for (std::size_t frame_index = 0; frame_index < frames; ++frame_index) { + aare::NDArray frame({rows, cols}); + for (ssize_t row = 0; row < rows; ++row) { + for (ssize_t col = 0; col < cols; ++col) { + frame(row, col) = + sample(frame_index, static_cast(row), + static_cast(col)); + } + } + chronological.fill_async(std::move(frame)); + chronological.flush(); + } + + const auto tiled_values = tiled.values(); + const auto chronological_values = chronological.values(); + REQUIRE(tiled_values.shape() == chronological_values.shape()); + CHECK(std::equal(tiled_values.begin(), tiled_values.end(), + chronological_values.begin())); + + const auto tiled_mean = tiled.pedestal_mean(); + const auto chronological_mean = chronological.pedestal_mean(); + REQUIRE(tiled_mean.shape() == chronological_mean.shape()); + CHECK(std::equal(tiled_mean.begin(), tiled_mean.end(), + chronological_mean.begin())); +} diff --git a/src/hist/PixelHistogram.test.cpp b/src/hist/PixelHistogram.test.cpp index d681c789..50b1fcec 100644 --- a/src/hist/PixelHistogram.test.cpp +++ b/src/hist/PixelHistogram.test.cpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #include #include diff --git a/src/hist/PixelHistogramImpl.test.cpp b/src/hist/PixelHistogramImpl.test.cpp index 3cc28424..f9456efa 100644 --- a/src/hist/PixelHistogramImpl.test.cpp +++ b/src/hist/PixelHistogramImpl.test.cpp @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #include #include @@ -72,6 +73,17 @@ TEST_CASE("Fill a small histogram from an NDArray") { REQUIRE(v(1, 1, 4) == 1); } +TEST_CASE("Flat pixel filling uses pixel-major histogram storage") { + aare::PixelHistogramImpl hist(2, 3, 4, 0.0f, 4.0f); + + hist.fill_flat_unchecked(0, 0.5f); + hist.fill_flat_unchecked(4, 2.5f); + + const auto values = hist.view(); + CHECK(values(0, 0, 0) == 1); + CHECK(values(1, 1, 2) == 1); +} + TEST_CASE("Check that pixel histogram does not overflow") { int rows = 1; int cols = 1; diff --git a/src/to_string.cpp b/src/to_string.cpp index d950b13c..2753695f 100644 --- a/src/to_string.cpp +++ b/src/to_string.cpp @@ -1,4 +1,5 @@ -#include "to_string.hpp" +// SPDX-License-Identifier: MPL-2.0 +#include "aare/to_string.hpp" namespace aare { @@ -35,9 +36,15 @@ template <> DetectorType string_to(const std::string &arg) { template <> TimingMode string_to(const std::string &arg) { if (arg == "auto") - return TimingMode::Auto; + return TimingMode::AUTO_TIMING; if (arg == "trigger") - return TimingMode::Trigger; + return TimingMode::TRIGGER_EXPOSURE; + if (arg == "gating") + return TimingMode::GATED; + if (arg == "burst_trigger") + return TimingMode::BURST_TRIGGER; + if (arg == "trigger_gating") + return TimingMode::TRIGGER_GATED; throw std::runtime_error("Could not decode timing mode from: \"" + arg + "\""); } @@ -235,6 +242,34 @@ template <> DACIndex string_to(const std::string &arg) { "\""); } +template <> UDPPortPosition string_to(const std::string &arg) { + if (arg == "left") + return UDPPortPosition::LEFT; + if (arg == "right") + return UDPPortPosition::RIGHT; + if (arg == "top") + return UDPPortPosition::TOP; + if (arg == "bottom") + return UDPPortPosition::BOTTOM; + throw std::runtime_error("Could not decode UDPPortPosition from: \"" + arg + + "\""); +} + +std::string to_string(const corner &arg) { + switch (arg) { + case corner::cTopLeft: + return "TopLeft"; + case corner::cTopRight: + return "TopRight"; + case corner::cBottomLeft: + return "BottomLeft"; + case corner::cBottomRight: + return "BottomRight"; + default: + throw std::runtime_error("Invalid corner value"); + } +} + std::string remove_unit(std::string &str) { auto it = str.begin(); while (it != str.end()) { diff --git a/src/to_string.test.cpp b/src/to_string.test.cpp index 2385700a..1489632f 100644 --- a/src/to_string.test.cpp +++ b/src/to_string.test.cpp @@ -1,10 +1,10 @@ // SPDX-License-Identifier: MPL-2.0 +#include "aare/to_string.hpp" #include "aare/defs.hpp" #include #include -#include "to_string.hpp" using aare::string_to; TEST_CASE("DetectorType string to enum") { @@ -36,9 +36,10 @@ TEST_CASE("DetectorType string to enum") { } TEST_CASE("TimingMode string to enum") { - REQUIRE(string_to("auto") == aare::TimingMode::Auto); + REQUIRE(string_to("auto") == + aare::TimingMode::AUTO_TIMING); REQUIRE(string_to("trigger") == - aare::TimingMode::Trigger); + aare::TimingMode::TRIGGER_EXPOSURE); REQUIRE_THROWS(string_to("invalid_mode")); } diff --git a/tests/raw_file_helpers.hpp b/tests/raw_file_helpers.hpp new file mode 100644 index 00000000..b1a9b17c --- /dev/null +++ b/tests/raw_file_helpers.hpp @@ -0,0 +1,61 @@ +// SPDX-License-Identifier: MPL-2.0 +#pragma once + +#include "aare/defs.hpp" + +#include +#include +#include +#include +#include + +class TemporaryRawFiles { + std::filesystem::path m_directory; + + public: + explicit TemporaryRawFiles(size_t modules = 1) { + const auto unique = + std::chrono::steady_clock::now().time_since_epoch().count(); + m_directory = std::filesystem::temp_directory_path() / + ("aare-raw-" + std::to_string(unique)); + std::filesystem::create_directory(m_directory); + const nlohmann::json metadata = { + {"Version", 7.2}, + {"Detector Type", "Jungfrau"}, + {"Timing Mode", "auto"}, + {"Geometry", {{"x", 1}, {"y", modules}}}, + {"Image Size in bytes", 12}, + {"Pixels", {{"x", 3}, {"y", 2}}}, + {"Max Frames Per File", 1}, + {"Total Frames", 2}, + {"Frames in File", 2}, + {"Frame Padding", 1}, + {"Frame Discard Policy", "nodiscard"}}; + std::ofstream(master_path()) << metadata; + for (size_t module = 0; module < modules; ++module) { + for (size_t file = 0; file < 2; ++file) { + std::ofstream output(data_path(module, file), std::ios::binary); + aare::DetectorHeader header{}; + header.frameNumber = 100 + file; + const uint16_t pixels[6] = {1, 2, 3, 4, 5, 6}; + output.write(reinterpret_cast(&header), + sizeof(header)); + output.write(reinterpret_cast(pixels), + sizeof(pixels)); + } + } + } + + TemporaryRawFiles(const TemporaryRawFiles &) = delete; + TemporaryRawFiles &operator=(const TemporaryRawFiles &) = delete; + ~TemporaryRawFiles() { std::filesystem::remove_all(m_directory); } + + std::filesystem::path master_path() const { + return m_directory / "run_master_0.json"; + } + + std::filesystem::path data_path(size_t module = 0, size_t file = 0) const { + return m_directory / ("run_d" + std::to_string(module) + "_f" + + std::to_string(file) + "_0.raw"); + } +}; diff --git a/tests/test_config.hpp.in b/tests/test_config.hpp.in index e314b8fe..8030213e 100644 --- a/tests/test_config.hpp.in +++ b/tests/test_config.hpp.in @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: MPL-2.0 #pragma once #include #include