Catch2 v3 is ~100 small translation units that together took 58.5 s of
summed compile time in a Release build, more than aare_core itself.
Batched into unity sources of 16 files they take 13 s in 10 steps. The
test executable itself is unchanged.
Clean Release build with tests and bindings: 28 s -> 23 s on 10 cores;
all 277 test cases pass, both through run_tests and ctest.
pybind11_add_module links pybind11::lto, which adds full -flto to the
extension and ignores the target's INTERPROCEDURAL_OPTIMIZATION property.
That link ran single-threaded for 19-22 s and was the longest step of the
build. LTO cannot reach aare_core (a plain static library) and the hot
loops are header templates compiled at -O3 inside the extension anyway.
Pass NO_EXTRAS so the standard CMake property decides (off by default),
and re-add the extras pybind11 would have applied: pybind11_strip for
optimized builds and pybind11::windows_extras (/bigobj, /MP) on MSVC.
set_property(TARGET _aare PROPERTY INTERPROCEDURAL_OPTIMIZATION ON) or
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON brings LTO back as ThinLTO.
Measured on a clean Release build with tests: 42 s -> 28 s, extension
link 19 s -> 0.1 s, exports identical, all tests pass. The stripped
binary grows from 4.7 MB to 5.6 MB. Pedestal, eta and the aare_core code
paths (fits, calibration, decoding) run at the same speed; the
ClusterFinder micro-benchmark is consistently 1-2 % slower without LTO.
python/src/module.cpp was one translation unit instantiating every cluster
size and value type; it compiled for 38 s and was the critical path of the
whole build. The bindings now live in bind_io.cpp (files, geometry, pixel
maps, calibration, test probes), bind_hist.cpp, bind_pedestal.cpp,
bind_fit.cpp, bind_eta.cpp and one bind_cluster_NxN.cpp per cluster size,
with the entry points declared in bindings.hpp and called from module.cpp
in the same order as before. The DEFINE_CLUSTER_BINDINGS macros moved to
bind_cluster_common.hpp.
Headers that several units include needed small fixes: allocate_image_data
and define_corner_enum are inline, the file-reader entry point is not, and
np_helper.hpp, bind_Defs.hpp, bind_FastPedestal.hpp,
bind_PedestalTrackingPixelHistogram.hpp and bind_ClusterFileSink.hpp now
include what they use, in particular pybind11/stl/filesystem.h for the
std::filesystem::path arguments they bind (previously supplied by include
order in module.cpp).
The module's exported names, types and docstrings are identical (verified
by dumping all 1948 of them), both test suites pass, and a clean Release
build with tests drops from 68 s to 42 s; the longest unit now takes 13 s
and the remaining 19 s is pybind11's LTO link.
Minuit2's ~60 small files took 45 s of summed compile time in a Release
build (32 s in Debug); batched into unity sources that drops to 9 s (5 s)
with all [fit] tests passing. libzmq gets the same treatment on its
"objects" OBJECT library for when AARE_ZMQ is on: 37 s -> 7 s of CPU,
same warnings, identical exported symbols apart from one implicitly
defined destructor that is now inlined at its single use.
All git-fetched dependencies are shallow clones like Catch2 already was.
A fresh configure with network access went from 13.4 s to 10.5 s here and
the checkouts shrink from 19 MB to 3.7 MB (fmt) and 15 MB to 3.7 MB
(pybind11).
Nothing in aare links libzmq at the moment, yet every configure fetched
it (a 25 MB clone) and ran its feature checks, which took 8 of the 10 s a
fresh configure needed. Gate the FetchContent/find_package block behind
AARE_ZMQ (default OFF) and expose the library as aare::zmq, so re-enabling
it later is -DAARE_ZMQ=ON plus one target_link_libraries line. The patch
and the AARE_FETCH_ZMQ / AARE_SYSTEM_LIBRARIES handling stay as they were.
A fresh configure without it takes about 3 s.
Now builds clean on MSVC: cl version 19.51.36260 for x64, from Visual
Studio 2026 Build Tools 18.10.3.
Major items:
- [[maybe_unused]] instead of gcc/clang specific pragmas
- Removed names variables that weren't used
- Cast + explicit uing64
Collection of fixes to make aare build on Windows. Requires the previous
PRs for compilation to succeed.
- Replaced `or` keyword with ||
- updated cmake file
- fix for 32 bit long in fseek
closes #395
closes #396
logger.hpp pulled in <sys/time.h> and used gettimeofday and localtime_r,
which do not exist on MSVC. Build the timestamp from std::chrono instead
and use localtime_s on MSVC. The output format is unchanged: <colour>-
HH:MM:SS.mmm LEVEL: message<reset>. The unused __SHORT_FORM_OF_FILE__
helper now also strips backslash separators. Add logger tests that pin
the timestamp and line format.
closes #394
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
On Windows std::filesystem::path holds wchar_t, so assigning a path
component to a std::string or passing path::c_str() to fopen does not
compile. Use path::string() at those sites, and drop a redundant path
round trip in NumpyFile.
closes #399
- MSVC has no ssize_t, which made every translation unit fail on
Windows.
- Define aare::ssize_t in defs.hpp: the platform ssize_t on POSIX (so
nothing changes there) and std::ptrdiff_t on MSVC.
- Code inside the namespace picks it up unchanged; the few files that
use ssize_t outside the namespace get a using-declaration. FilePtr.hpp
now includes defs.hpp since it had no aare include.
Closing #393
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
Makes the minimizer behind `FitModel` selectable and adds two built-in,
dependency-free solvers next to Minuit2's Migrad and Fumili.
`Minimizer.Migrad` stays the default, so existing fits behave as before.
- **`Minimizer` enum** (C++ `FitModel::SetMinimizer`, Python `minimizer`
constructor argument and property): `Migrad`, `Fumili`,
`LevenbergMarquardt`, `VarPro`.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Khalil Ferjaoui <khalilferjaoui@yahoo.fr>
- For set_nSigma only call update_pedestal if we have accumulated enough
samples otherwise just write it
- For update threshold check if pd is ready and if not throw
closes #390
- Introduce a standard way to bind numpy arrays: py::array_t<T> and
.noconvert() on the argument
- Take py::array_t by value since it's just a small wrapper
- Update bindings
- Removed faulty move (pointed out by: #227)
- Fixed license for the queue
- Runtime check for capacity=>2
Thank you @kferjaoui for finding the issue. In the end I found it better
to remove the operators rather than modifying the queue
Cleaned up the helper functions by:
- Added template for make_view (and new make_const_view) instead of the
copy paste implementation
- checks c_style and dimensions inside make_view
- test binding to be able to probe the checks in make_view
Call sites can be reviewed in a second PR since usage of const or
strictness of conversion depends...
We have no code in aare to deal with RawFiles that have missing packets
if they are not padded to full size or frames with packet loss dropped.
Added:
- check that we don't try to read files that are not supported
- python bindings for frame discard policy
at least we get the check in:
closes #171
---------
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- error upon warning when building documentation (warnings often
indicate malrendered documentation)
- fixed some warnings
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- seek to the edge of a file + read_into would select the wrong file
- Visible for example in f.read_frame(index) in python
Added fix, release notes and a small test.
closes #371
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- renamed pedestal.hpp to bind_Pedestal.hpp for python bindings
- variance is now always calculated in double and made private
- cost correctness in a few places
- removed push_fast and explicit updates of mean from Pedestal. If
performance is needed use FastPedestal
- Pedestal no longer caches std
Improved API for ClusterFile
- read_frame now returns a std::optional<ClusterVector> to differentiate
between an empty frame and end of file
- f.frames() produces an iterator over frames
- f.chunks() produces an iterator reading chunks of clusters
- python string repr for Cluster and Eta
- Context manager for RawMasterFile
- slice operator for ROI
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- Updated error message to mention out of range and which file it
applies to
- If there are different number of frames in the subfiles, take the
minimum and print a warning
Closes: #363
Closes: #362
- fixed test in ClusterFinderMT (use minimum one pedestal to accumulate
dummy pedestal)
- added minimum_pedestal_samples as additional constructor argument in
ClusterFinder and ClusterFinderMT
- bugfix: race conditions for empty queues in collect() - check directly
when needed
- Default `ClusterVector` move operations simplifying the code and
fixing a bug.
- Fixed inconsistent frame number type (uint64/int32)
- Validate Python masks as one-dimensional, C-contiguous Boolean arrays,
handle empty masks safely, and reserve filtered storage based on the
selected cluster count.
- Align the C++ and Python API documentation with the implementation,
including concise `hitmap` and reduction documentation and a correctly
rendered constructor example.
- handles file reading of disabled udp ports
- adds members disabled_udp_ports in master file as an optional
- adds member diasbled_udp_port_types in master file as an optional
- Treats enabled udp ports as ROIs
- merges/combines consecutive ROis into one
### To discuss:
- right now can only handle either ROI or disabled udp ports (disabled
udp ports has higher precedence)
- Should Frames in File be removed from Master File as value is nonsense
for disabled udp ports
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder.
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro
**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state
**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)
**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)
**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
Minor API change:
- NDView<T> implicitly converts to NDView<const T> (as std::span in
C++20)
- expand4to8bit and expand24to32bit now takes a view over const uint8_t
to show that the input is not changes
closes #291
Before v8.0 we should always try to read analog samples. This bug
resulted in old Mönch03 data not being reordered.
1. Added test for master file parsing
2. Fixed code
closing #348
Erik Fröjdh
