Now builds clean on MSVC: cl version 19.51.36260 for x64, from Visual
Studio 2026 Build Tools 18.10.3.
Major items:
- [[maybe_unused]] instead of gcc/clang specific pragmas
- Removed names variables that weren't used
- Cast + explicit uing64
Collection of fixes to make aare build on Windows. Requires the previous
PRs for compilation to succeed.
- Replaced `or` keyword with ||
- updated cmake file
- fix for 32 bit long in fseek
closes #395
closes #396
logger.hpp pulled in <sys/time.h> and used gettimeofday and localtime_r,
which do not exist on MSVC. Build the timestamp from std::chrono instead
and use localtime_s on MSVC. The output format is unchanged: <colour>-
HH:MM:SS.mmm LEVEL: message<reset>. The unused __SHORT_FORM_OF_FILE__
helper now also strips backslash separators. Add logger tests that pin
the timestamp and line format.
closes #394
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
- MSVC has no ssize_t, which made every translation unit fail on
Windows.
- Define aare::ssize_t in defs.hpp: the platform ssize_t on POSIX (so
nothing changes there) and std::ptrdiff_t on MSVC.
- Code inside the namespace picks it up unchanged; the few files that
use ssize_t outside the namespace get a using-declaration. FilePtr.hpp
now includes defs.hpp since it had no aare include.
Closing #393
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
Makes the minimizer behind `FitModel` selectable and adds two built-in,
dependency-free solvers next to Minuit2's Migrad and Fumili.
`Minimizer.Migrad` stays the default, so existing fits behave as before.
- **`Minimizer` enum** (C++ `FitModel::SetMinimizer`, Python `minimizer`
constructor argument and property): `Migrad`, `Fumili`,
`LevenbergMarquardt`, `VarPro`.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Khalil Ferjaoui <khalilferjaoui@yahoo.fr>
- For set_nSigma only call update_pedestal if we have accumulated enough
samples otherwise just write it
- For update threshold check if pd is ready and if not throw
closes #390
- Introduce a standard way to bind numpy arrays: py::array_t<T> and
.noconvert() on the argument
- Take py::array_t by value since it's just a small wrapper
- Update bindings
We have no code in aare to deal with RawFiles that have missing packets
if they are not padded to full size or frames with packet loss dropped.
Added:
- check that we don't try to read files that are not supported
- python bindings for frame discard policy
at least we get the check in:
closes #171
---------
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- renamed pedestal.hpp to bind_Pedestal.hpp for python bindings
- variance is now always calculated in double and made private
- cost correctness in a few places
- removed push_fast and explicit updates of mean from Pedestal. If
performance is needed use FastPedestal
- Pedestal no longer caches std
Improved API for ClusterFile
- read_frame now returns a std::optional<ClusterVector> to differentiate
between an empty frame and end of file
- f.frames() produces an iterator over frames
- f.chunks() produces an iterator reading chunks of clusters
- python string repr for Cluster and Eta
- Context manager for RawMasterFile
- slice operator for ROI
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- Updated error message to mention out of range and which file it
applies to
- If there are different number of frames in the subfiles, take the
minimum and print a warning
Closes: #363
Closes: #362
- fixed test in ClusterFinderMT (use minimum one pedestal to accumulate
dummy pedestal)
- added minimum_pedestal_samples as additional constructor argument in
ClusterFinder and ClusterFinderMT
- bugfix: race conditions for empty queues in collect() - check directly
when needed
- Default `ClusterVector` move operations simplifying the code and
fixing a bug.
- Fixed inconsistent frame number type (uint64/int32)
- Validate Python masks as one-dimensional, C-contiguous Boolean arrays,
handle empty masks safely, and reserve filtered storage based on the
selected cluster count.
- Align the C++ and Python API documentation with the implementation,
including concise `hitmap` and reduction documentation and a correctly
rendered constructor example.
- handles file reading of disabled udp ports
- adds members disabled_udp_ports in master file as an optional
- adds member diasbled_udp_port_types in master file as an optional
- Treats enabled udp ports as ROIs
- merges/combines consecutive ROis into one
### To discuss:
- right now can only handle either ROI or disabled udp ports (disabled
udp ports has higher precedence)
- Should Frames in File be removed from Master File as value is nonsense
for disabled udp ports
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder.
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro
**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state
**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)
**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)
**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
Minor API change:
- NDView<T> implicitly converts to NDView<const T> (as std::span in
C++20)
- expand4to8bit and expand24to32bit now takes a view over const uint8_t
to show that the input is not changes
closes #291
- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
- Move Chi2.hpp from include/aare/ to src/ (private)
- Pimpl on FitModel<Model>: MnUserParameters/MnStrategy behind opaque
src/FitModelImpl.hpp, no Minuit2 includes in public headers
- Move fit_pixel/fit_3d bodies to Fit.cpp with explicit instantiations
for all 8 models; drop FCN template param from public API
- CMake: aare::Minuit2 wrapped in $<BUILD_INTERFACE:...> (hidden from
exported targets, same pattern as lmfit), MINUIT2_INSTALL OFF, Chi2.hpp
removed from PUBLICHEADERS
- Update python bindings and benchmark callsites accordingly
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Co-authored-by: Alice <alice.mazzoleni@psi.ch>
Collection of small improvements in usability:
- NDView works for large arrays
- Direct subtraction of Pedestal from np.array
- len() support for python bindings of files
Multi threaded filling of per pixel histograms for example for detector calibration
1. PixelHistogram - Generic variant expects already pedestal subtracted
data
2. PedestalTrackingHistogram - Terrible name, useful class. Keeps it's
own pedestal and does conversion and pedestal tracking in the worker
threads.
---------
Co-authored-by: Lars Erik Fröjd <froejdh_e@pc-jungfrau-02.psi.ch>
Add options for var cluster_finder_X:
1. number of neighbors (for better segmentation of clusters)
2. option to empty the surrounding pixels
---------
Co-authored-by: xiangyu.xie <xiangyu.xie@psi.ch>
- Allowing the users more flexibility to play around with custom eta
functions without touching the c++ code
- passing vector of eta values to ``transform_eta_values``
```
from aare import Interpolator, ClusterVector, Etai, Cluster
import numpy as np
def custom_eta(cluster_pixel_coordinate_x, cluster_pixel_coordinate_y, cluster_data):
# dummy custom eta function that just returns the sum of the cluster data
eta = Etai()
eta.x = 0.1 # dummy x value
eta.y = 0.1 # dummy y value
eta.sum = np.sum(cluster_data) # sum of the cluster data as the "energy
return eta
# Create a dummy eta distribution and bins
eta_distribution = np.zeros((10, 10, 1)) # dummy eta distribution
etax_bins = np.linspace(0, 1.0, 11)
etay_bins = np.linspace(0, 1.0, 11)
e_bins = np.array([0., 10.]) # dummy energy bins
# Create the interpolator
interpolator = Interpolator(eta_distribution, etax_bins, etay_bins, e_bins)
# Create a dummy cluster vector
cluster_vector = ClusterVector()
cluster_vector.push_back(Cluster(10, 5, np.ones(shape=9, dtype = np.int32)))
cluster_vector.push_back(Cluster(20, 10, np.ones(shape=9, dtype = np.int32)))
# Create dummy etas for the clusters
cluster_array = np.array(cluster_vector)
etas = np.array([custom_eta(cluster["x"], cluster["y"], cluster["data"]) for cluster in cluster_array])
# transform eta values to uniform coordinates
uniform_coordinates = interpolator.transform_eta_values(etas)
# Interpolate to get the photon coordinates e.g. apply interpolation logic
photon_coordinates_x = cluster_array["x"] + uniform_coordinates["x"] # add to pixel coordinate
photon_coordinates_y = cluster_array["y"] + uniform_coordinates["y"] # add to pixel coordinate
```
advantage: full control over interpolation logic,
downside: inefficient quite some loops in python
- passing pre computed eta values to interpolate function
```
Interpolator.interpolate(cluster_vector, etas)
```
downside: less flexibility in interpolation logic.
downside: People might misuse it instead of using interpolate directly
with a pre compiled eta function implemented in c++
To improve codebase quality and reduce human error, this PR introduces
the pre-commit framework. This ensures that all code adheres to project
standards before it is even committed, maintaining a consistent style
and catching common mistakes early.
Key Changes:
- Code Formatting: Automated C++ formatting using clang-format (based on
the project's .clang-format file).
- Syntax Validation: Basic checks for file integrity and syntax.
- Spell Check: Automated scanning for typos in source code and comments.
- CMake Formatting: Standardization of CMakeLists.txt and .cmake
configuration files.
- GitHub Workflow: Added a CI action that validates every Pull Request
against the pre-commit configuration to ensure compliance.
The configuration includes a [ci] block to handle automated fixes within
the PR. Currently, this is disabled. If we want the CI to automatically
commit formatting fixes back to the PR branch, this can be toggled to
true in .pre-commit-config.yaml.
```yaml
ci:
autofix_commit_msg: [pre-commit] auto fixes from pre-commit hooks
autofix_prs: false
autoupdate_schedule: monthly
```
The last large commit with the fit functions, for example, was not
formatted according to the clang-format rules. This PR would allow to
avoid similar mistakes in the future.
Python fomat with `ruff` for tests and sanitiser for `.ipynb` notebooks
can be added as well.
- Set parameter starting values, limits or fix through name as well as
index
- Updated parameter names for the scurve
- Fast approximation to erf function (~10% speedup of fitting)
---------
Co-authored-by: Khalil Ferjaoui <khalilferjaoui@yahoo.fr>
## Unified Minuit2 fitting framework with FitModel API
### Models (`Models.hpp`)
Consolidate all model structs (Gaussian, RisingScurve, FallingScurve)
into a
single header. Each model provides: `eval`, `eval_and_grad`, `is_valid`,
`estimate_par`, `compute_steps`, and `param_info` metadata. No Minuit2
dependency.
### Chi2 functors (`Chi2.hpp`)
Generic `Chi2Model1DGrad` (analytic gradient) templated on the model
struct.
Replaces the separate Chi2Gaussian, Chi2GaussianGradient,
Chi2Scurves, and Chi2ScurvesGradient headers.
### FitModel (`FitModel.hpp`)
Configuration object wrapping `MnUserParameters`, strategy, tolerance,
and
user-override tracking. User constraints (fixed parameters, start
values, limits)
always take precedence over automatic data-driven estimates.
### Fit functions (`Fit.hpp`)
- `fit_pixel<Model, FCN>(model, x, y, y_err)` -> single-pixel,
self-contained
- `fit_pixel<Model, FCN>(model, upar_local, x, y, y_err)` -> pre-cloned
upar for hot loops
- `fit_3d<Model, FCN>(model, x, y, y_err, ..., n_threads)` ->
row-parallel over pixel grid
### Python bindings
- `Pol1`, `Pol2`, `Gaussian`, `RisingScurve`, `FallingScurve` model
classes with
`FixParameter`, `SetParLimits`, `SetParameter`, and properties for
`max_calls`, `tolerance`, `compute_errors`
- Single `fit(model, x, y, y_err, n_threads)` dispatch replacing the old
`fit_gaus_minuit`, `fit_gaus_minuit_grad`, `fit_scurve_minuit_grad`,
etc.
### Benchmarks
- Updated `fit_benchmark.cpp` (Google Benchmark) to use the new FitModel
API
- Jupyter notebooks for 1D and 3D S-curve fitting (lmfit vs Minuit2
analytic)
- ~1.8x speedup over lmfit, near-linear thread scaling up to physical
core count
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Matterhorn10 Transform
some other Transformations from pyctbGUI
added method get_reading_mode for easier error handling in decoders
## TODO:
- proper error handling for all other decoders
- proper documentation for all other decoders
- refactoring all other decoders to store hard coded values in a Struct
ChipSpecification
Reading multiple ROI's for aare
- read_frame, read_n etc throws for multiple ROIs
- new functions read_ROIs, read_n_ROIs
- read_roi_into (used for python bindings - to not copy)
all these functions use get_frame or get_frame_into where one passes the
roi_index
## Refactoring:
- each roi keeps track of its subfiles that one has to open e.g.
subfiles can be opened several times
- refactored class DetectorGeometry - keep track of the updated module
geometries in new class ROIGeometry.
- ModuleGeometry updates based on ROI
## ROIGeometry:
- stores number of modules overlapping with ROI and its indices
- size of ROI
Note: only tested size of the resulting frames not the actual values
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Co-authored-by: Erik Fröjdh <erik.frojdh@gmail.com>
- automatically run python tests
- automatically run test using data files on local runner from gitea
- fixed some of the workflows
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Erik Fröjdh
lunin_l