## Summary
Makes the minimizer behind `FitModel` selectable and adds two built-in,
dependency-free solvers next to Minuit2's Migrad and Fumili.
`Minimizer.Migrad` stays the default, so existing fits behave as before.
- **`Minimizer` enum** (C++ `FitModel::SetMinimizer`, Python `minimizer`
constructor argument and property): `Migrad`, `Fumili`,
`LevenbergMarquardt`, `VarPro`.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Khalil Ferjaoui <khalilferjaoui@yahoo.fr>
We have no code in aare to deal with RawFiles that have missing packets
if they are not padded to full size or frames with packet loss dropped.
Added:
- check that we don't try to read files that are not supported
- python bindings for frame discard policy
at least we get the check in:
closes #171
---------
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- error upon warning when building documentation (warnings often
indicate malrendered documentation)
- fixed some warnings
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- renamed pedestal.hpp to bind_Pedestal.hpp for python bindings
- variance is now always calculated in double and made private
- cost correctness in a few places
- removed push_fast and explicit updates of mean from Pedestal. If
performance is needed use FastPedestal
- Pedestal no longer caches std
Improved API for ClusterFile
- read_frame now returns a std::optional<ClusterVector> to differentiate
between an empty frame and end of file
- f.frames() produces an iterator over frames
- f.chunks() produces an iterator reading chunks of clusters
- Updated error message to mention out of range and which file it
applies to
- If there are different number of frames in the subfiles, take the
minimum and print a warning
Closes: #363
Closes: #362
- fixed test in ClusterFinderMT (use minimum one pedestal to accumulate
dummy pedestal)
- added minimum_pedestal_samples as additional constructor argument in
ClusterFinder and ClusterFinderMT
- bugfix: race conditions for empty queues in collect() - check directly
when needed
- Default `ClusterVector` move operations simplifying the code and
fixing a bug.
- Fixed inconsistent frame number type (uint64/int32)
- Validate Python masks as one-dimensional, C-contiguous Boolean arrays,
handle empty masks safely, and reserve filtered storage based on the
selected cluster count.
- Align the C++ and Python API documentation with the implementation,
including concise `hitmap` and reduction documentation and a correctly
rendered constructor example.
- handles file reading of disabled udp ports
- adds members disabled_udp_ports in master file as an optional
- adds member diasbled_udp_port_types in master file as an optional
- Treats enabled udp ports as ROIs
- merges/combines consecutive ROis into one
### To discuss:
- right now can only handle either ROI or disabled udp ports (disabled
udp ports has higher precedence)
- Should Frames in File be removed from Master File as value is nonsense
for disabled udp ports
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder.
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro
**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state
**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)
**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)
**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
- reshape image directly in decoder such that first dimension is num
counters
- take into account chip artefact in decoder.
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Multi threaded filling of per pixel histograms for example for detector calibration
1. PixelHistogram - Generic variant expects already pedestal subtracted
data
2. PedestalTrackingHistogram - Terrible name, useful class. Keeps it's
own pedestal and does conversion and pedestal tracking in the worker
threads.
---------
Co-authored-by: Lars Erik Fröjd <froejdh_e@pc-jungfrau-02.psi.ch>
- Allowing the users more flexibility to play around with custom eta
functions without touching the c++ code
- passing vector of eta values to ``transform_eta_values``
```
from aare import Interpolator, ClusterVector, Etai, Cluster
import numpy as np
def custom_eta(cluster_pixel_coordinate_x, cluster_pixel_coordinate_y, cluster_data):
# dummy custom eta function that just returns the sum of the cluster data
eta = Etai()
eta.x = 0.1 # dummy x value
eta.y = 0.1 # dummy y value
eta.sum = np.sum(cluster_data) # sum of the cluster data as the "energy
return eta
# Create a dummy eta distribution and bins
eta_distribution = np.zeros((10, 10, 1)) # dummy eta distribution
etax_bins = np.linspace(0, 1.0, 11)
etay_bins = np.linspace(0, 1.0, 11)
e_bins = np.array([0., 10.]) # dummy energy bins
# Create the interpolator
interpolator = Interpolator(eta_distribution, etax_bins, etay_bins, e_bins)
# Create a dummy cluster vector
cluster_vector = ClusterVector()
cluster_vector.push_back(Cluster(10, 5, np.ones(shape=9, dtype = np.int32)))
cluster_vector.push_back(Cluster(20, 10, np.ones(shape=9, dtype = np.int32)))
# Create dummy etas for the clusters
cluster_array = np.array(cluster_vector)
etas = np.array([custom_eta(cluster["x"], cluster["y"], cluster["data"]) for cluster in cluster_array])
# transform eta values to uniform coordinates
uniform_coordinates = interpolator.transform_eta_values(etas)
# Interpolate to get the photon coordinates e.g. apply interpolation logic
photon_coordinates_x = cluster_array["x"] + uniform_coordinates["x"] # add to pixel coordinate
photon_coordinates_y = cluster_array["y"] + uniform_coordinates["y"] # add to pixel coordinate
```
advantage: full control over interpolation logic,
downside: inefficient quite some loops in python
- passing pre computed eta values to interpolate function
```
Interpolator.interpolate(cluster_vector, etas)
```
downside: less flexibility in interpolation logic.
downside: People might misuse it instead of using interpolate directly
with a pre compiled eta function implemented in c++
## Unified Minuit2 fitting framework with FitModel API
### Models (`Models.hpp`)
Consolidate all model structs (Gaussian, RisingScurve, FallingScurve)
into a
single header. Each model provides: `eval`, `eval_and_grad`, `is_valid`,
`estimate_par`, `compute_steps`, and `param_info` metadata. No Minuit2
dependency.
### Chi2 functors (`Chi2.hpp`)
Generic `Chi2Model1DGrad` (analytic gradient) templated on the model
struct.
Replaces the separate Chi2Gaussian, Chi2GaussianGradient,
Chi2Scurves, and Chi2ScurvesGradient headers.
### FitModel (`FitModel.hpp`)
Configuration object wrapping `MnUserParameters`, strategy, tolerance,
and
user-override tracking. User constraints (fixed parameters, start
values, limits)
always take precedence over automatic data-driven estimates.
### Fit functions (`Fit.hpp`)
- `fit_pixel<Model, FCN>(model, x, y, y_err)` -> single-pixel,
self-contained
- `fit_pixel<Model, FCN>(model, upar_local, x, y, y_err)` -> pre-cloned
upar for hot loops
- `fit_3d<Model, FCN>(model, x, y, y_err, ..., n_threads)` ->
row-parallel over pixel grid
### Python bindings
- `Pol1`, `Pol2`, `Gaussian`, `RisingScurve`, `FallingScurve` model
classes with
`FixParameter`, `SetParLimits`, `SetParameter`, and properties for
`max_calls`, `tolerance`, `compute_errors`
- Single `fit(model, x, y, y_err, n_threads)` dispatch replacing the old
`fit_gaus_minuit`, `fit_gaus_minuit_grad`, `fit_scurve_minuit_grad`,
etc.
### Benchmarks
- Updated `fit_benchmark.cpp` (Google Benchmark) to use the new FitModel
API
- Jupyter notebooks for 1D and 3D S-curve fitting (lmfit vs Minuit2
analytic)
- ~1.8x speedup over lmfit, near-linear thread scaling up to physical
core count
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Matterhorn10 Transform
some other Transformations from pyctbGUI
added method get_reading_mode for easier error handling in decoders
## TODO:
- proper error handling for all other decoders
- proper documentation for all other decoders
- refactoring all other decoders to store hard coded values in a Struct
ChipSpecification
This PR adds support for decoding digital data from the my320 test chip.
- Added BitOffset (strong type)
- Expand 24 to 32 bit
- Python bindings for decoding my302
- Improved docs
- added rosenblatttransform
- added 3x3 eta methods
- interpolation can be used with various eta functions
- added documentation for interpolation, eta calculation
- exposed full eta struct in python
- disable ClusterFinder for 2x2 clusters
- factory function for ClusterVector
---------
Co-authored-by: Dhanya Thattil <dhanya.thattil@psi.ch>
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
When refactoring the dispatch of the python binding for ClusterFile I
forgot chunk_size. Adding it back in.
Excluded from release notes since the bug was introduced after the last
release and now fixed before the next release.
1. added back chunk_size
2. removed a few commented out lines
closes #197
- added scurve fitting with two different signs (scurve, scurve2)
- at the moment no option to set initial parameters
---------
Co-authored-by: JulianHeymes <julian.heymes@psi.ch>
Changes to be able to run the example notebooks:
- Invert gain map on setting (multiplication is faster but user supplies
ADU/energy)
- Cast after applying gain map not to loose precision (Important for
int32 clusters)
- "factor" for ClusterFileSink
- Cluster size available to be able to create the right file sink
New function apply_custom_weights (can we find a better name) that takes
a uint16 and a NDView<double,1> of bases for the conversion. For each
supplied weight it is used as base (instead of 2) to convert from bits
to a double.
---------
Co-authored-by: siebsi <sieb.patr@gmail.com>
mazzol_a
Erik Fröjdh
maliakal_d
lunin_landmazzol_a
