Every bundled model now declares the parameters it is linear in
(linear_par, basis_and_grad) and, except the polynomials, a vectorisable
bulk basis pass (basis_columns) built on the new model::fast_exp. The
Levenberg-Marquardt iteration is factored into a shared DampedGaussNewton
driver with unchanged results; VarPro solves the linear parameters exactly
at every trial point and iterates over the nonlinear ones, rejecting
degenerate trial points and polishing the converged point with one
undamped step. Limits on linear parameters, pixels that do not converge
and models without the separable structure fall back to
LevenbergMarquardt. Parameter errors are the same Gauss-Newton estimates.
The [fit] tests run VarPro through every configuration and check the
separable structure of each model against its derivatives.
Measured on an M1 Max, single thread: FallingScurve population 5.5 us per
fit against 11.7 us with LevenbergMarquardt, and 2 wrong minima in 4096
pixels against 30; Gaussian 3.3 vs 4.3 us; GaussianChargeSharingKb 33 vs
49 us; Pol2 0.34 vs 1.52 us. Migrad stays the default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Introduce a standard way to bind numpy arrays: py::array_t<T> and
.noconvert() on the argument
- Take py::array_t by value since it's just a small wrapper
- Update bindings
Bring the dependency-free Levenberg-Marquardt solver from dev/custom-fit in
as Minimizer::LevenbergMarquardt next to Minuit2's Migrad (default) and
Fumili, selected through the FitModel constructor, SetMinimizer() or the
Python `minimizer` argument and property.
- FitModel becomes plain data (limits, fixed flags, start values, minimizer
settings) with index and limit validation. Minuit2 user parameters are
built per pixel in src/FitMinuit2.cpp, the only translation unit that
includes Minuit2 headers.
- fit_3d fits through a per-thread PixelFitter so the LM path allocates
nothing per pixel, and no minimizer allocates a per-pixel result array.
- Fixed parameters and parameters ending on a limit report an error of 0
with every minimizer; user start values are applied before the validity
check and clamped into the limits.
- Minuit2's Fumili cannot converge once a two-sided limit is active, so
such pixels are refitted with Migrad. The Minuit2 minimizers keep a wide
two-sided range for one-sided limits because Minuit2's one-sided
transform converges markedly slower.
- The [fit] tests run every scenario through all three minimizers and
check that they agree; the benchmark gains LM and data-cube cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Removed faulty move (pointed out by: #227)
- Fixed license for the queue
- Runtime check for capacity=>2
Thank you @kferjaoui for finding the issue. In the end I found it better
to remove the operators rather than modifying the queue
Cleaned up the helper functions by:
- Added template for make_view (and new make_const_view) instead of the
copy paste implementation
- checks c_style and dimensions inside make_view
- test binding to be able to probe the checks in make_view
Call sites can be reviewed in a second PR since usage of const or
strictness of conversion depends...
We have no code in aare to deal with RawFiles that have missing packets
if they are not padded to full size or frames with packet loss dropped.
Added:
- check that we don't try to read files that are not supported
- python bindings for frame discard policy
at least we get the check in:
closes #171
---------
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- error upon warning when building documentation (warnings often
indicate malrendered documentation)
- fixed some warnings
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- seek to the edge of a file + read_into would select the wrong file
- Visible for example in f.read_frame(index) in python
Added fix, release notes and a small test.
closes #371
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- renamed pedestal.hpp to bind_Pedestal.hpp for python bindings
- variance is now always calculated in double and made private
- cost correctness in a few places
- removed push_fast and explicit updates of mean from Pedestal. If
performance is needed use FastPedestal
- Pedestal no longer caches std
Improved API for ClusterFile
- read_frame now returns a std::optional<ClusterVector> to differentiate
between an empty frame and end of file
- f.frames() produces an iterator over frames
- f.chunks() produces an iterator reading chunks of clusters
- python string repr for Cluster and Eta
- Context manager for RawMasterFile
- slice operator for ROI
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- Updated error message to mention out of range and which file it
applies to
- If there are different number of frames in the subfiles, take the
minimum and print a warning
Closes: #363
Closes: #362
- fixed test in ClusterFinderMT (use minimum one pedestal to accumulate
dummy pedestal)
- added minimum_pedestal_samples as additional constructor argument in
ClusterFinder and ClusterFinderMT
- bugfix: race conditions for empty queues in collect() - check directly
when needed
- Default `ClusterVector` move operations simplifying the code and
fixing a bug.
- Fixed inconsistent frame number type (uint64/int32)
- Validate Python masks as one-dimensional, C-contiguous Boolean arrays,
handle empty masks safely, and reserve filtered storage based on the
selected cluster count.
- Align the C++ and Python API documentation with the implementation,
including concise `hitmap` and reduction documentation and a correctly
rendered constructor example.
- handles file reading of disabled udp ports
- adds members disabled_udp_ports in master file as an optional
- adds member diasbled_udp_port_types in master file as an optional
- Treats enabled udp ports as ROIs
- merges/combines consecutive ROis into one
### To discuss:
- right now can only handle either ROI or disabled udp ports (disabled
udp ports has higher precedence)
- Should Frames in File be removed from Master File as value is nonsense
for disabled udp ports
---------
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder.
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro
**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state
**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)
**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)
**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
Minor API change:
- NDView<T> implicitly converts to NDView<const T> (as std::span in
C++20)
- expand4to8bit and expand24to32bit now takes a view over const uint8_t
to show that the input is not changes
closes #291
Before v8.0 we should always try to read analog samples. This bug
resulted in old Mönch03 data not being reordered.
1. Added test for master file parsing
2. Fixed code
closing #348
- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
Erik Frojdh
