Every bundled model now declares the parameters it is linear in
(linear_par, basis_and_grad) and, except the polynomials, a vectorisable
bulk basis pass (basis_columns) built on the new model::fast_exp. The
Levenberg-Marquardt iteration is factored into a shared DampedGaussNewton
driver with unchanged results; VarPro solves the linear parameters exactly
at every trial point and iterates over the nonlinear ones, rejecting
degenerate trial points and polishing the converged point with one
undamped step. Limits on linear parameters, pixels that do not converge
and models without the separable structure fall back to
LevenbergMarquardt. Parameter errors are the same Gauss-Newton estimates.
The [fit] tests run VarPro through every configuration and check the
separable structure of each model against its derivatives.
Measured on an M1 Max, single thread: FallingScurve population 5.5 us per
fit against 11.7 us with LevenbergMarquardt, and 2 wrong minima in 4096
pixels against 30; Gaussian 3.3 vs 4.3 us; GaussianChargeSharingKb 33 vs
49 us; Pol2 0.34 vs 1.52 us. Migrad stays the default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Introduce a standard way to bind numpy arrays: py::array_t<T> and
.noconvert() on the argument
- Take py::array_t by value since it's just a small wrapper
- Update bindings
Bring the dependency-free Levenberg-Marquardt solver from dev/custom-fit in
as Minimizer::LevenbergMarquardt next to Minuit2's Migrad (default) and
Fumili, selected through the FitModel constructor, SetMinimizer() or the
Python `minimizer` argument and property.
- FitModel becomes plain data (limits, fixed flags, start values, minimizer
settings) with index and limit validation. Minuit2 user parameters are
built per pixel in src/FitMinuit2.cpp, the only translation unit that
includes Minuit2 headers.
- fit_3d fits through a per-thread PixelFitter so the LM path allocates
nothing per pixel, and no minimizer allocates a per-pixel result array.
- Fixed parameters and parameters ending on a limit report an error of 0
with every minimizer; user start values are applied before the validity
check and clamped into the limits.
- Minuit2's Fumili cannot converge once a two-sided limit is active, so
such pixels are refitted with Migrad. The Minuit2 minimizers keep a wide
two-sided range for one-sided limits because Minuit2's one-sided
transform converges markedly slower.
- The [fit] tests run every scenario through all three minimizers and
check that they agree; the benchmark gains LM and data-cube cases.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
We have no code in aare to deal with RawFiles that have missing packets
if they are not padded to full size or frames with packet loss dropped.
Added:
- check that we don't try to read files that are not supported
- python bindings for frame discard policy
at least we get the check in:
closes #171
---------
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- error upon warning when building documentation (warnings often
indicate malrendered documentation)
- fixed some warnings
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- seek to the edge of a file + read_into would select the wrong file
- Visible for example in f.read_frame(index) in python
Added fix, release notes and a small test.
closes #371
Co-authored-by: AliceMazzoleni99 <alice.mazzoleni@psi.ch>
- renamed pedestal.hpp to bind_Pedestal.hpp for python bindings
- variance is now always calculated in double and made private
- cost correctness in a few places
- removed push_fast and explicit updates of mean from Pedestal. If
performance is needed use FastPedestal
- Pedestal no longer caches std
Improved API for ClusterFile
- read_frame now returns a std::optional<ClusterVector> to differentiate
between an empty frame and end of file
- f.frames() produces an iterator over frames
- f.chunks() produces an iterator reading chunks of clusters
- Updated error message to mention out of range and which file it
applies to
- If there are different number of frames in the subfiles, take the
minimum and print a warning
Closes: #363
Closes: #362
- Default `ClusterVector` move operations simplifying the code and
fixing a bug.
- Fixed inconsistent frame number type (uint64/int32)
- Validate Python masks as one-dimensional, C-contiguous Boolean arrays,
handle empty masks safely, and reserve filtered storage based on the
selected cluster count.
- Align the C++ and Python API documentation with the implementation,
including concise `hitmap` and reduction documentation and a correctly
rendered constructor example.
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder.
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro
**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state
**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)
**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)
**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
Multi threaded filling of per pixel histograms for example for detector calibration
1. PixelHistogram - Generic variant expects already pedestal subtracted
data
2. PedestalTrackingHistogram - Terrible name, useful class. Keeps it's
own pedestal and does conversion and pedestal tracking in the worker
threads.
---------
Co-authored-by: Lars Erik Fröjd <froejdh_e@pc-jungfrau-02.psi.ch>
To improve codebase quality and reduce human error, this PR introduces
the pre-commit framework. This ensures that all code adheres to project
standards before it is even committed, maintaining a consistent style
and catching common mistakes early.
Key Changes:
- Code Formatting: Automated C++ formatting using clang-format (based on
the project's .clang-format file).
- Syntax Validation: Basic checks for file integrity and syntax.
- Spell Check: Automated scanning for typos in source code and comments.
- CMake Formatting: Standardization of CMakeLists.txt and .cmake
configuration files.
- GitHub Workflow: Added a CI action that validates every Pull Request
against the pre-commit configuration to ensure compliance.
The configuration includes a [ci] block to handle automated fixes within
the PR. Currently, this is disabled. If we want the CI to automatically
commit formatting fixes back to the PR branch, this can be toggled to
true in .pre-commit-config.yaml.
```yaml
ci:
autofix_commit_msg: [pre-commit] auto fixes from pre-commit hooks
autofix_prs: false
autoupdate_schedule: monthly
```
The last large commit with the fit functions, for example, was not
formatted according to the clang-format rules. This PR would allow to
avoid similar mistakes in the future.
Python fomat with `ruff` for tests and sanitiser for `.ipynb` notebooks
can be added as well.
This PR adds support for decoding digital data from the my320 test chip.
- Added BitOffset (strong type)
- Expand 24 to 32 bit
- Python bindings for decoding my302
- Improved docs
- added rosenblatttransform
- added 3x3 eta methods
- interpolation can be used with various eta functions
- added documentation for interpolation, eta calculation
- exposed full eta struct in python
- disable ClusterFinder for 2x2 clusters
- factory function for ClusterVector
---------
Co-authored-by: Dhanya Thattil <dhanya.thattil@psi.ch>
Co-authored-by: Erik Fröjdh <erik.frojdh@psi.ch>
- No changes or evaluation of existing tests
- Tags for including tests that require data is changed to
**[.with-data]** and **--with-data** for C++ and python respectively
- Minor update to docs
- Added missing files to the test data repo
Added a section on the ideas behind the library and also explaining the
branching strategy.
---------
Co-authored-by: Dhanya Thattil <dhanya.thattil@psi.ch>
maliakal_d
lunin_l
