FastPedestal + optimizations in the cluster finder (#341)
Build on RHEL9 / build (push) Successful in 2m42s
Build on RHEL8 / build (push) Successful in 3m21s
Run tests using data on local RHEL8 / build (push) Failing after 3m56s
Build on local RHEL8 / build (push) Successful in 2m54s

Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder. 
- ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro


**Changes**
- No bounds check on interior of frame. (only within half of a cluster)
(~15% improvement)
- Cache threshold (std*n_rms) Biggest improvement comes from not
computing std for each access
- Cache pedestal subtracted frame
- Switched to FastPedestal which assumes we reached steady state

**Things tried but rejected:**
- Always store cluster values, commit on val==max (~15% drop in frame
rate)

**Options**
- using 16 bit clusters and 16 bit pedestal. (slows down the single
threaded case but allow us to reach 12k with multi threading)

**Other notes on performance**
- Passing in memory frames from python and doing nothing: 0.8us/frame
- Passing + pedestal subtraction: 28us/frame
- Passing + pedestal + cluster finding (no store): 652us/frame
- Passing + pedestal + cluster finding: 886 us/frame
This commit is contained in:
Erik Fröjdh authored and GitHub committed 2026-09-01 18:41:12 +02:00
1 parent 568f94926b
commit eddb919328
44 files changed
+2220 -320

No files matched your search

@@ -15,6 +15,14 @@ Use ``push_pedestal_no_update()`` to seed the pedestal estimate, then
asynchronous fills are drained by ``flush()``, and snapshot methods such as
``values()`` and ``pedestal_mean()`` return numpy arrays.
``fill_from_file()`` uses parallel file-reader workers and a double-buffered
pipeline. After the initial batch has been read, histogram processing of one
batch overlaps reading of the next. ``reader_threads`` and
``reader_chunk_size`` tune the I/O stage independently of the histogram worker
count. Two fixed-capacity buffers are allocated once and reused by alternating
their read and histogram roles. Their approximate memory use is ``2 *
reader_threads * reader_chunk_size * rows * cols * sizeof(uint16)``.
.. py:currentmodule:: aare
.. autoclass:: PedestalTrackingPixelHistogram
+9
View File
@@ -0,0 +1,9 @@
Pedestal
========
.. toctree::
:caption: Pedestal
:maxdepth: 1
pyFastPedestal
pyPedestal
@@ -0,0 +1,83 @@
FastPedestal
============
``FastPedestal`` calculates a running mean, variance and standard deviation for each pixel in a
series of frames. The python binding only exposes ``uint16`` input but the underlying
C++ class is templated. Initialize it with ``n_samples`` frames using
``add_init_frame()``. Once ``ready`` is true, use ``push_ema()`` to update the exponential
moving average initialized by the mean and with smoothing factor 1/n_samples.
.. warning::
FastPedestal is not usable until you have added ``n_samples`` initial frames with ``add_init_frame(raw)``.
You can check the state with ``ready``.
The public factory selects the bound C++ specialization from ``dtype``:
* ``numpy.float64`` creates ``FastPedestal_d``
* ``numpy.float32`` creates ``FastPedestal_f``
* ``numpy.int16`` creates ``FastPedestal_i16``
The internal calculations are done with double, but the cached mean and on demand var and std are returned in the specified type.
Factory
-------
.. py:currentmodule:: aare
.. autofunction:: FastPedestal
Loading from a file
-------------------
``FastPedestal.from_file()`` initializes the pedestal from ``n_samples``
frames after ``skip_first``, then applies steady-state updates for any frames
remaining in the file. The input frames must contain ``uint16`` data; ``dtype``
selects the output type of the pedestal statistics.
.. autofunction:: aare.FastPedestal.from_file
.. code-block:: python
pedestal = FastPedestal.from_file(
"frames.npy", n_samples=100, skip_first=10, dtype=np.float32
)
Example
-------
.. code-block:: python
import numpy as np
from aare import FastPedestal
pedestal = FastPedestal(512, 1024, n_samples=100, dtype=np.float32)
# Initialize with n_samples frames
for frame in initialization_frames:
pedestal.add_init_frame(frame)
# Now we can push a frame for pedestal update
if pedestal.ready:
pedestal.push_ema(next_frame)
# Mean and std are also ready
mean = pedestal.mean()
noise = pedestal.std()
# Direct pedestal subtraction is also supported
for frame in raw_data:
image = frame - pedestal
Complete API
------------
The API below is for the ``float64`` specialization. All dtype variants share
the same API.
.. autoclass:: aare._aare.FastPedestal_d
:special-members: __init__
:members:
:undoc-members:
:show-inheritance:
:inherited-members:
+42
View File
@@ -0,0 +1,42 @@
Pedestal
========
``Pedestal`` calculates a running mean and variance for each pixel in a series
of ``uint16`` frames. ``push()`` updates the cached mean immediately. For
faster batch initialization, use ``push_no_update()`` for each frame and call
``update_mean()`` after the batch.
Three specializations are available from :mod:`aare`:
* ``Pedestal_d`` uses ``float64`` storage
* ``Pedestal_f`` uses ``float32`` storage
* ``Pedestal_i16`` uses ``int16`` storage
Example
-------
.. code-block:: python
from aare import Pedestal_d
pedestal = Pedestal_d(512, 1024, 100)
for frame in initialization_frames:
pedestal.push_no_update(frame)
pedestal.update_mean()
mean = pedestal.mean()
noise = pedestal.std()
Complete API
------------
The API below is for the ``float64`` specialization. All dtype variants share
the same API.
.. autoclass:: aare._aare.Pedestal_d
:special-members: __init__
:members:
:undoc-members:
:show-inheritance:
:inherited-members: