mirror of
https://github.com/slsdetectorgroup/aare.git
synced 2026-11-01 22:41:28 +01:00
FastPedestal + optimizations in the cluster finder (#341)
Optimized pedestal tracking with FastPedestal and performance improvements to the cluster finder. - ~2x faster hitting 14k FPS on benchmark dataset on 16 core threadripper pro **Changes** - No bounds check on interior of frame. (only within half of a cluster) (~15% improvement) - Cache threshold (std*n_rms) Biggest improvement comes from not computing std for each access - Cache pedestal subtracted frame - Switched to FastPedestal which assumes we reached steady state **Things tried but rejected:** - Always store cluster values, commit on val==max (~15% drop in frame rate) **Options** - using 16 bit clusters and 16 bit pedestal. (slows down the single threaded case but allow us to reach 12k with multi threading) **Other notes on performance** - Passing in memory frames from python and doing nothing: 0.8us/frame - Passing + pedestal subtraction: 28us/frame - Passing + pedestal + cluster finding (no store): 652us/frame - Passing + pedestal + cluster finding: 886 us/frame
This commit is contained in:
1 parent
568f94926b
commit
eddb919328
44 files changed
+2220
-320
No files matched your search
@@ -15,6 +15,14 @@ Use ``push_pedestal_no_update()`` to seed the pedestal estimate, then
|
||||
asynchronous fills are drained by ``flush()``, and snapshot methods such as
|
||||
``values()`` and ``pedestal_mean()`` return numpy arrays.
|
||||
|
||||
``fill_from_file()`` uses parallel file-reader workers and a double-buffered
|
||||
pipeline. After the initial batch has been read, histogram processing of one
|
||||
batch overlaps reading of the next. ``reader_threads`` and
|
||||
``reader_chunk_size`` tune the I/O stage independently of the histogram worker
|
||||
count. Two fixed-capacity buffers are allocated once and reused by alternating
|
||||
their read and histogram roles. Their approximate memory use is ``2 *
|
||||
reader_threads * reader_chunk_size * rows * cols * sizeof(uint16)``.
|
||||
|
||||
.. py:currentmodule:: aare
|
||||
|
||||
.. autoclass:: PedestalTrackingPixelHistogram
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
Pedestal
|
||||
========
|
||||
|
||||
.. toctree::
|
||||
:caption: Pedestal
|
||||
:maxdepth: 1
|
||||
|
||||
pyFastPedestal
|
||||
pyPedestal
|
||||
@@ -0,0 +1,83 @@
|
||||
FastPedestal
|
||||
============
|
||||
|
||||
``FastPedestal`` calculates a running mean, variance and standard deviation for each pixel in a
|
||||
series of frames. The python binding only exposes ``uint16`` input but the underlying
|
||||
C++ class is templated. Initialize it with ``n_samples`` frames using
|
||||
``add_init_frame()``. Once ``ready`` is true, use ``push_ema()`` to update the exponential
|
||||
moving average initialized by the mean and with smoothing factor 1/n_samples.
|
||||
|
||||
.. warning::
|
||||
|
||||
FastPedestal is not usable until you have added ``n_samples`` initial frames with ``add_init_frame(raw)``.
|
||||
You can check the state with ``ready``.
|
||||
|
||||
The public factory selects the bound C++ specialization from ``dtype``:
|
||||
|
||||
* ``numpy.float64`` creates ``FastPedestal_d``
|
||||
* ``numpy.float32`` creates ``FastPedestal_f``
|
||||
* ``numpy.int16`` creates ``FastPedestal_i16``
|
||||
|
||||
The internal calculations are done with double, but the cached mean and on demand var and std are returned in the specified type.
|
||||
|
||||
Factory
|
||||
-------
|
||||
|
||||
.. py:currentmodule:: aare
|
||||
|
||||
.. autofunction:: FastPedestal
|
||||
|
||||
Loading from a file
|
||||
-------------------
|
||||
|
||||
``FastPedestal.from_file()`` initializes the pedestal from ``n_samples``
|
||||
frames after ``skip_first``, then applies steady-state updates for any frames
|
||||
remaining in the file. The input frames must contain ``uint16`` data; ``dtype``
|
||||
selects the output type of the pedestal statistics.
|
||||
|
||||
.. autofunction:: aare.FastPedestal.from_file
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
pedestal = FastPedestal.from_file(
|
||||
"frames.npy", n_samples=100, skip_first=10, dtype=np.float32
|
||||
)
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import numpy as np
|
||||
from aare import FastPedestal
|
||||
|
||||
pedestal = FastPedestal(512, 1024, n_samples=100, dtype=np.float32)
|
||||
|
||||
# Initialize with n_samples frames
|
||||
for frame in initialization_frames:
|
||||
pedestal.add_init_frame(frame)
|
||||
|
||||
# Now we can push a frame for pedestal update
|
||||
if pedestal.ready:
|
||||
pedestal.push_ema(next_frame)
|
||||
|
||||
# Mean and std are also ready
|
||||
mean = pedestal.mean()
|
||||
noise = pedestal.std()
|
||||
|
||||
# Direct pedestal subtraction is also supported
|
||||
for frame in raw_data:
|
||||
image = frame - pedestal
|
||||
|
||||
Complete API
|
||||
------------
|
||||
|
||||
The API below is for the ``float64`` specialization. All dtype variants share
|
||||
the same API.
|
||||
|
||||
.. autoclass:: aare._aare.FastPedestal_d
|
||||
:special-members: __init__
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
:inherited-members:
|
||||
@@ -0,0 +1,42 @@
|
||||
Pedestal
|
||||
========
|
||||
|
||||
``Pedestal`` calculates a running mean and variance for each pixel in a series
|
||||
of ``uint16`` frames. ``push()`` updates the cached mean immediately. For
|
||||
faster batch initialization, use ``push_no_update()`` for each frame and call
|
||||
``update_mean()`` after the batch.
|
||||
|
||||
Three specializations are available from :mod:`aare`:
|
||||
|
||||
* ``Pedestal_d`` uses ``float64`` storage
|
||||
* ``Pedestal_f`` uses ``float32`` storage
|
||||
* ``Pedestal_i16`` uses ``int16`` storage
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from aare import Pedestal_d
|
||||
|
||||
pedestal = Pedestal_d(512, 1024, 100)
|
||||
|
||||
for frame in initialization_frames:
|
||||
pedestal.push_no_update(frame)
|
||||
|
||||
pedestal.update_mean()
|
||||
mean = pedestal.mean()
|
||||
noise = pedestal.std()
|
||||
|
||||
Complete API
|
||||
------------
|
||||
|
||||
The API below is for the ``float64`` specialization. All dtype variants share
|
||||
the same API.
|
||||
|
||||
.. autoclass:: aare._aare.Pedestal_d
|
||||
:special-members: __init__
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
:inherited-members:
|
||||
Reference in new issue
Block a user