mirror of
https://github.com/slsdetectorgroup/aare.git
synced 2026-10-23 12:51:59 +02:00
Prototype multi threaded file reader (#343)
- Reading files from nfs shares tops out at ~1GB/s for single threaded reads. - MultiThreadedFileReader uses our File wrapper to read a generic file in parallel - Placed in aare::experimental to show that it's not production ready
This commit is contained in:
1 parent
533ecb8a4a
commit
e26db97b5c
26 files changed
+1021
-15
No files matched your search
@@ -0,0 +1,30 @@
|
||||
MultiThreadedFileReader
|
||||
=======================
|
||||
|
||||
.. py:currentmodule:: aare.experimental
|
||||
|
||||
The reader returns a NumPy array with shape ``(frames, rows, cols)`` and
|
||||
preserves the source pixel dtype. Each iteration reads at most
|
||||
``n_threads * chunk_size`` frames—one chunk per worker. File I/O runs with the
|
||||
Python GIL released.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from aare.experimental import MultiThreadedFileReader
|
||||
|
||||
with MultiThreadedFileReader(
|
||||
"frames.npy", n_threads=4, chunk_size=128, total_frames=10_000
|
||||
) as reader:
|
||||
for frames in reader:
|
||||
process(frames)
|
||||
|
||||
Call ``read()`` directly for the next batch, or ``read_all()`` for all frames
|
||||
remaining from the current position. ``tell()`` and ``seek()`` expose the
|
||||
iteration position. The context manager closes all worker files on exit.
|
||||
``close()`` is also available for explicit cleanup and may be called
|
||||
repeatedly.
|
||||
|
||||
.. autoclass:: MultiThreadedFileReader
|
||||
:members:
|
||||
:undoc-members:
|
||||
:show-inheritance:
|
||||
Reference in new issue
Block a user