Prototype multi threaded file reader (#343)
Build on RHEL9 / build (push) Successful in 2m40s
Build on RHEL8 / build (push) Successful in 3m4s
Run tests using data on local RHEL8 / build (push) Successful in 3m51s
Build on local RHEL8 / build (push) Successful in 2m46s

- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
This commit is contained in:
Erik Fröjdh
2026-08-11 15:28:54 +02:00
committed by GitHub
parent 533ecb8a4a
commit e26db97b5c
26 changed files with 1021 additions and 15 deletions
@@ -0,0 +1,30 @@
MultiThreadedFileReader
=======================
.. py:currentmodule:: aare.experimental
The reader returns a NumPy array with shape ``(frames, rows, cols)`` and
preserves the source pixel dtype. Each iteration reads at most
``n_threads * chunk_size`` frames—one chunk per worker. File I/O runs with the
Python GIL released.
.. code-block:: python
from aare.experimental import MultiThreadedFileReader
with MultiThreadedFileReader(
"frames.npy", n_threads=4, chunk_size=128, total_frames=10_000
) as reader:
for frames in reader:
process(frames)
Call ``read()`` directly for the next batch, or ``read_all()`` for all frames
remaining from the current position. ``tell()`` and ``seek()`` expose the
iteration position. The context manager closes all worker files on exit.
``close()`` is also available for explicit cleanup and may be called
repeatedly.
.. autoclass:: MultiThreadedFileReader
:members:
:undoc-members:
:show-inheritance:
+10
View File
@@ -0,0 +1,10 @@
Experimental
============
APIs in this module may change without notice.
.. toctree::
:caption: Experimental
:maxdepth: 1
MultiThreadedFileReader