Prototype multi threaded file reader (#343)
Build on RHEL9 / build (push) Successful in 2m40s
Build on RHEL8 / build (push) Successful in 3m4s
Run tests using data on local RHEL8 / build (push) Successful in 3m51s
Build on local RHEL8 / build (push) Successful in 2m46s

- Reading files from nfs shares tops out at ~1GB/s for single threaded
reads.
- MultiThreadedFileReader uses our File wrapper to read a generic file
in parallel
- Placed in aare::experimental to show that it's not production ready
This commit is contained in:
Erik Fröjdh authored and GitHub committed 2026-08-11 15:28:54 +02:00
1 parent 533ecb8a4a
commit e26db97b5c
26 files changed
+1021 -15

No files matched your search

+37
View File
@@ -0,0 +1,37 @@
MultiThreadedFileReader
=======================
``MultiThreadedFileReader`` reads one chunk per worker on each call. Each
worker owns an independent :cpp:class:`aare::File`, while all workers write
into non-overlapping regions of the destination buffer. The output is ordered
by frame index and successive calls advance through the file.
.. code-block:: cpp
#include "aare/MultiThreadedFileReader.hpp"
// Four workers, chunks of 128 frames, and at most 10,000 frames.
aare::experimental::MultiThreadedFileReader reader(path, 4, 128, 10'000);
while (reader.remaining_frames() != 0) {
// Contains at most 4 * 128 frames.
auto batch = reader.read();
process(batch);
}
Omit the final argument to read every frame in the source. An explicit value
of zero requests an empty result. The low-level ``read_into`` overload avoids
an allocation when the caller already owns a buffer of at least
``reader.next_read_bytes()`` bytes. Use ``read_all()`` to read every frame
remaining from the current position, and ``seek()`` to reposition the reader.
Call ``close()`` to release all worker file handles early.
.. note::
Multiple workers do not guarantee faster reads. Performance depends on the
storage device and file format, so the thread count and chunk size should be
benchmark-driven.
.. doxygenclass:: aare::experimental::MultiThreadedFileReader
:members:
:undoc-members:
:private-members: