Merge branch 'main' into testing/cuda_cf_api

Three files conflicted:

- ClusterFinderMT::collect(): both sides rewrote it. Keep this branch's
  version (f4bbbc9), which reads the stop flags before sweeping, over
  main's (afce91b), which still samples queue emptiness mid-condition.
  The staged splice had also lost a brace, swallowing popFront().
- python/aare/ClusterFinder.py: restore the CUDA API deleted by the
  resolution (_cuda_available, _CUDA_ALGORITHMS, ClusterFinderCUDA,
  find_cluster_views_batched_iter, ClusterFinderCUDAGraph), and main's
  min_pedestal_samples forwarding, dropped as an adjacent hunk.
- bind_ClusterVector.hpp: keep py::module_local() (133cedf) alongside
  main's new docstrings; without it _aare and _aare_cuda cannot both
  register Cluster/ClusterVector under pybind11 3.

Does not build on its own: ClusterFinderFrozen and the CUDA drivers
still call Pedestal APIs main removed. Fixed in the follow-up.

docs/src/python/cluster/pyClusterVector.rst

@@ -3,17 +3,16 @@
 ClusterVector
 ================
 
-The ClusterVector, holds clusters from the ClusterFinder. Since it is templated
-in C++  we use a suffix indicating the type of cluster it holds. The suffix follows
-the same pattern as for ClusterFile i.e. ``ClusterVector_Cluster3x3i``
-for a vector holding 3x3 integer clusters.
+A ClusterVector stores fixed-size clusters contiguously. Since it is templated
+in C++, each bound class has a suffix indicating the cluster type. The suffix
+follows the same pattern as ClusterFile; for example,
+``ClusterVector_Cluster3x3i`` stores 3x3 clusters with 32-bit integer pixels.
 
 
-At the moment the functionality from python is limited and it is not supported
-to push_back clusters to the vector. The intended use case is to pass it to 
-C++ functions that support the ClusterVector or to view it as a numpy array.
+The intended use case is to pass a ClusterVector to C++ functions that support
+it or to view it as a NumPy array.
 
-**View ClusterVector as numpy array**
+**View ClusterVector as a NumPy array**
 
 .. code:: python
 
@@ -25,7 +24,16 @@ C++ functions that support the ClusterVector or to view it as a numpy array.
     clusters = np.array(cluster_vector)
 
     # Avoid copying the data by passing copy=False
-    clusters = np.array(cluster_vector, copy = False)
+    clusters = np.array(cluster_vector, copy=False)
+
+.. warning::
+
+   A NumPy array created with ``copy=False`` is a view of the ClusterVector's
+   current storage. Do not call ``push_back`` or otherwise change the
+   ClusterVector while using the view. A ``push_back`` that reallocates the
+   backing buffer leaves existing NumPy views pointing to invalid memory, and
+   appending without reallocation does not update their shape. Use
+   ``copy=True`` if the ClusterVector may change after creating the array.
 
 
 .. py:currentmodule:: aare
@@ -35,7 +43,8 @@ C++ functions that support the ClusterVector or to view it as a numpy array.
     :undoc-members:
     :inherited-members:
 
-Below is the API of the ClusterVector_Cluster3x3i but all variants share the same API.
+Below is the API of ``ClusterVector_Cluster3x3i``. All variants share the same
+API.
 
 .. autoclass::  aare._aare.ClusterVector_Cluster3x3i
     :special-members: __init__, __call__ 
@@ -45,14 +54,39 @@ Below is the API of the ClusterVector_Cluster3x3i but all variants share the sam
     :inherited-members:
 
 
-**Free Functions:** 
+**Free Functions:**
 
-.. autofunction:: reduce_to_3x3
+.. py:function:: hitmap(image_size, clusters)
    :noindex:
 
-   Reduce a single Cluster to 3x3 by taking the 3x3 subcluster with highest photon energy.
+   Count cluster centers into an ``int32`` image. ``image_size`` is given as
+   ``(rows, columns)``, and output element ``[y, x]`` contains the number of
+   photon hits at that coordinate. Out-of-bounds hits are ignored. All registered
+   ClusterVector variants are accepted.
 
-.. autofunction:: reduce_to_2x2
+   :param tuple[int, int] image_size: Shape of the output image.
+   :param ClusterVector clusters: Clusters whose centers are counted.
+   :return: Hit counts with shape ``image_size``.
+   :rtype: numpy.ndarray
+
+.. py:function:: reduce_to_3x3(clustervector)
    :noindex:
 
-   Reduce a single Cluster to 2x2 by taking the 2x2 subcluster with highest photon energy.
+   Return a new vector containing the central 3x3 block of every input cluster.
+   Cluster order, coordinates, frame number, and pixel dtype are preserved.
+
+   :param ClusterVector clustervector: Input clusters with both dimensions at
+      least 3 and at least one dimension greater than 3.
+   :return: Reduced 3x3 clusters.
+   :rtype: ClusterVector
+
+.. py:function:: reduce_to_2x2(clustervector)
+   :noindex:
+
+   Return a new vector containing the highest-sum center-adjacent 2x2 block of
+   every input cluster. Cluster order, coordinates, frame number, and pixel
+   dtype are preserved.
+
+   :param ClusterVector clustervector: Input clusters of size 2x2 or larger.
+   :return: Reduced 2x2 clusters.
+   :rtype: ClusterVector

docs/src/python/file/pyClusterFile.rst

@@ -1,26 +1,78 @@
 
 ClusterFile
-============
+===========
 
+The :func:`ClusterFile` factory is the main interface for reading and writing
+cluster files. Use ``mode="r"`` to read, ``mode="w"`` to truncate and
+write, or ``mode="a"`` to append.
 
-The :class:`ClusterFile` class is the main interface to read and write clusters in aare. Unfortunately the
-old file format does not include metadata like the cluster size and the data type. This means that the
-user has to know this information from other sources. Specifying the wrong cluster size or data type
-will lead to garbage data being read. 
+The format does not store the cluster dimensions, value type, coordinate type,
+padding, or byte order. The ``cluster_size`` and ``dtype`` arguments must match
+the writer; otherwise the bytes are interpreted incorrectly.
+
+Use ``frames()`` when frame boundaries and the frame number matter:
+
+.. code-block:: python
+
+    with ClusterFile("clusters.clust") as file:
+        for frame in file.frames():
+            print(frame.frame_number, frame.size)
+
+Each stored frame is yielded, including empty frames and frames whose clusters
+are all rejected by filtering. Missing frame numbers are not synthesized.
+The explicit ``read_frame()`` method returns one frame or ``None`` at EOF.
+
+Use ``chunks()``  to iterate over chunks of ``chunk_size`` clusters. Per default 
+the chunk size defined in the constructor is used. Alternatively one can pass 
+an explicit chunk size to the function for that specific traversal:
+
+.. code-block:: python
+
+    with ClusterFile("clusters.clust", chunk_size=1000) as file:
+        for clusters in file.chunks(10000):
+            process(clusters)
+
+Chunk sizes must be positive and count selected clusters after filtering.
+``file.chunks(10000)`` does not overwrite the constructor argument  ``chunk_size``
+. Iterating directly
+over ``file`` or calling ``next(file)`` still reads chunks using that default.
+Chunks and ``read_clusters(n_clusters)`` may split or combine frames, so their
+frame numbers are not reliable per-cluster metadata. Empty chunks are not
+yielded.
+Chunk reads and iteration raise an error if they encounter an incomplete frame
+header or cluster record, including when ROI or noise filtering is enabled.
+They return fewer clusters than requested only at a clean end of file.
+
+Both iterators consume the current file position without rewinding. Creating
+an iterator does not read; each ``next()`` reads one result without prefetching
+the following result. Use one traversal at a time and separate readers for
+independent cursors. Switching to frames after a chunk stops partway through a
+frame raises an error until the remaining clusters in that frame have been
+read. Exhausted iterators continue to raise ``StopIteration``.
+
+Iterators keep their file object alive, but explicitly closing the file or
+leaving its ``with`` block prevents further reads. Every yielded
+``ClusterVector`` owns its storage and can be retained after iteration advances
+or the file closes. NumPy views also remain valid across those operations;
+as usual, resizing or modifying the allocation of their backing vector can
+invalidate them.
+
+When a gain map is configured, it is applied to every cluster whose complete
+footprint lies inside the map. Clusters whose footprint extends beyond the
+gain-map boundaries remain in the returned cluster vector, but all their data
+values are set to zero.
 
 .. py:currentmodule:: aare
 
-.. autoclass:: ClusterFile
-    :members:
-    :undoc-members:
-    :inherited-members:
+.. autofunction:: ClusterFile
 
 
-Below is the API of the ClusterFile_Cluster3x3i but all variants share the same API.
+Below is the API of ``ClusterFile_Cluster3x3i``; all compiled variants share the
+same API.
 
 .. autoclass:: aare._aare.ClusterFile_Cluster3x3i
     :special-members: __init__
     :members:
     :undoc-members:
     :show-inheritance:
-    :inherited-members:
+    :inherited-members:

docs/src/python/file/pyCtbRawFile.rst

@@ -6,6 +6,10 @@ Read analog, digital and transceiver samples from a raw file containing
 data from the Chip Test Board. Uses :mod:`aare.transform` to decode the 
 data into a format that the user can work with. 
 
+``read_frame(index)`` reads the frame at the zero-based index across the raw
+subfiles and advances the position to ``index + 1``. A subsequent
+``read_frame()`` reads the following frame, including across subfile boundaries.
+
 .. code:: python
 
     import aare
@@ -22,4 +26,4 @@ data into a format that the user can work with.
     :members:
     :undoc-members:
     :show-inheritance:
-    :inherited-members:
+    :inherited-members:

docs/src/python/file/pyRawFile.rst

@@ -3,8 +3,45 @@ RawFile
 
 .. py:currentmodule:: aare
 
+Reading requires frame padding to be enabled 
+or ``Frame Discard Policy`` to be ``discardpartial``.  Construction raises
+``RuntimeError`` with the master path before opening data subfiles if this
+requirement is not met. This applies to JSON and legacy ``.raw`` masters,
+including when opened through ``File``. ``RawMasterFile`` can still be used
+to inspect their metadata.
+
+``total_frames`` and ``len(reader)`` report the minimum actual frame count
+across all selected subfiles and ROIs, including each subfile's complete
+``.raw`` series. ``read()`` uses this count, and ``read_n()`` and
+``read_n_with_roi()`` limit batches to the remaining frames. Counts are
+determined when the reader is constructed.
+An empty subfile makes the count zero, and construction fails if no subfiles
+are selected.
+
+The C++ logger prints one warning to stderr on opening if the subfile counts
+differ or their minimum differs from ``reader.master.frames_in_file``. The
+warning includes the count range, the recorded master count, and the count
+used for reading. Master metadata is preserved; its expected frame count
+does not affect reading bounds. Detector frame-number mismatches can still
+cause synchronization errors within these bounds.
+
+Reading errors raise ``RuntimeError`` with the attempted zero-based frame index
+and the failing file path. If a data subfile fails, its path and the underlying
+failure reason are reported. Out-of-range subfile errors include the available frame count across
+that subfile's series. This applies to ``read_frame()``, ``read_n()``,
+and the ROI reading methods. The frame index is the position in the file, not
+the frame number recorded by the detector.
+
+Synchronization errors also identify the last ``.raw`` data file read for the
+module that ran out of frames. An out-of-range request rejected before accessing
+a subfile reports the master path.
+
+For a file containing five frames, the valid indices are 0 through 4. Attempting
+to read index 5 raises an error stating: ``Frame index 5 is out of range: file
+contains 5 frames (indices are zero-based)``.
+
 .. autoclass:: RawFile
     :members:
     :undoc-members:
     :show-inheritance:
-    :inherited-members:
+    :inherited-members:

docs/src/python/pedestal/pyFastPedestal.rst

@@ -1,7 +1,7 @@
 FastPedestal
 ============
 
-``FastPedestal`` calculates a running mean, variance and standard deviation for each pixel in a
+``FastPedestal`` calculates a running mean and standard deviation for each pixel in a
 series of frames. The python binding only exposes ``uint16`` input but the underlying
 C++ class is templated. Initialize it with ``n_samples`` frames using
 ``add_init_frame()``. Once ``ready`` is true, use ``push_ema()`` to update the exponential 
@@ -18,7 +18,10 @@ The public factory selects the bound C++ specialization from ``dtype``:
 * ``numpy.float32`` creates ``FastPedestal_f``
 * ``numpy.int16`` creates ``FastPedestal_i16``
 
-The internal calculations are done with double, but the cached mean and on demand var and std are returned in the specified type.
+Internal moments and variance are calculated in double precision. Variance is
+private and stays in double precision through the square root; the cached mean
+and on-demand standard deviation are returned in the specified type. Negative
+variance caused by floating-point roundoff is clamped to zero.
 
 Factory
 -------

docs/src/python/pedestal/pyPedestal.rst

@@ -1,30 +1,57 @@
 Pedestal
 ========
 
-``Pedestal`` calculates a running mean and variance for each pixel in a series
-of ``uint16`` frames. ``push()`` updates the cached mean immediately. For
-faster batch initialization, use ``push_no_update()`` for each frame and call
-``update_mean()`` after the batch.
+``Pedestal`` calculates a running mean and population standard deviation for
+each pixel in a series of ``uint16`` frames. ``push()`` updates the cached mean
+immediately; ``std()`` calculates the noise from the current statistics.
 
-Three specializations are available from :mod:`aare`:
+``push()`` and ``push_with_threshold()`` require C-contiguous, two-dimensional
+NumPy frames with dtype ``uint16`` and shape matching the pedestal.
+``push_with_threshold()`` also requires a C-contiguous, two-dimensional
+threshold array with the pedestal's output dtype and shape. Noncontiguous
+inputs raise ``TypeError``; inputs with the wrong number of dimensions raise
+``ValueError``. Mismatched frame or threshold shapes raise ``RuntimeError``.
+Use ``numpy.ascontiguousarray()`` to copy a sliced or transposed array into the
+required layout when needed.
 
-* ``Pedestal_d`` uses ``float64`` storage
-* ``Pedestal_f`` uses ``float32`` storage
-* ``Pedestal_i16`` uses ``int16`` storage
+Internal sums and sums of squares always use ``float64``. Three
+specializations are available from :mod:`aare` for the mean and standard
+deviation output types:
+
+* ``Pedestal_d`` returns ``float64``
+* ``Pedestal_f`` returns ``float32``
+* ``Pedestal_i16`` returns ``int16``
+
+The public ``Pedestal`` factory selects the specialization from ``dtype``,
+defaulting to ``numpy.float64``.
+
+Constructor dimensions and ``n_samples`` must be positive integers. Negative
+values raise ``TypeError`` and zero values raise ``RuntimeError``.
+Internally, negative variance caused by floating-point roundoff is clamped to
+zero before taking its square root, keeping the standard deviation finite for
+nearly constant inputs. Only the final standard deviation is converted to the
+output dtype.
+
+Factory
+-------
+
+.. py:currentmodule:: aare
+
+.. autofunction:: Pedestal
 
 Example
 -------
 
 .. code-block:: python
 
-   from aare import Pedestal_d
+   import numpy as np
+   from aare import Pedestal
 
-   pedestal = Pedestal_d(512, 1024, 100)
+   pedestal = Pedestal(512, 1024, n_samples=100, dtype=np.float32)
 
    for frame in initialization_frames:
-       pedestal.push_no_update(frame)
+       pedestal.push(frame)
 
-   pedestal.update_mean()
    mean = pedestal.mean()
    noise = pedestal.std()