aare/RELEASE.md

Release notes

Next

New Features:

• Added the Python xy type with row and col fields, enabling access to RawFile.geometry, RawMasterFile.detector_layout, and RawMasterFile.udp_interfaces_per_module. It unpacks as row, col.
• Added Windows wheels (win_amd64, Python 3.12 to 3.14) to the wheel build, so releases publish them to PyPI alongside the Linux and macOS wheels.

Build:

• libzmq is only fetched or searched for with -DAARE_ZMQ=ON (default OFF) and is then available as aare::zmq. Nothing in aare links it yet, and skipping it removes about 8 s from every CMake configure.
• Minuit2, Catch2, and libzmq when enabled, are compiled as unity builds (about 4-5x less compile time), and the fetched dependencies are shallow clones, which makes a first configure faster and the build directory smaller.
• The Python extension is compiled from one translation unit per binding area instead of a single file, so the bindings build in parallel and editing one binding header recompiles only its unit. The module's contents and registration order are unchanged.
• The Python extension is no longer link-time optimized by default. The hot loops are header templates compiled at -O3 inside the extension and LTO cannot reach the static aare_core, so it only added a 20 s single-threaded link and made the binary 15 % smaller. LTO follows the standard CMake property now: set_property(TARGET _aare PROPERTY INTERPROCEDURAL_OPTIMIZATION ON) or -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=ON restores it.

2026.9.30

New Features:

• Added pedestal_ready to ClusterFinder and ClusterFinderMT in C++ and Python to check whether enough pedestal frames have been pushed to find clusters.
• Added the Python FrameDiscardPolicy enum with NoDiscard, Discard, and DiscardPartial, enabling access to RawMasterFile.frame_discard_policy.
• Added the Python Pedestal factory with dtype selection, matching FastPedestal and defaulting to float64 output.
• Added FastPedestal in C++ and Python for per-pixel running mean and population standard deviation. It supports exponentially
• Added ClusterFile.frames() and ClusterFile.chunks() in C++ and Python for iteration from the current file position. Frames preserve empty frames and frame numbers; chunks support an optional positive size override. Python's default file iteration continues to yield chunks.
• Added FastPedestal in C++ and Python for per-pixel running mean, population variance, and standard deviation. It supports exponentially weighted updates, initialization from files, direct subtraction from NumPy arrays, and float64, float32, and int16 output types.
• Added the Pedestal_i16 Python binding alongside FastPedestal_d, FastPedestal_f, and FastPedestal_i16.
• PedestalTrackingPixelHistogram.fill_from_file() now uses parallel, double-buffered file reading and accepts reader_threads and reader_chunk_size tuning parameters.
• Added the AARE_TUNE_LOCAL CMake option to build with -march=native and -mtune=native when supported. Binaries built with this option are specific to the local CPU and may not be portable.
• Added string representator in python for Cluster and Eta
• Added roi slice method in python for easy slicing of numpy arrays array[roi.slice()].
• added context manager for aare.RawMasterFile
• Added a configurable minimizer for the fit models, selected with the minimizer constructor argument or property in Python, or FitModel::SetMinimizer in C++. Minimizer.Migrad remains the default. Minimizer.Fumili is Minuit2's Gauss-Newton minimizer, which uses the analytic model derivatives and needs far fewer function evaluations. Minimizer.LevenbergMarquardt is a built-in, dependency-free Levenberg-Marquardt solver with the analytic Jacobian; it reflects steps at parameter limits and fits data cubes without per-pixel allocations. A fit that stalls away from a minimum, or whose width collapses below the spacing of the scan points, is reported as failed (all zeros) rather than returned as a result, so noisy or signal-free pixels may fail. With compute_errors, Fumili and LevenbergMarquardt report parameter errors from their linearised covariance instead of running Hesse. Pixels for which Fumili does not reach a valid minimum, which Minuit2's implementation cannot once a two-sided limit becomes active, are refitted with Migrad.
• Minimizer.LevenbergMarquardt is about twice as fast for every model: it evaluates the Jacobian at the trial point instead of re-evaluating accepted points, accumulates the normal equations with fixed-size loops, needs a single factorisation per iteration, and the erf-based models compute one exponential per point instead of two. Results are unchanged, except that GaussianChargeSharing and GaussianChargeSharingKb now evaluate the erf from the Gaussian's exponential, a rounding-level change. One iteration costs one model evaluation of the max_calls budget, so pixels that used to exhaust the default budget of 100 may now converge.
• Added Minimizer.VarPro, a built-in variable projection solver. Every bundled model now declares the parameters it is linear in (linear_par and basis_and_grad in C++, checked against the model derivatives by the tests). Each trial point solves those exactly and the Levenberg-Marquardt iteration runs over the nonlinear parameters only, so the fit needs fewer evaluations and does not depend on start values of the linear parameters. When the basis functions are nearly collinear, as for a polynomial on a narrow range far from x = 0, one more pass over the data refines the linear solution to the precision of the other minimizers. Limits on linear parameters, pixels that do not converge and models without the separable structure fall back to LevenbergMarquardt. Parameter errors are the same Gauss-Newton estimates. The Gaussian, plateau, charge-sharing and S-curve models also provide their basis functions for all scan points at once (basis_columns), a loop that compiles to vector instructions with the new model::fast_exp; eval and eval_and_grad, and with them the other minimizers, are unchanged. Minimizer.Migrad remains the default.
• NDArray/NDView expressions now support scalar operands (2 * a + b / 4) and can be assigned to an existing NDArray, reusing its buffer.

API Changes:

• The C++ NDArray view constructor and NDArray::copy_from() take NDView<const T, Ndim>. Mutable views still convert implicitly, so existing callers are unaffected, and read-only views can now be copied.

• Added C++ overloads of adc_sar_05_06_07_08decode64to16, adc_sar_05_decode64to16 and adc_sar_04_decode64to16 that take the packed samples as NDView<const uint8_t, 2>. Words are assembled with memcpy, so the buffer does not need to be 8-byte aligned. The existing NDView<uint64_t, 2> overloads are unchanged.

• NDArray + - * / with a scalar now returns a lazy expression instead of an NDArray and no longer converts the scalar to the element type.

• FitModel is now plain data without a pimpl (C++ users need to rebuild); it gained strategy(), lower_limit(), upper_limit() and value() accessors and lost impl(). Bad parameter indices raise std::out_of_range (IndexError in Python) and SetParLimits rejects lo >= hi. The models in Models.hpp no longer provide the Minuit specific compute_steps and compute_ranges helpers.

• Fit behaviour common to all minimizers: user start and fixed values are applied before the validity check and free start values are clamped into their limits; one-sided limits are open-ended for LevenbergMarquardt while the Minuit2 minimizers use a wide two-sided range as before; with compute_errors, fixed parameters and parameters ending on a limit report an error of 0 (fixed parameters previously reported 1.0 with Minuit2).

• FastPedestal variance is now a private double intermediate. Removed the C++ variance()/variance_unchecked() APIs and Python var(). Standard deviation is calculated before conversion to the output type, avoiding overflow of intermediate variance for int16 output. Negative variance from floating-point roundoff is clamped to zero.

• Pedestal now always accumulates sums and sums of squares in double, like FastPedestal. Mean and standard deviation output types are unchanged. Removed the C++ get_sum()/get_sum2() getters and Python sum/sum2 properties; internal sums are no longer exposed.

• Removed the public Pedestal.variance() and cached_std() APIs and C++ update_std(). Use std() to calculate the current population standard deviation; variance is now an internal implementation detail.

• Added ClusterFile::read_frame(ClusterVector&) for allocation-reusing C++ reads. It returns false at a clean end of file. The value-returning C++ overload now returns std::optional<ClusterVector<ClusterType>> and Python read_frame() returns None at end of file. Incomplete frames still raise an error.

• Added an explicit boolean conversion to the C++ FilePtr type.

• Removed the public C++ ClusterFile::open method. Construct a new ClusterFile to reopen a file or change its mode.

• ClusterFinder now uses FastPedestal. It must receive 1000 pedestal frames before cluster finding; find_clusters() raises an error until initialization is complete. Added update_threshold() to recompute the per-pixel detection thresholds.

• Added the queue_depth constructor argument to ClusterFinderMT to configure the number of preallocated frame buffers per worker thread.

• Exposed ClusterVector.empty() in the Python API.

• Added ClusterVector.estimate_n_clusters

• Removed the lmfit dependency and the legacy fit_gaus, fit_pol1, fit_scurve, and fit_scurve2 APIs. Use Gaussian, Pol1, RisingScurve, or FallingScurve and call model.fit(...) (or fit(model, ...)) instead.

• Removed the legacy gaus, pol1, scurve, and scurve2 function evaluators. Model objects are callable and provide the replacement, for example Gaussian()(x, par).

• NDView<T, Ndim> now converts to NDView<const T, Ndim>; expand4to8bit and expand24to32bit accept const input views.

• RawMasterFile::geometry() is deprecetad and returns full detector geometry information including module geometry. Use RawMasterFile::module_layout() to get num_modules in x an y

• RawMasterFile::rois() always returns a list of rois (no optional). Per default it returns a list of one ROI element spawing the entire detector

• TimingMode::Auto changed to TimingMode::AUTO_TIMING, TimingMode::Trigger changed to TimingMode::TRIGGER_EXPOSURE

Bugfixes:

• Fitting a 3D data cube in Python without y_err now returns par_err when compute_errors is set, as fitting a single pixel already did.

• Samples with a zero error are ignored by every minimizer, as documented, also when they hold NaN. They are now removed before the start values are estimated; previously a masked NaN at a sample used by the estimate, such as the first or last point of Pol1, made even Migrad fail. A pixel with fewer remaining samples than free parameters is reported as failed.

• Fitting a single pixel checks that x, y and y_err have the same size and raises RuntimeError otherwise. Previously a shorter y or y_err was read past its end.

• Setting nSigma on ClusterFinder and ClusterFinderMT before the pedestal is ready no longer raises an error. The new value is used when the threshold is initialized. Previously ClusterFinderMT.set_nSigma() also left the worker threads with different nSigma values.

• The Python Cluster constructors validate that the data array holds exactly one value per pixel and raise ValueError otherwise. Previously a longer array wrote past the end of the cluster data.

• The Python CTB decoding helpers adc_sar_*decode64to16, apply_custom_weights, expand24to32bit, expand4to8bit and decode_my302 validate their input through make_view. They require C-contiguous arrays of the documented rank and reject other dtypes with TypeError instead of converting a copy. The ADC SAR decoders no longer reinterpret the byte buffer as aligned 64-bit words; a row length that is not a multiple of 8 bytes raises ValueError instead of dropping the trailing bytes. Read-only arrays are accepted where the input is only read. decode_my302 raises ValueError instead of RuntimeError for a wrong input size.

• Mismatched operators inhibited vectorization in gcc of NDArray math operators

• NDArray/NDView math operators and expressions were not vectorized for uint8_t and 64 bit integers, up to 30x slower than a plain loop.

• Removed the move constructor and move assignment of ProducerConsumerQueue. They left the moved-from queue with a null buffer, crashing its destructor if it still held elements, and leaked the target's buffer on assignment. The queue is now non-movable, as in the upstream folly implementation; hold it in a unique_ptr to move it around.

• ProducerConsumerQueue rejects sizes below two with std::invalid_argument in all build types. The previous assertion was compiled out of Release builds, so a zero-size queue overflowed its buffer on the first write.

• Corrected the license metadata for vendored third-party code. ProducerConsumerQueue.hpp is tagged Apache-2.0 like upstream folly and NumpyHelpers.cpp is tagged MIT like upstream libnpy. The MPL 2.0, Apache 2.0, and MIT texts ship in LICENSES/, and both the conda package and the Python wheel declare MPL-2.0 AND Apache-2.0 AND MIT with all three license files. Building the wheel now requires scikit-build-core 0.11 or newer.

• The wheel and conda package now ship THIRD-PARTY-NOTICES.txt and the LGPL 2.1 text for the libraries compiled into the extension: Minuit2 (LGPL-2.1-or-later), {fmt} and nlohmann/json (MIT), and pybind11 (BSD-3-Clause). Their license expressions include these licenses. Minuit2 is now fetched at tag v6-40-02 instead of master, and libzmq at v4.3.5 (MPL-2.0) instead of v4.3.4 (LGPL-3.0 with a static-linking exception).

• RawFile and File reject raw files with frame padding disabled unless the frame discard policy is discardpartial. The constructor reports the master path before opening data subfiles. Legacy .raw master files now parse the frame discard policy so unpadded discardpartial files remain readable.

• Fixed CtbRawFile.read_frame(index) and reads after seek(index) at subfile boundaries skipping a subfile, returning the wrong frame or raising Subfile index out of range.

• Pedestal reports mismatched frame shapes with exceptions in all push overloads, including Debug builds, instead of aborting on assertions.

• Python Pedestal constructors reject negative dimensions and sample counts instead of converting them to large unsigned values.

• Pedestal clamps negative variance from floating-point roundoff to zero, preventing NaN standard deviations for nearly constant inputs.

• Python Pedestal.push() now requires C-contiguous uint16 frames without implicit conversion. Both push() and push_with_threshold() validate that frames and thresholds are two-dimensional before constructing views, preventing incorrect results from unsupported array layouts or ranks.

• Python ClusterFinder.push_pedestal_frame() and find_clusters() now require C-contiguous uint16 frames without implicit conversion, raising TypeError instead of silently copying and casting the input. The frame argument can also be passed by keyword as frame.

• Fixed a leaked empty ClusterVector at the end of Python ClusterFile iteration. Chunk iteration now rejects a zero chunk size.

• ClusterFile::read_clusters and Python iteration now report incomplete frame headers and cluster records instead of treating truncated files as a clean end of file, with or without ROI or noise filtering.

• ClusterFile::write_frame now reports incomplete writes instead of silently continuing with a truncated file.

• Gain-map application now checks the complete cluster footprint, preventing out-of-bounds access for cluster sizes larger than 3x3.

• RawFile now derives its frame count from the shortest selected raw subfile series across all ROIs. Frame-number reads use the same bounds, and Python len(reader) returns the adjusted count. A warning is printed when subfile counts differ or their minimum differs from the recorded master count. The expected frame count is not used and master metadata is preserved. Warning-level logging is now enabled in non-verbose builds.

• Raw frame, batch, ROI, and frame-number read errors now include the attempted frame index and file path in C++ and Python, preserving subfile error details. Subfile errors omit redundant master-file context and C++ source locations; out-of-range errors include the available frame count. Synchronization errors identify the last raw data file for the affected module. Top-level frame bounds errors state the total frame count and that indices are zero-based.

• Fixed ClusterVector move operations to transfer storage instead of copying every cluster.

• Validate that ClusterVector masks are one-dimensional, C-contiguous Boolean arrays.

• Preserve signed ClusterVector frame numbers when filtering or reducing cluster dimensions.

• Fixed broken reading of old (pre reordering) Moench03

• Supports reading all timing modes supported in slsDetectorPackage (auto, trigger, gating, burst_trigger, trigger_gating)

2026.7.2

New Features:

• Added a new Minuit2-based fitting framework for Gaussian, RisingScurve, FallingScurve, Pol1 and Pol2 models.
• setter and getter for nSigma for ClusterFinder aare.ClusterFinder().nSigma = 2, aare.ClusterFinderMT().set_nSigma(2)
• mask opeartor for ClusterVector masked_clustervector = aare.ClusterVector()(mask)
• passing pre computed eta values to aare.Interpolator.interpolate alongside clusters
• Added PixelHistogram and PedestalTrackingPixelHistogram
• aare.transfrom.Matterhorn10Transform handles counter artefact in chip. Mind that enabling only one counter or three counters or enabling the wromg two counters e.g. 0,1 will still lead to erreneous data.
• aare.transfrom.Matterhorn10Transform reshapes data such that first dimension is number of counters
• Added support for len() for files. Returns the number of frames
• Added support for direct subtraction of Pedestal from numpy array
• Added support to read files with disabled udp ports

Bugfixes:

• Fixed split_task(first, last, n_threads) so task ranges now correctly respect the first offset. Previously, non-zero starting indices could generate incorrect subranges.
• Fixed overflow issue causing failed allocations for NDArrays abouve ~2GB
• Fixed libfmt failures due to consteval when building with C++20

2026.3.17

New Features:

• Decoding transceiver data from Matterhorn10 transformed_data = aare.transform.Matterhorn10Transform(num_counters=2, dynamic_range=16)(data)
• Expanding 24 to 32 bit data aare._aare.expand24to32bit(data, offset=4)
• Decoding digital data from Mythen 302 transformed_data = aare.transform.Mythen302Transform(offset=4)(data)
• added aare.Interpolator.transform_eta_values. Function transforms \eta-values to uniform spatial coordinates. Should only be used for easier debugging.
• New to_string, string_to
• Added exptime and period members to RawMasterFile including decoding
• Removed redundant arr.value(ix,iy...) on NDArray use arr(ix,iy...)
• Removed Print/Print_some/Print_all form NDArray (operator << still works)
• Added const* version of .data()
• reading multiple ROI's supported for aare.
  • Use aare.RawFile.read_roi(roi_index=0) to read a specific ROI for the current frame
  • Use aare.RawFile.read_rois() to read multiple ROIs for the current frame
  • Use aare.RawFile.read_n_with_roi(num_frames = 2, roi_index = 0) to read multiple frames for a specific ROI.
  • Note read_frame and read_n is not supported for multiple ROI's.
• Building conda/pypi pkgs for python 3.14. Removing 3.11 builds.

Bugfixes:

• multi threaded cluster finder doesnt drop frames if queues are full
• Round before casting in the cluster finder to avoid biasing clusters by truncating

2025.11.21

New Features:

• Added SPDX-License-Identifier: MPL-2.0 to source files

• Calculate Eta3 supports all cluster types

• interpolation class supports using cross eta3x3 and eta3x3 on full cluster as well as eta2x2 on full cluster

• interpolation class has option to calculate the rosenblatt transform

• reduction operations to reduce Clusters of general size to 2x2 or 3x3 clusters

• max_sum_2x2 including index of subcluster with highest energy is now available from Python API

• interpolation supports bilinear interpolation of eta values for more fine grained transformed uniform coordinates

• Interpolation is documented

• Added tell to ClusterFile. Returns position in bytes for debugging

Resolved Features:

• calculate_eta coincides with theoretical definition

Bugfixes:

• eta calculation assumes correct photon center
• eta transformation to uniform coordinates starts at 0
• Bug in interpolation
• File supports reading new master json file format (multiple ROI's not supported yet)

API Changes:

• ClusterFinder for 2x2 Cluster disabled
• eta stores corner as enum class cTopLeft, cTopRight, BottomLeft, cBottomRight indicating 2x2 subcluster with largest energy relative to cluster center
• max_sum_2x2 returns corner as index

2025.8.22

Features:

• Apply calibration works in G0 if passes a 2D calibration and pedestal
• count pixels that switch
• calculate pedestal (also g0 version)
• NDArray::view() needs an lvalue to reduce issues with the view outliving the array

Bugfixes:

• Now using glibc 2.17 in conda builds (was using the host)
• Fixed shifted pixels in clusters close to the edge of a frame

2025.7.18

Features:

• Cluster finder now works with 5x5, 7x7 and 9x9 clusters
• Added ClusterVector::empty() member
• Added apply_calibration function for Jungfrau data

Bugfixes:

• Fixed reading RawFiles with ROI fully excluding some sub files.
• Decoding of MH02 files placed the pixels in wrong position
• Removed unused file: ClusterFile.cpp

2025.5.22

Features:

• Added scurve fitting

Bugfixes:

• Fixed crash when opening raw files with large number of data files

Download, Documentation & Support

Download

The Source Code: https://github.com/slsdetectorgroup/aare

Documentation

Documentation including installation details: https://github.com/slsdetectorgroup/aare

Support

erik.frojdh@psi.ch
alice.mazzoleni@psi.ch
dhanya.thattil@psi.ch