detectors/aare
25
0
Fork 0
mirror of https://github.com/slsdetectorgroup/aare.git synced 2025-06-05 12:30:39 +02:00
Code Issues Actions Packages Releases Activity

Feature/reactivate python bindings (#74)

Browse Source 
major changes:
- add python bindings for all c++ features except network_io
- changes to cross compile on windows,linux and macos
- fix bugs with cluster_finder
- use Dtype in Frame instead of bitdepth
- remove boost::program_options and replace with our implementation 
- add Transforms class that applies a sequence of functions (c++ or
python functions) on a Frame.
- remove frame reorder and flip from SubFile.cpp. use Transforms instead
- Test clusterFinder and Pedestal results in comparison with
slsDetectorCalibration

---------

Co-authored-by: Bechir <bechir.brahem420@gmail.com>
Co-authored-by: Erik Fröjdh <erik.frojdh@gmail.com>
This commit is contained in:
Bechir Braham 2024-07-04 11:51:48 +02:00 committed by GitHub
parent 5b7ab5a810
commit 68dcfca74e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
107 changed files with 11451 additions and 1476 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.env
View File

@ -1 +1 @@
export AARE_ROOT_DIR="/home/l_bechir/github/aare/"
export AARE_ROOT_DIR="/home/bb/github/aare/"

36
.github/workflows/clang-tidy.yml vendored
View File

@ -1,36 +0,0 @@
name: clang-tidy
on:
push:
jobs:
clang-tidy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: mamba-org/setup-micromamba@v1
with:
micromamba-version: '1.5.6-0' # any version from https://github.com/mamba-org/micromamba-releases
environment-file: aare-environment.yml
init-shell: bash
cache-environment: true
post-cleanup: 'all'
- name: update clang-tidy
shell: bash -el {0}
run: |
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 17
sudo apt install clang-tidy-17
- name: cmake
shell: bash -el {0}
run: |
mkdir build
cd build
cmake .. -DAARE_SYSTEM_LIBRARIES="ON" -DCMAKE_BUILD_TYPE="Debug" -DAARE_PYTHON_BINDINGS="OFF" -DAARE_IN_GITHUB_ACTIONS="ON"
- name: linting checks
shell: bash -el {0}
run: |
clang-tidy-17 --version
echo $GITHUB_ACTIONS
cd build
cmake --build . --target clang-tidy

26
.github/workflows/common-workflow.yml vendored
View File

@ -11,13 +11,18 @@ on:
type: string
default: Debug
use-python-bindings:
# required: true
required: true
type: string
default: OFF
os:
required: true
type: string
default: ubuntu-latest
jobs:
build-and-test:
runs-on: ubuntu-latest
name: "Build and test[os: ${{ inputs.os }}, build-type: ${{ inputs.build-type }}, use-system-libraries: ${{ inputs.use-system-libraries }}, use-python-bindings: ${{ inputs.use-python-bindings }}]"
runs-on: ${{ inputs.os }}
steps:
- name: Dump GitHub context
env:
@ -42,9 +47,6 @@ jobs:
echo "build-type: ${{inputs.build-type}}"
echo "use-python-bindings: ${{inputs.use-python-bindings}}"
- uses: actions/checkout@v4
- uses: mamba-org/setup-micromamba@v1
if: ${{ contains(inputs.use-system-libraries, 'ON')}}
@ -54,6 +56,7 @@ jobs:
init-shell: bash
cache-environment: true
post-cleanup: 'all'
- name: build
shell: bash -el {0}
run: |
@ -65,7 +68,18 @@ jobs:
run: |
cd build/
./run_tests
if: ${{ inputs.os != 'windows-latest' }}
- name: run tests
run: |
cd build/
dir
cd Debug/
dir
start run_tests.exe
# start /B /WAIT "run_tests.exe"
if: ${{ inputs.os == 'windows-latest' }}
- name: run examples
if: ${{ inputs.os != 'windows-latest' }}
# find all examples in build/examples and run them
run: |
pwd
@ -73,7 +87,7 @@ jobs:
ls build/examples/*_example
# examples to run
cd build/examples
examples=(raw_example cluster_example json_example logger_example multiport_example mythen_example)
examples=(raw_example json_example logger_example multiport_example mythen_example)
examples+=(numpy_read_example numpy_write_example)
for example in "${examples[@]}"; do
echo "Running example: $example"

21
.github/workflows/config.yml vendored
View File

@ -11,18 +11,23 @@ jobs:
# fail-fast is true by default
fail-fast: false
matrix:
use-system-libraries: ["OFF","ON"]
build-type: ["Debug","Release"]
# use-python-bindings: ["OFF","ON"]
# exclude:
# # excludes combinations that are not supported
# - use-python-bindings: "ON"
# use-sanitizers: "ON"
os: ["macos-latest", "ubuntu-latest", "windows-latest"]
build-type: ["Release", "Debug"]
use-system-libraries: ["ON", "OFF"]
use-python-bindings: ["OFF", "ON"]
exclude:
# excludes combinations that are not supported
- os: "windows-latest"
build-type: "Release" # TODO: fix this
- os: "windows-latest"
build-type: "Debug"
use-system-libraries: "ON" # different _ITERATOR_DEBUG_LEVEL for system libraries
uses: ./.github/workflows/common-workflow.yml # calls the one above
with:
use-system-libraries: ${{ matrix.use-system-libraries }}
build-type: ${{ matrix.build-type }}
# use-python-bindings: ${{ matrix.use-python-bindings }}
use-python-bindings: ${{ matrix.use-python-bindings }}
os: ${{ matrix.os }}
secrets: inherit

33
.github/workflows/format.yml vendored
View File

@ -1,33 +0,0 @@
name: test-formatting
on:
push:
jobs:
test-formatting:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: mamba-org/setup-micromamba@v1
with:
micromamba-version: '1.5.6-0' # any version from https://github.com/mamba-org/micromamba-releases
environment-file: aare-environment.yml
init-shell: bash
cache-environment: true
post-cleanup: 'all'
- name: cmake
shell: bash -el {0}
run: |
mkdir build
cd build
cmake .. -DAARE_SYSTEM_LIBRARIES="ON" -DCMAKE_BUILD_TYPE="Debug" -DAARE_PYTHON_BINDINGS="OFF" -DAARE_IN_GITHUB_ACTIONS="ON"
- name: linting checks
shell: bash -el {0}
run: |
# find all examples in build/examples and run them
cd build
cmake --build . --target=check-format

1
.gitignore vendored
View File

@ -1,4 +1,5 @@
.vscode
.env
.env.dev
.cproject
.project

6
.ipynb_checkpoints/Untitled-checkpoint.ipynb Normal file
View File

@ -0,0 +1,6 @@
{
"cells": [],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}

157
CMakeLists.txt
View File

@ -21,7 +21,7 @@ include(FetchContent)
include(cmake/helpers.cmake)
default_build_type("Release")
option(AARE_USE_WARNINGS "Eable warnings" ON)
option(AARE_USE_WARNINGS "Enable warnings" ON)
option(AARE_PYTHON_BINDINGS "Build python bindings" ON)
option(AARE_TESTS "Build tests" ON)
option(AARE_EXAMPLES "Build examples" ON)
@ -32,7 +32,7 @@ option(AARE_FETCH_PYBIND11 "Use FetchContent to download pybind11" ON)
option(AARE_FETCH_CATCH "Use FetchContent to download catch2" ON)
option(AARE_FETCH_JSON "Use FetchContent to download nlohmann::json" ON)
option(AARE_FETCH_ZMQ "Use FetchContent to download libzmq" ON)
option(AARE_FETCH_BOOST "Use FetchContent to download boost" ON)
option(ENABLE_DRAFTS "Enable zmq drafts (depends on gnutls or nss)" OFF)
#Convenience option to use system libraries
@ -44,7 +44,6 @@ if(AARE_SYSTEM_LIBRARIES)
set(AARE_FETCH_CATCH OFF CACHE BOOL "Disabled FetchContent for catch2" FORCE)
set(AARE_FETCH_JSON OFF CACHE BOOL "Disabled FetchContent for nlohmann::json" FORCE)
set(AARE_FETCH_ZMQ OFF CACHE BOOL "Disabled FetchContent for libzmq" FORCE)
set(AARE_FETCH_BOOST OFF CACHE BOOL "Disabled FetchContent for boost" FORCE)
endif()
@ -52,6 +51,7 @@ endif()
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
if(AARE_FETCH_ZMQ)
FetchContent_Declare(
libzmq
GIT_REPOSITORY https://github.com/zeromq/libzmq.git
@ -65,6 +65,7 @@ if(AARE_FETCH_ZMQ)
FetchContent_Populate(libzmq)
add_subdirectory(${libzmq_SOURCE_DIR} ${libzmq_BINARY_DIR} EXCLUDE_FROM_ALL)
endif()
else()
find_package(ZeroMQ 4 REQUIRED)
endif()
@ -80,36 +81,56 @@ if (AARE_FETCH_FMT)
USES_TERMINAL_DOWNLOAD TRUE
)
FetchContent_MakeAvailable(fmt)
set_property(TARGET fmt PROPERTY POSITION_INDEPENDENT_CODE ON)
else()
find_package(fmt 6 REQUIRED)
endif()
if (AARE_FETCH_BOOST)
set(BOOST_INCLUDE_LIBRARIES program_options)
set(BOOST_ENABLE_CMAKE ON)
FetchContent_Declare(
Boost
GIT_REPOSITORY https://github.com/boostorg/boost.git
GIT_TAG boost-1.80.0
)
FetchContent_MakeAvailable(Boost)
set(Boost_LIBRARIES Boost::program_options)
else()
find_package(Boost 1.80 REQUIRED COMPONENTS program_options)
endif()
add_library(aare_compiler_flags INTERFACE)
target_compile_features(aare_compiler_flags INTERFACE cxx_std_17)
#TODO! Explicitly setting flags is not cross platform compatible
#################
# MSVC specific #
#################
if(MSVC)
add_compile_definitions(AARE_MSVC)
if(CMAKE_BUILD_TYPE STREQUAL "Release")
message(STATUS "Release build")
target_compile_options(aare_compiler_flags INTERFACE /O2)
else()
message(STATUS "Debug build")
target_compile_options(
aare_compiler_flags
INTERFACE
/Od
/Zi
/MDd
/D_ITERATOR_DEBUG_LEVEL=2
)
target_link_options(
aare_compiler_flags
INTERFACE
/DEBUG:FULL
)
endif()
target_compile_options(
aare_compiler_flags
INTERFACE
/w # disable warnings
)
else()
######################
# GCC/Clang specific #
######################
if(CMAKE_BUILD_TYPE STREQUAL "Release")
message(STATUS "Release build")
target_compile_options(aare_compiler_flags INTERFACE -O3)
else()
message(STATUS "Debug build")
target_compile_options(
aare_compiler_flags
INTERFACE
@ -120,54 +141,64 @@ else()
)
if (NOT AARE_PYTHON_BINDINGS)
target_compile_options(
aare_compiler_flags
INTERFACE
-fdiagnostics-parseable-fixits
# -fdiagnostics-generate-patch
-fdiagnostics-show-template-tree
-fsanitize=address,undefined,pointer-compare
-fno-sanitize-recover
# -D_FORTIFY_SOURCE=2 # not needed for debug builds
# -fstack-protector # cause errors wih folly (ProducerConsumerQueue.hpp)
-fno-omit-frame-pointer
)
target_compile_options(
aare_compiler_flags
INTERFACE
-fdiagnostics-parseable-fixits
# -fdiagnostics-generate-patch
-fdiagnostics-show-template-tree
-fsanitize=address,undefined,pointer-compare
-fno-sanitize-recover
# -D_FORTIFY_SOURCE=2 # not needed for debug builds
# -fstack-protector # cause errors wih folly? (ProducerConsumerQueue.hpp)
-fno-omit-frame-pointer
)
target_link_libraries(
aare_compiler_flags
INTERFACE
-fdiagnostics-parseable-fixits
# -fdiagnostics-generate-patch
-fdiagnostics-show-template-tree
-fsanitize=address,undefined,pointer-compare
-fno-sanitize-recover
# -D_FORTIFY_SOURCE=2
-fno-omit-frame-pointer
-lstdc++fs
)
endif()
target_link_libraries(
aare_compiler_flags
INTERFACE
-fdiagnostics-parseable-fixits
# -fdiagnostics-generate-patch
-fdiagnostics-show-template-tree
-fsanitize=address,undefined,pointer-compare
-fno-sanitize-recover
# -D_FORTIFY_SOURCE=2
-fno-omit-frame-pointer
)
endif()
endif()
if(AARE_USE_WARNINGS)
target_compile_options(
aare_compiler_flags
INTERFACE
-Wall
-Wextra
-pedantic
-Wshadow
-Wnon-virtual-dtor
-Woverloaded-virtual
-Wdouble-promotion
-Wformat=2
-Wredundant-decls
-Wvla
-Wdouble-promotion
-Werror=return-type #important can cause segfault in optimzed builds
)
target_compile_options(
aare_compiler_flags
INTERFACE
-Wall
-Wextra
-pedantic
-Wshadow
-Wnon-virtual-dtor
-Woverloaded-virtual
-Wdouble-promotion
-Wformat=2
-Wredundant-decls
-Wvla
-Wdouble-promotion
-Werror=return-type #important can cause segfault in optimzed builds
)
endif()
endif() #GCC/Clang specific
if(AARE_TESTS)
enable_testing()
add_subdirectory(tests)
@ -178,7 +209,7 @@ add_subdirectory(src)
#Overall target to link to when using the library
add_library(aare INTERFACE)
target_link_libraries(aare INTERFACE core file_io utils network_io processing ${Boost_LIBRARIES})
target_link_libraries(aare INTERFACE core file_io utils network_io processing aare_compiler_flags)
target_include_directories(aare INTERFACE
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>

1
aare-environment.yml
View File

@ -8,4 +8,3 @@ dependencies:
- nlohmann_json # should be removed
- catch2
- zeromq
- boost-cpp

10
examples/CMakeLists.txt
View File

@ -1,8 +1,9 @@
set(EXAMPLE_LIST "json_example;logger_example;numpy_read_example;multiport_example;raw_example;zmq_restream_example")
set(EXAMPLE_LIST "${EXAMPLE_LIST};mythen_example;numpy_write_example;zmq_receiver_example;zmq_sender_example;")
set(EXAMPLE_LIST "${EXAMPLE_LIST};cluster_example;zmq_multi_receiver;zmq_task_ventilator;zmq_worker;zmq_sink")
set(EXAMPLE_LIST "${EXAMPLE_LIST};testing_pedestal;cluster_finder_example;")
set(EXAMPLE_LIST "json_example;logger_example;numpy_read_example;multiport_example;raw_example;")
set(EXAMPLE_LIST "${EXAMPLE_LIST};mythen_example;numpy_write_example;zmq_sender_example;")
set(EXAMPLE_LIST "${EXAMPLE_LIST};cluster_example;zmq_multi_receiver;zmq_worker;zmq_sink")
set(EXAMPLE_LIST "${EXAMPLE_LIST};zmq_task_ventilator;zmq_restream_example;zmq_receiver_example")
set(EXAMPLE_LIST "${EXAMPLE_LIST};testing_pedestal;cluster_finder_example;reorder_moench_example")
foreach(example ${EXAMPLE_LIST})
add_executable(${example} ${example}.cpp)
@ -11,7 +12,6 @@ foreach(example ${EXAMPLE_LIST})
endforeach()
message(STATUS "Boost_LIBRARIES: ${Boost_LIBRARIES}")

45
examples/cluster_example.cpp
View File

@ -7,33 +7,26 @@
using namespace aare;
int main() {
auto PROJECT_ROOT_DIR = std::filesystem::path(getenv("AARE_ROOT_DIR"));
std::filesystem::path const fpath("/mnt/sls_det_storage/moench_data/testNewFW20230714/cu_half_speed_master_4.json");
NDArray<double, 2> frame({10, 10});
frame = 0;
for (int i = 5; i < 8; i++) {
for (int j = 5; j < 8; j++) {
frame(i, j) = (i + j) % 10;
}
std::filesystem::path const fpath("/home/l_bechir/tmp/testNewFW20230714/cu_half_speed_master_4.json");
auto f = File(fpath, "r");
// calculate pedestal
Pedestal pedestal(400,400,1000);
for (int i = 0; i < 1000; i++) {
auto frame = f.read();
pedestal.push<uint16_t>(frame);
}
for (int i = 0; i < frame.shape(0); i++) {
for (int j = 0; j < frame.shape(1); j++) {
std::cout << frame(i, j) << " ";
}
std::cout << std::endl;
// find clusters
ClusterFinder clusterFinder(3, 3, 5, 0);
f.seek(0);
std::vector<std::vector<Cluster>> clusters_vector;
auto start = std::chrono::high_resolution_clock::now();
for (int i = 0; i < 1000; i++) {
auto frame = f.iread(i);
auto clusters = clusterFinder.find_clusters_without_threshold(frame.view<uint16_t>(), pedestal,false);
clusters_vector.emplace_back(clusters);
}
auto end = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> elapsed_seconds = end-start;
std::cout << "elapsed time: " << elapsed_seconds.count() << "s\n";
ClusterFinder clusterFinder(3, 3, 1, 1); // 3x3 cluster, 1 nSigma, 1 threshold
Pedestal p(10, 10);
auto clusters = clusterFinder.find_clusters(frame.span(), p);
aare::logger::info("nclusters:", clusters.size());
for (auto &cluster : clusters) {
aare::logger::info("cluster center:", cluster.to_string<double>());
}
}

8
examples/cluster_finder_example.cpp
View File

@ -15,7 +15,7 @@ int main() {
logger::info("RAW file");
logger::info("file rows:", file.rows(), "cols:", file.cols());
logger::info(file.total_frames());
Frame frame(0, 0, 0);
Frame frame(0, 0, Dtype::NONE);
for (auto i = 0; i < 10000; i++) {
if (file.frame_number(i) == 23389) {
logger::info("frame_number:", file.frame_number(i));
@ -41,18 +41,18 @@ int main() {
logger::info("Pedestal mean:", p.mean(0, 0), "std:", p.standard_deviation(0, 0));
logger::info("Pedestal mean:", p.mean(200, 200), "std:", p.standard_deviation(200, 200));
FileConfig cfg;
cfg.dtype = DType(typeid(double));
cfg.dtype = Dtype(typeid(double));
cfg.rows = p.rows();
cfg.cols = p.cols();
NumpyFile np_pedestal("/home/l_bechir/tmp/testNewFW20230714/pedestal.npy", "w", cfg);
cfg.dtype = DType(typeid(uint16_t));
cfg.dtype = Dtype(typeid(uint16_t));
NumpyFile np_frame("/home/l_bechir/tmp/testNewFW20230714/frame.npy", "w", cfg);
np_pedestal.write(p.mean());
np_frame.write(frame.view<uint16_t>());
auto clusters = clusterFinder.find_clusters(frame.view<uint16_t>(), p);
auto clusters = clusterFinder.find_clusters_without_threshold(frame.view<uint16_t>(), p);
logger::info("nclusters:", clusters.size());
// aare::logger::info("nclusters:", clusters.size());

4
examples/numpy_write_example.cpp
View File

@ -9,10 +9,10 @@ using aare::Frame;
int main() {
auto path = std::filesystem::path("/tmp/test.npy");
auto dtype = aare::DType(typeid(uint32_t));
auto dtype = aare::Dtype(typeid(uint32_t));
FileConfig const cfg = {dtype, 100, 100};
File npy(path, "w", cfg);
Frame f(100, 100, dtype.bitdepth());
Frame f(100, 100, dtype);
for (int i = 0; i < 10000; i++) {
f.set<uint32_t>(i / 100, i % 100, i);
}

4
examples/raw_example.cpp
View File

@ -33,9 +33,9 @@ int main() {
config.max_frames_per_file = 100;
config.rows = 1024;
config.cols = 512;
config.dtype = aare::DType::UINT16;
config.dtype = aare::Dtype::UINT16;
File file2(path2, "w", config);
Frame frame(1024, 512, 16);
Frame frame(1024, 512, aare::Dtype::UINT16);
for (int i = 0; i < 1024; i++) {
for (int j = 0; j < 512; j++) {

18
examples/reorder_moench_example.cpp Normal file
View File

@ -0,0 +1,18 @@
#include "aare.hpp"
#include "aare/examples/defs.hpp"
#include <iostream>
using namespace aare;
int main() {
std::filesystem::path const fpath("/mnt/sls_det_storage/moench_data/testNewFW20230714/cu_half_speed_master_4.json");
File file(fpath, "r");
Frame frame = file.iread(0);
std::cout << frame.rows() << " " << frame.cols() << " " << frame.bitdepth() << std::endl;
Transforms transforms;
transforms.add(Transforms::reorder_moench());
transforms(frame);
}

8
examples/testing_pedestal.cpp
View File

@ -11,7 +11,7 @@ using namespace aare;
void print_vpair(std::vector<std::pair<int, double>> &v) {
std::cout << "[ ";
for (int i = 0; i < v.size(); i++) {
for (unsigned int i = 0; i < v.size(); i++) {
std::cout << "(" << v[i].first << "," << v[i].second << "), ";
}
std::cout << "]" << std::endl;
@ -19,7 +19,7 @@ void print_vpair(std::vector<std::pair<int, double>> &v) {
int range(int min, int max, int i, int steps) { return min + (max - min) * i / steps; }
int main() {
const int rows = 1, cols = 1;
double MEAN = 5.0, STD = 1.0, VAR = STD * STD, TOLERANCE = 0.1;
double MEAN = 5.0, STD = 1.0;
unsigned seed = std::chrono::system_clock::now().time_since_epoch().count();
std::default_random_engine generator(seed);
@ -53,7 +53,7 @@ int main() {
long double sum2 = 0;
// fill 1000 first values of pedestal
for (int x = 0; x < 1000; x++) {
Frame frame(rows, cols, 64);
Frame frame(rows, cols, Dtype::DOUBLE);
double val = distribution(generator);
frame.set<double>(0, 0, val);
pedestal.push<double>(frame);
@ -63,7 +63,7 @@ int main() {
}
for (int x = 0, aa = 0; x < 100000; x++, aa++) {
Frame frame(rows, cols, 64);
Frame frame(rows, cols, Dtype::DOUBLE);
double val = distribution(generator);
frame.set<double>(0, 0, val);
pedestal.push<double>(frame);

2
examples/zmq_multi_receiver.cpp
View File

@ -1,12 +1,10 @@
#include "aare.hpp"
#include "aare/examples/defs.hpp"
#include <boost/program_options.hpp>
#include <cassert>
#include <fmt/core.h>
#include <string>
using namespace aare;
namespace po = boost::program_options;
using namespace std;
int main() {

32
examples/zmq_receiver_example.cpp
View File

@ -1,39 +1,19 @@
#include "aare.hpp"
#include "aare/examples/defs.hpp"
#include <boost/program_options.hpp>
#include <cassert>
#include <fmt/core.h>
#include <string>
using namespace aare;
namespace po = boost::program_options;
using namespace std;
int main(int argc, char **argv) {
aare::logger::set_verbosity(aare::logger::DEBUG);
po::options_description desc("options");
desc.add_options()("help", "produce help message")("port,p", po::value<uint16_t>()->default_value(5555),
"port number");
po::positional_options_description pd;
pd.add("port", 1);
po::variables_map vm;
try {
auto parsed = po::command_line_parser(argc, argv).options(desc).positional(pd).run();
po::store(parsed, vm);
po::notify(vm);
} catch (const boost::program_options::error &e) {
cout << e.what() << "\n";
cout << desc << "\n";
return 1;
}
if (vm.count("help")) {
cout << desc << "\n";
return 1;
}
auto port = vm["port"].as<uint16_t>();
ArgParser parser("Zmq receiver example");
parser.add_option("port", "p", true, false, "5555", "port number");
auto args = parser.parse(argc, argv);
int port = std::stoi(args["port"]);
std::string const endpoint = "tcp://127.0.0.1:" + std::to_string(port);
aare::ZmqSocketReceiver socket(endpoint);
@ -43,7 +23,7 @@ int main(int argc, char **argv) {
aare::logger::info("Received ", v.size(), " frames");
aare::logger::info("acquisition:", v[0].header.acqIndex);
aare::logger::info("Header size:", v[0].header.to_string().size());
aare::logger::info("Frame size:", v[0].frame.size());
aare::logger::info("Frame size:", v[0].frame.bytes());
aare::logger::info("Header:", v[0].header.to_string());
// for (ZmqFrame zmq_frame : v) {

52
examples/zmq_restream_example.cpp
View File

@ -1,56 +1,26 @@
#include "aare.hpp"
#include "aare/examples/defs.hpp"
#include <boost/program_options.hpp>
#include <chrono>
#include <thread>
using namespace aare;
using namespace std;
namespace po = boost::program_options;
int main(int argc, char **argv) {
aare::logger::set_verbosity(aare::logger::DEBUG);
po::options_description desc("Allowed options");
desc.add_options()("help", "produce help message")("file,f", po::value<string>(), "input file")(
"port,p", po::value<uint16_t>(), "port number")("fps", po::value<uint16_t>()->default_value(1),
"frames per second (default 1)")("loop,l",
"loop over the file");
po::positional_options_description pd;
pd.add("file", -1);
ArgParser parser("Zmq restream example");
parser.add_option("file", "f", true, true, "", "input file");
parser.add_option("port", "p", true, true, "", "port number");
parser.add_option("fps", "fp", true, false, "1", "frames per second");
parser.add_option("loop", "l", false, false, "", "loop over the file");
po::variables_map vm;
try {
auto parsed = po::command_line_parser(argc, argv).options(desc).positional(pd).run();
po::store(parsed, vm);
po::notify(vm);
} catch (const boost::program_options::error &e) {
cout << e.what() << "\n";
cout << desc << "\n";
return 1;
}
if (vm.count("help")) {
cout << desc << "\n";
return 1;
}
if (vm.count("file") != 1) {
aare::logger::error("file is required");
cout << desc << "\n";
return 1;
}
if (vm.count("port") != 1) {
aare::logger::error("port is required");
cout << desc << "\n";
return 1;
}
std::string const path = vm["file"].as<string>();
uint16_t const port = vm["port"].as<uint16_t>();
bool const loop = vm.count("loop") == 1;
uint16_t const fps = vm["fps"].as<uint16_t>();
auto args = parser.parse(argc, argv);
std::string const path = args["file"];
uint16_t const port = std::stoi(args["port"]);
bool const loop = args["loop"] == "1";
uint16_t const fps = std::stoi(args["fps"]);
aare::logger::debug("ARGS: file:", path, "port:", port, "fps:", fps, "loop:", loop);
auto d = round<std::chrono::milliseconds>(std::chrono::duration<double>{1. / fps});
@ -78,7 +48,7 @@ int main(int argc, char **argv) {
header.shape.row = frame.rows();
header.shape.col = frame.cols();
header.bitmode = frame.bitdepth();
header.size = frame.size();
header.size = frame.bytes();
sender.send({header, frame});
std::this_thread::sleep_for(d);

13
examples/zmq_sender_example.cpp
View File

@ -4,7 +4,11 @@
#include <ctime> // std::time
#include <fmt/core.h>
#include <string>
#include <unistd.h> // sleep
// sleep
#include <chrono>
#include <thread>
using namespace aare;
int main() {
@ -12,7 +16,7 @@ int main() {
std::string const endpoint = "tcp://*:5555";
aare::ZmqSocketSender socket(endpoint);
socket.bind();
Frame frame(1024, 1024, sizeof(uint32_t) * 8);
Frame frame(1024, 1024, Dtype::UINT32);
for (int i = 0; i < 1024; i++) {
for (int j = 0; j < 1024; j++) {
frame.set(i, j, i + j);
@ -26,6 +30,7 @@ int main() {
std::vector<ZmqFrame> zmq_frames;
// send two exact frames
std::chrono::milliseconds sleep_time(1); // or whatever
int acqid = 0;
while (true) {
zmq_frames.clear();
@ -34,7 +39,7 @@ int main() {
aare::logger::info("acquisition:", header.acqIndex);
aare::logger::info("Header size:", header.to_string().size());
aare::logger::info("Frame size:", frame.size());
aare::logger::info("Frame size:", frame.bytes());
aare::logger::info("Number of frames:", n_frames);
for (size_t i = 0; i < n_frames; i++) {
@ -42,7 +47,7 @@ int main() {
}
size_t const rc = socket.send(zmq_frames);
aare::logger::info("Sent bytes", rc);
sleep(1);
std::this_thread::sleep_for(sleep_time);
}
return 0;
}

2
examples/zmq_sink.cpp
View File

@ -2,12 +2,10 @@
#include "aare/examples/defs.hpp"
#include "zmq.h"
#include <boost/program_options.hpp>
#include <cassert>
#include <fmt/core.h>
#include <string>
using namespace aare;
namespace po = boost::program_options;
using namespace std;
int main() {

30
examples/zmq_task_ventilator.cpp
View File

@ -1,39 +1,19 @@
#include "aare.hpp"
#include "aare/examples/defs.hpp"
#include "zmq.h"
#include <boost/program_options.hpp>
#include <cassert>
#include <fmt/core.h>
#include <string>
using namespace aare;
namespace po = boost::program_options;
using namespace std;
string setup(int argc, char **argv) {
logger::set_verbosity(logger::DEBUG);
po::options_description desc("options");
desc.add_options()("help", "produce help message")("port,p", po::value<uint16_t>()->default_value(5555),
"port number");
po::positional_options_description pd;
pd.add("port", 1);
po::variables_map vm;
try {
auto parsed = po::command_line_parser(argc, argv).options(desc).positional(pd).run();
po::store(parsed, vm);
po::notify(vm);
} catch (const boost::program_options::error &e) {
cout << e.what() << "\n";
cout << desc << "\n";
exit(1);
}
if (vm.count("help")) {
cout << desc << "\n";
exit(1);
}
auto port = vm["port"].as<uint16_t>();
ArgParser parser("Zmq task ventilator");
parser.add_option("port", "p", true, false, "5555", "port number");
auto args = parser.parse(argc, argv);
int port = std::stoi(args["port"]);
return "tcp://127.0.0.1:" + to_string(port);
}
@ -55,7 +35,7 @@ int process(const std::string &endpoint) {
logger::info(zframe.header.to_string());
// 3. create task
Task *task = Task::init(zframe.frame.data(), zframe.frame.size());
Task *task = Task::init(zframe.frame.data(), zframe.frame.bytes());
task->opcode = (size_t)Task::Operation::PEDESTAL;
task->id = zframe.header.frameNumber;

2
examples/zmq_worker.cpp
View File

@ -3,12 +3,10 @@
#include "aare/examples/defs.hpp"
#include "zmq.h"
#include <boost/program_options.hpp>
#include <cassert>
#include <fmt/core.h>
#include <string>
using namespace aare;
namespace po = boost::program_options;
using namespace std;
int main() {

2
extra/Pedestal testing.ipynb
View File

@ -826,7 +826,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.12.3"
}
},
"nbformat": 4,

5
include/aare/core.hpp
View File

@ -1,8 +1,9 @@
#include "aare/core/CircularFifo.hpp"
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/core/Frame.hpp"
#include "aare/core/NDArray.hpp"
#include "aare/core/NDView.hpp"
#include "aare/core/defs.hpp"
// #include "aare/core/VariableSizeClusterFinder.hpp"
#include "aare/core/ProducerConsumerQueue.hpp"
#include "aare/core/ProducerConsumerQueue.hpp"
#include "aare/core/Transforms.hpp"

1
include/aare/file_io.hpp
View File

@ -1,4 +1,3 @@
#include "aare/file_io/ClusterFile.hpp"
#include "aare/file_io/ClusterFileV2.hpp"
#include "aare/file_io/File.hpp"
#include "aare/file_io/FileInterface.hpp"

1
include/aare/utils.hpp
View File

@ -1,3 +1,4 @@
#include "aare/utils/arg_parser.hpp"
#include "aare/utils/compare_files.hpp"
#include "aare/utils/json.hpp"
#include "aare/utils/logger.hpp"

5
src/core/CMakeLists.txt
View File

@ -2,7 +2,7 @@
set(SourceFiles
${CMAKE_CURRENT_SOURCE_DIR}/src/defs.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/DType.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/Dtype.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/Frame.cpp
)
@ -20,13 +20,14 @@ endif()
if(AARE_TESTS)
set(TestSources
${CMAKE_CURRENT_SOURCE_DIR}/test/defs.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/DType.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/Dtype.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/Frame.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/ProducerConsumerQueue.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/NDArray.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/NDView.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/CircularFifo.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/wrappers.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/Transforms.test.cpp
)
target_sources(tests PRIVATE ${TestSources} )

56
src/core/include/aare/core/DType.hpp
View File

@ -1,56 +0,0 @@
#pragma once
#include <cstdint>
#include <string>
#include <typeinfo>
namespace aare {
/**
* @brief enum class to define the endianess of the system
*/
enum class endian {
#ifdef _WIN32
little = 0,
big = 1,
native = little
#else
little = __ORDER_LITTLE_ENDIAN__,
big = __ORDER_BIG_ENDIAN__,
native = __BYTE_ORDER__
#endif
};
/**
* @brief class to define the data type of the pixels
* @note only native endianess is supported
*/
class DType {
// TODO! support for non native endianess?
static_assert(sizeof(long) == sizeof(int64_t), "long should be 64bits"); // NOLINT
public:
enum TypeIndex { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT, DOUBLE, ERROR };
uint8_t bitdepth() const;
uint8_t bytes() const;
explicit DType(const std::type_info &t);
explicit DType(std::string_view sv);
// not explicit to allow conversions form enum to DType
DType(DType::TypeIndex ti); // NOLINT
bool operator==(const DType &other) const noexcept;
bool operator!=(const DType &other) const noexcept;
bool operator==(const std::type_info &t) const;
bool operator!=(const std::type_info &t) const;
// bool operator==(DType::TypeIndex ti) const;
// bool operator!=(DType::TypeIndex ti) const;
std::string to_string() const;
private:
TypeIndex m_type{TypeIndex::ERROR};
};
} // namespace aare

83
src/core/include/aare/core/Dtype.hpp Normal file
View File

@ -0,0 +1,83 @@
#pragma once
#include <cstdint>
#include <map>
#include <string>
#include <typeinfo>
namespace aare {
// The format descriptor is a single character that specifies the type of the data
// - python documentation: https://docs.python.org/3/c-api/arg.html#numbers
// - py::format_descriptor<T>::format() (in pybind11) does not return the same format as
// written in python.org documentation.
// - numpy also doesn't use the same format. and also numpy associates the format
// with variable bitdepth types. (e.g. long is int64 on linux64 and int32 on win64)
// https://numpy.org/doc/stable/reference/arrays.scalars.html
//
// github issue discussing this:
// https://github.com/pybind/pybind11/issues/1908#issuecomment-658358767
//
// [IN LINUX] the difference is for int64 (long) and uint64 (unsigned long). The format
// descriptor is 'q' and 'Q' respectively and in the documentation it is 'l' and 'k'.
// in practice numpy doesn't seem to care when reading buffer info: the library
// interprets 'q' or 'l' as int64 and 'Q' or 'L' as uint64.
// for this reason we decided to use the same format descriptor as pybind to avoid
// any further discrepancies.
// in the following order:
// int8, uint8, int16, uint16, int32, uint32, int64, uint64, float, double
const char DTYPE_FORMAT_DSC[] = {'b', 'B', 'h', 'H', 'i', 'I', 'q', 'Q', 'f', 'd'};
// on linux64 & apple
const char NUMPY_FORMAT_DSC[] = {'b', 'B', 'h', 'H', 'i', 'I', 'l', 'L', 'f', 'd'};
/**
* @brief enum class to define the endianess of the system
*/
enum class endian {
#ifdef _WIN32
little = 0,
big = 1,
native = little
#else
little = __ORDER_LITTLE_ENDIAN__,
big = __ORDER_BIG_ENDIAN__,
native = __BYTE_ORDER__
#endif
};
/**
* @brief class to define the data type of the pixels
* @note only native endianess is supported
*/
class Dtype {
public:
enum TypeIndex { INT8, UINT8, INT16, UINT16, INT32, UINT32, INT64, UINT64, FLOAT, DOUBLE, ERROR, NONE };
uint8_t bitdepth() const;
size_t bytes() const;
std::string format_descr() const { return std::string(1, DTYPE_FORMAT_DSC[static_cast<int>(m_type)]); }
std::string numpy_descr() const { return std::string(1, NUMPY_FORMAT_DSC[static_cast<int>(m_type)]); }
explicit Dtype(const std::type_info &t);
explicit Dtype(std::string_view sv);
static Dtype from_bitdepth(uint8_t bitdepth);
// not explicit to allow conversions form enum to DType
Dtype(Dtype::TypeIndex ti); // NOLINT
bool operator==(const Dtype &other) const noexcept;
bool operator!=(const Dtype &other) const noexcept;
bool operator==(const std::type_info &t) const;
bool operator!=(const std::type_info &t) const;
// bool operator==(DType::TypeIndex ti) const;
// bool operator!=(DType::TypeIndex ti) const;
std::string to_string() const;
void set_type(Dtype::TypeIndex ti) { m_type = ti; }
private:
TypeIndex m_type{TypeIndex::ERROR};
};
} // namespace aare

101
src/core/include/aare/core/Frame.hpp
View File

@ -1,4 +1,5 @@
#pragma once
#include "aare/core/Dtype.hpp"
#include "aare/core/NDArray.hpp"
#include "aare/core/defs.hpp"
#include <cstddef>
@ -14,83 +15,61 @@ namespace aare {
* should be able to work with streams coming from files or network
*/
class Frame {
size_t m_rows;
size_t m_cols;
size_t m_bitdepth;
uint32_t m_rows;
uint32_t m_cols;
Dtype m_dtype;
std::byte *m_data;
public:
Frame(size_t rows, size_t cols, size_t m_bitdepth);
Frame(std::byte *bytes, size_t rows, size_t cols, size_t m_bitdepth);
std::byte *get(size_t row, size_t col);
Frame(uint32_t rows, uint32_t cols, Dtype dtype);
Frame(const std::byte *bytes, uint32_t rows, uint32_t cols, Dtype dtype);
~Frame() noexcept;
// disable copy and assignment
Frame &operator=(const Frame &other)=delete;
Frame(const Frame &other)=delete;
// enable move
Frame &operator=(Frame &&other) noexcept;
Frame(Frame &&other) noexcept;
// explicit copy
Frame copy() const;
uint32_t rows() const;
uint32_t cols() const;
size_t bitdepth() const;
Dtype dtype() const;
uint64_t size() const;
size_t bytes() const;
std::byte *data() const;
std::byte *get(uint32_t row, uint32_t col);
// TODO! can we, or even want to remove the template?
template <typename T> void set(size_t row, size_t col, T data) {
assert(sizeof(T) == m_bitdepth / 8);
if (row >= m_rows or col >= m_cols) {
template <typename T> void set(uint32_t row, uint32_t col, T data) {
assert(sizeof(T) == m_dtype.bytes());
if (row >= m_rows || col >= m_cols) {
throw std::out_of_range("Invalid row or column index");
}
std::memcpy(m_data + (row * m_cols + col) * (m_bitdepth / 8), &data, m_bitdepth / 8);
std::memcpy(m_data + (row * m_cols + col) * m_dtype.bytes(), &data, m_dtype.bytes());
}
size_t rows() const { return m_rows; }
size_t cols() const { return m_cols; }
size_t bitdepth() const { return m_bitdepth; }
size_t size() const { return m_rows * m_cols * m_bitdepth / 8; }
std::byte *data() const { return m_data; }
Frame &operator=(const Frame &other) {
if (this == &other) {
return *this;
template <typename T> T get_t(uint32_t row, uint32_t col) {
assert(sizeof(T) == m_dtype.bytes());
if (row >= m_rows || col >= m_cols) {
throw std::out_of_range("Invalid row or column index");
}
m_rows = other.rows();
m_cols = other.cols();
m_bitdepth = other.bitdepth();
m_data = new std::byte[m_rows * m_cols * m_bitdepth / 8];
if (m_data == nullptr) {
throw std::bad_alloc();
}
std::memcpy(m_data, other.m_data, m_rows * m_cols * m_bitdepth / 8);
return *this;
T data;
std::memcpy(&data, m_data + (row * m_cols + col) * m_dtype.bytes(), m_dtype.bytes());
return data;
}
Frame &operator=(Frame &&other) noexcept {
m_rows = other.rows();
m_cols = other.cols();
m_bitdepth = other.bitdepth();
if (m_data != nullptr) {
delete[] m_data;
}
m_data = other.m_data;
other.m_data = nullptr;
other.m_rows = other.m_cols = other.m_bitdepth = 0;
return *this;
}
// add move constructor
Frame(Frame &&other) noexcept
: m_rows(other.rows()), m_cols(other.cols()), m_bitdepth(other.bitdepth()), m_data(other.m_data) {
other.m_data = nullptr;
other.m_rows = other.m_cols = other.m_bitdepth = 0;
}
// copy constructor
Frame(const Frame &other)
: m_rows(other.rows()), m_cols(other.cols()), m_bitdepth(other.bitdepth()),
m_data(new std::byte[m_rows * m_cols * m_bitdepth / 8]) {
std::memcpy(m_data, other.m_data, m_rows * m_cols * m_bitdepth / 8);
}
template <typename T> NDView<T, 2> view() {
std::array<ssize_t, 2> shape = {static_cast<ssize_t>(m_rows), static_cast<ssize_t>(m_cols)};
std::array<int64_t, 2> shape = {static_cast<int64_t>(m_rows), static_cast<int64_t>(m_cols)};
T *data = reinterpret_cast<T *>(m_data);
return NDView<T, 2>(data, shape);
}
template <typename T> NDArray<T> image() { return NDArray<T>(this->view<T>()); }
~Frame() { delete[] m_data; }
};
} // namespace aare

104
src/core/include/aare/core/NDArray.hpp
View File

@ -8,7 +8,6 @@ TODO! Add expression templates for operators
*/
#include "aare/core/NDView.hpp"
#include <algorithm>
#include <array>
#include <cmath>
@ -20,15 +19,15 @@ TODO! Add expression templates for operators
namespace aare {
template <typename T, ssize_t Ndim = 2> class NDArray {
template <typename T, int64_t Ndim = 2> class NDArray {
public:
NDArray() : shape_(), strides_(c_strides<Ndim>(shape_)), data_(nullptr){};
explicit NDArray(std::array<ssize_t, Ndim> shape)
explicit NDArray(std::array<int64_t, Ndim> shape)
: shape_(shape), strides_(c_strides<Ndim>(shape_)),
size_(std::accumulate(shape_.begin(), shape_.end(), 1, std::multiplies<>())), data_(new T[size_]){};
NDArray(std::array<ssize_t, Ndim> shape, T value) : NDArray(shape) { this->operator=(value); }
NDArray(std::array<int64_t, Ndim> shape, T value) : NDArray(shape) { this->operator=(value); }
/* When constructing from a NDView we need to copy the data since
NDArray expect to own its data, and span is just a view*/
@ -73,7 +72,7 @@ template <typename T, ssize_t Ndim = 2> class NDArray {
template <typename V> NDArray &operator/=(const NDArray<V, Ndim> &other) {
// check shape
if (shape_ == other.shape()) {
for (int i = 0; i < size_; ++i) {
for (uint32_t i = 0; i < size_; ++i) {
data_[i] /= other(i);
}
return *this;
@ -123,12 +122,13 @@ template <typename T, ssize_t Ndim = 2> class NDArray {
T *data() { return data_; }
std::byte *buffer() { return reinterpret_cast<std::byte *>(data_); }
ssize_t size() const { return size_; }
uint64_t size() const { return size_; }
size_t total_bytes() const { return size_ * sizeof(T); }
std::array<ssize_t, Ndim> shape() const noexcept { return shape_; }
ssize_t shape(ssize_t i) const noexcept { return shape_[i]; }
std::array<ssize_t, Ndim> strides() const noexcept { return strides_; }
std::array<ssize_t, Ndim> byte_strides() const noexcept {
std::array<int64_t, Ndim> shape() const noexcept { return shape_; }
int64_t shape(int64_t i) const noexcept { return shape_[i]; }
std::array<int64_t, Ndim> strides() const noexcept { return strides_; }
size_t bitdepth() const noexcept { return sizeof(T) * 8; }
std::array<int64_t, Ndim> byte_strides() const noexcept {
auto byte_strides = strides_;
for (auto &val : byte_strides)
val *= sizeof(T);
@ -150,14 +150,14 @@ template <typename T, ssize_t Ndim = 2> class NDArray {
}
private:
std::array<ssize_t, Ndim> shape_;
std::array<ssize_t, Ndim> strides_;
ssize_t size_{};
std::array<int64_t, Ndim> shape_;
std::array<int64_t, Ndim> strides_;
uint64_t size_{};
T *data_;
};
// Move assign
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(NDArray<T, Ndim> &&other) noexcept {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(NDArray<T, Ndim> &&other) noexcept {
if (this != &other) {
delete[] data_;
data_ = other.data_;
@ -169,15 +169,15 @@ template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator+(const NDArray &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator+(const NDArray &other) {
NDArray result(*this);
result += other;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator+=(const NDArray<T, Ndim> &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator+=(const NDArray<T, Ndim> &other) {
// check shape
if (shape_ == other.shape_) {
for (int i = 0; i < size_; ++i) {
for (uint32_t i = 0; i < size_; ++i) {
data_[i] += other.data_[i];
}
return *this;
@ -185,32 +185,32 @@ template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator
throw(std::runtime_error("Shape of ImageDatas must match"));
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator-(const NDArray &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator-(const NDArray &other) {
NDArray result{*this};
result -= other;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator-=(const NDArray<T, Ndim> &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator-=(const NDArray<T, Ndim> &other) {
// check shape
if (shape_ == other.shape_) {
for (int i = 0; i < size_; ++i) {
for (uint32_t i = 0; i < size_; ++i) {
data_[i] -= other.data_[i];
}
return *this;
}
throw(std::runtime_error("Shape of ImageDatas must match"));
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator*(const NDArray &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator*(const NDArray &other) {
NDArray result = *this;
result *= other;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator*=(const NDArray<T, Ndim> &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator*=(const NDArray<T, Ndim> &other) {
// check shape
if (shape_ == other.shape_) {
for (int i = 0; i < size_; ++i) {
for (uint32_t i = 0; i < size_; ++i) {
data_[i] *= other.data_[i];
}
return *this;
@ -218,19 +218,19 @@ template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator
throw(std::runtime_error("Shape of ImageDatas must match"));
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator/(const NDArray &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator/(const NDArray &other) {
NDArray result = *this;
result /= other;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator&=(const T &mask) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator&=(const T &mask) {
for (auto it = begin(); it != end(); ++it)
*it &= mask;
return *this;
}
// template <typename T, ssize_t Ndim>
// template <typename T, int64_t Ndim>
// NDArray<T, Ndim>& NDArray<T, Ndim>::operator/=(const NDArray<T, Ndim>&
// other)
// {
@ -245,7 +245,7 @@ template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator
// }
// }
template <typename T, ssize_t Ndim> NDArray<bool, Ndim> NDArray<T, Ndim>::operator>(const NDArray &other) {
template <typename T, int64_t Ndim> NDArray<bool, Ndim> NDArray<T, Ndim>::operator>(const NDArray &other) {
if (shape_ == other.shape_) {
NDArray<bool> result{shape_};
for (int i = 0; i < size_; ++i) {
@ -256,7 +256,7 @@ template <typename T, ssize_t Ndim> NDArray<bool, Ndim> NDArray<T, Ndim>::operat
throw(std::runtime_error("Shape of ImageDatas must match"));
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(const NDArray<T, Ndim> &other) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(const NDArray<T, Ndim> &other) {
if (this != &other) {
delete[] data_;
shape_ = other.shape_;
@ -268,79 +268,79 @@ template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator
return *this;
}
template <typename T, ssize_t Ndim> bool NDArray<T, Ndim>::operator==(const NDArray<T, Ndim> &other) const {
template <typename T, int64_t Ndim> bool NDArray<T, Ndim>::operator==(const NDArray<T, Ndim> &other) const {
if (shape_ != other.shape_)
return false;
for (int i = 0; i != size_; ++i)
for (uint32_t i = 0; i != size_; ++i)
if (data_[i] != other.data_[i])
return false;
return true;
}
template <typename T, ssize_t Ndim> bool NDArray<T, Ndim>::operator!=(const NDArray<T, Ndim> &other) const {
template <typename T, int64_t Ndim> bool NDArray<T, Ndim>::operator!=(const NDArray<T, Ndim> &other) const {
return !((*this) == other);
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator++() {
for (int i = 0; i < size_; ++i)
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator++() {
for (uint32_t i = 0; i < size_; ++i)
data_[i] += 1;
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(const T &value) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator=(const T &value) {
std::fill_n(data_, size_, value);
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator+=(const T &value) {
for (int i = 0; i < size_; ++i)
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator+=(const T &value) {
for (uint32_t i = 0; i < size_; ++i)
data_[i] += value;
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator+(const T &value) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator+(const T &value) {
NDArray result = *this;
result += value;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator-=(const T &value) {
for (int i = 0; i < size_; ++i)
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator-=(const T &value) {
for (uint32_t i = 0; i < size_; ++i)
data_[i] -= value;
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator-(const T &value) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator-(const T &value) {
NDArray result = *this;
result -= value;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator/=(const T &value) {
for (int i = 0; i < size_; ++i)
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator/=(const T &value) {
for (uint32_t i = 0; i < size_; ++i)
data_[i] /= value;
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator/(const T &value) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator/(const T &value) {
NDArray result = *this;
result /= value;
return result;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator*=(const T &value) {
for (int i = 0; i < size_; ++i)
template <typename T, int64_t Ndim> NDArray<T, Ndim> &NDArray<T, Ndim>::operator*=(const T &value) {
for (uint32_t i = 0; i < size_; ++i)
data_[i] *= value;
return *this;
}
template <typename T, ssize_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator*(const T &value) {
template <typename T, int64_t Ndim> NDArray<T, Ndim> NDArray<T, Ndim>::operator*(const T &value) {
NDArray result = *this;
result *= value;
return result;
}
template <typename T, ssize_t Ndim> void NDArray<T, Ndim>::Print() {
template <typename T, int64_t Ndim> void NDArray<T, Ndim>::Print() {
if (shape_[0] < 20 && shape_[1] < 20)
Print_all();
else
Print_some();
}
template <typename T, ssize_t Ndim> void NDArray<T, Ndim>::Print_all() {
template <typename T, int64_t Ndim> void NDArray<T, Ndim>::Print_all() {
for (auto row = 0; row < shape_[0]; ++row) {
for (auto col = 0; col < shape_[1]; ++col) {
std::cout << std::setw(3);
@ -349,7 +349,7 @@ template <typename T, ssize_t Ndim> void NDArray<T, Ndim>::Print_all() {
std::cout << "\n";
}
}
template <typename T, ssize_t Ndim> void NDArray<T, Ndim>::Print_some() {
template <typename T, int64_t Ndim> void NDArray<T, Ndim>::Print_some() {
for (auto row = 0; row < 5; ++row) {
for (auto col = 0; col < 5; ++col) {
std::cout << std::setw(7);
@ -359,15 +359,15 @@ template <typename T, ssize_t Ndim> void NDArray<T, Ndim>::Print_some() {
}
}
template <typename T, ssize_t Ndim> void save(NDArray<T, Ndim> &img, std::string &pathname) {
template <typename T, int64_t Ndim> void save(NDArray<T, Ndim> &img, std::string &pathname) {
std::ofstream f;
f.open(pathname, std::ios::binary);
f.write(img.buffer(), img.size() * sizeof(T));
f.close();
}
template <typename T, ssize_t Ndim>
NDArray<T, Ndim> load(const std::string &pathname, std::array<ssize_t, Ndim> shape) {
template <typename T, int64_t Ndim>
NDArray<T, Ndim> load(const std::string &pathname, std::array<int64_t, Ndim> shape) {
NDArray<T, Ndim> img{shape};
std::ifstream f;
f.open(pathname, std::ios::binary);

61
src/core/include/aare/core/NDView.hpp
View File

@ -3,16 +3,18 @@
#include <array>
#include <cassert>
#include <cstdint>
#include <functional>
#include <iomanip>
#include <iostream>
#include <numeric>
#include <stdexcept>
#include <vector>
namespace aare {
template <ssize_t Ndim> using Shape = std::array<ssize_t, Ndim>;
template <int64_t Ndim> using Shape = std::array<int64_t, Ndim>;
// TODO! fix mismatch between signed and unsigned
template <ssize_t Ndim> Shape<Ndim> make_shape(const std::vector<size_t> &shape) {
template <int64_t Ndim> Shape<Ndim> make_shape(const std::vector<size_t> &shape) {
if (shape.size() != Ndim)
throw std::runtime_error("Shape size mismatch");
Shape<Ndim> arr;
@ -20,41 +22,41 @@ template <ssize_t Ndim> Shape<Ndim> make_shape(const std::vector<size_t> &shape)
return arr;
}
template <ssize_t Dim = 0, typename Strides> ssize_t element_offset(const Strides & /*unused*/) { return 0; }
template <int64_t Dim = 0, typename Strides> int64_t element_offset(const Strides & /*unused*/) { return 0; }
template <ssize_t Dim = 0, typename Strides, typename... Ix>
ssize_t element_offset(const Strides &strides, ssize_t i, Ix... index) {
template <int64_t Dim = 0, typename Strides, typename... Ix>
int64_t element_offset(const Strides &strides, int64_t i, Ix... index) {
return i * strides[Dim] + element_offset<Dim + 1>(strides, index...);
}
template <ssize_t Ndim> std::array<ssize_t, Ndim> c_strides(const std::array<ssize_t, Ndim> &shape) {
std::array<ssize_t, Ndim> strides{};
template <int64_t Ndim> std::array<int64_t, Ndim> c_strides(const std::array<int64_t, Ndim> &shape) {
std::array<int64_t, Ndim> strides{};
std::fill(strides.begin(), strides.end(), 1);
for (ssize_t i = Ndim - 1; i > 0; --i) {
for (int64_t i = Ndim - 1; i > 0; --i) {
strides[i - 1] = strides[i] * shape[i];
}
return strides;
}
template <ssize_t Ndim> std::array<ssize_t, Ndim> make_array(const std::vector<ssize_t> &vec) {
template <int64_t Ndim> std::array<int64_t, Ndim> make_array(const std::vector<int64_t> &vec) {
assert(vec.size() == Ndim);
std::array<ssize_t, Ndim> arr{};
std::array<int64_t, Ndim> arr{};
std::copy_n(vec.begin(), Ndim, arr.begin());
return arr;
}
template <typename T, ssize_t Ndim = 2> class NDView {
template <typename T, int64_t Ndim = 2> class NDView {
public:
NDView() = default;
~NDView() = default;
NDView(const NDView &) = default;
NDView(NDView &&) = default;
NDView(T *buffer, std::array<ssize_t, Ndim> shape)
NDView(T *buffer, std::array<int64_t, Ndim> shape)
: buffer_(buffer), strides_(c_strides<Ndim>(shape)), shape_(shape),
size_(std::accumulate(std::begin(shape), std::end(shape), 1, std::multiplies<>())) {}
// NDView(T *buffer, const std::vector<ssize_t> &shape)
// NDView(T *buffer, const std::vector<int64_t> &shape)
// : buffer_(buffer), strides_(c_strides<Ndim>(make_array<Ndim>(shape))), shape_(make_array<Ndim>(shape)),
// size_(std::accumulate(std::begin(shape), std::end(shape), 1, std::multiplies<>())) {}
@ -66,18 +68,19 @@ template <typename T, ssize_t Ndim = 2> class NDView {
return buffer_[element_offset(strides_, index...)];
}
ssize_t size() const { return size_; }
uint64_t size() const { return size_; }
size_t total_bytes() const { return size_ * sizeof(T); }
std::array<int64_t, Ndim> strides() const noexcept { return strides_; }
T *begin() { return buffer_; }
T *end() { return buffer_ + size_; }
T &operator()(ssize_t i) const { return buffer_[i]; }
T &operator[](ssize_t i) const { return buffer_[i]; }
T &operator()(int64_t i) const { return buffer_[i]; }
T &operator[](int64_t i) const { return buffer_[i]; }
bool operator==(const NDView &other) const {
if (size_ != other.size_)
return false;
for (ssize_t i = 0; i != size_; ++i) {
for (uint64_t i = 0; i != size_; ++i) {
if (buffer_[i] != other.buffer_[i])
return false;
}
@ -119,28 +122,38 @@ template <typename T, ssize_t Ndim = 2> class NDView {
}
auto &shape() { return shape_; }
auto shape(ssize_t i) const { return shape_[i]; }
auto shape(int64_t i) const { return shape_[i]; }
T *data() { return buffer_; }
void print_all() const;
private:
T *buffer_{nullptr};
std::array<ssize_t, Ndim> strides_{};
std::array<ssize_t, Ndim> shape_{};
ssize_t size_{};
std::array<int64_t, Ndim> strides_{};
std::array<int64_t, Ndim> shape_{};
uint64_t size_{};
template <class BinaryOperation> NDView &elemenwise(T val, BinaryOperation op) {
for (ssize_t i = 0; i != size_; ++i) {
for (uint64_t i = 0; i != size_; ++i) {
buffer_[i] = op(buffer_[i], val);
}
return *this;
}
template <class BinaryOperation> NDView &elemenwise(const NDView &other, BinaryOperation op) {
for (ssize_t i = 0; i != size_; ++i) {
for (uint64_t i = 0; i != size_; ++i) {
buffer_[i] = op(buffer_[i], other.buffer_[i]);
}
return *this;
}
};
template <typename T, int64_t Ndim> void NDView<T, Ndim>::print_all() const {
for (auto row = 0; row < shape_[0]; ++row) {
for (auto col = 0; col < shape_[1]; ++col) {
std::cout << std::setw(3);
std::cout << (*this)(row, col) << " ";
}
std::cout << "\n";
}
}
} // namespace aare

147
src/core/include/aare/core/Transforms.hpp Normal file
View File

@ -0,0 +1,147 @@
#pragma once
#include "aare/core/Dtype.hpp"
#include "aare/core/Frame.hpp"
#include "aare/core/defs.hpp"
#include <unordered_set>
#include <vector>
namespace aare {
class Transforms {
// vector to store the transformations function pointers
std::vector<std::function<Frame &(Frame &)>> m_transformations{};
public:
static NDArray<uint64_t, 2> MOENCH_ORDER_MAP;
Transforms() = default;
Transforms(std::vector<std::function<Frame &(Frame &)>> transformations_) : m_transformations(transformations_) {}
void add(std::function<Frame &(Frame &)> transformation) { m_transformations.push_back(transformation); }
void add(std::vector<std::function<Frame &(Frame &)>> transformations) {
for (auto &transformation : transformations) {
this->add(transformation);
}
}
Frame &apply(Frame &frame) {
for (auto &transformation : m_transformations) {
transformation(frame);
}
return frame;
}
Frame &operator()(Frame &frame) { return apply(frame); }
static std::function<Frame &(Frame &)> identity() {
return [](Frame &frame) -> Frame & { return frame; };
}
static std::function<Frame &(Frame &)> zero() {
return [](Frame &frame) -> Frame & {
std::memset(frame.data(), 0, frame.bytes());
return frame;
};
}
static std::function<Frame &(Frame &)> flip_horizental() {
return [](Frame &frame) -> Frame & {
if (frame.rows() == 1) {
return (frame);
}
std::byte *buffer = new std::byte[frame.cols() * frame.bitdepth() / 8];
for (uint64_t src = 0, dst = frame.rows() - 1; src < frame.rows() / 2; dst--, src++) {
auto src_ptr = frame.data() + src * frame.cols() * frame.bitdepth() / 8;
auto dst_ptr = frame.data() + dst * frame.cols() * frame.bitdepth() / 8;
std::memcpy(buffer, src_ptr, frame.cols() * frame.bitdepth() / 8);
std::memcpy(src_ptr, dst_ptr, frame.cols() * frame.bitdepth() / 8);
std::memcpy(dst_ptr, buffer, frame.cols() * frame.bitdepth() / 8);
}
delete[] buffer;
return (frame);
};
}
static std::function<Frame &(Frame &)> reorder(NDView<uint64_t, 2> &order_map) {
if (order_map.size() == 0) {
throw std::runtime_error("Order map is empty");
}
// verify that the order map doesn't have duplicates
std::unordered_set<uint64_t> seen(order_map.begin(), order_map.end());
if (seen.size() != order_map.size()) {
throw std::runtime_error("Order map has duplicates");
}
// verify that the order map has all the values from 0 to rows * cols
for (uint64_t i = 0; i < order_map.size(); i++) {
if (order_map[i] >= order_map.size()) {
throw std::runtime_error("Order map has values less than 0 or greater than rows * cols");
}
}
return [order_map](Frame &frame) -> Frame & {
// verify that the order map has the same shape as the frame
if (order_map.shape(0) != frame.rows() || order_map.shape(1) != frame.cols()) {
throw std::runtime_error("Order map shape mismatch");
}
Frame frame_copy = frame.copy();
// prepare variable for performance optimization (not tested)
const uint64_t pixel_depth = frame.bitdepth() / 8;
std::byte *const dst_data = frame.data();
std::byte *const src_data = frame_copy.data();
const uint64_t size = order_map.size();
uint64_t idx = 0;
uint64_t new_idx;
std::byte *src;
std::byte *dst;
// reorder the frame
for (; idx < size; idx++) {
new_idx = order_map[idx];
src = src_data + idx * pixel_depth;
dst = dst_data + new_idx * pixel_depth;
std::memcpy(dst, src, pixel_depth);
};
return frame;
};
}
static std::function<Frame &(Frame &)> reorder(std::vector<uint64_t> &order_map) {
int64_t tmp = static_cast<int64_t>(order_map.size());
NDView<uint64_t, 2> order_map_view(order_map.data(), {tmp, 1});
return reorder(order_map_view);
}
static std::function<Frame &(Frame &)> reorder(NDArray<uint64_t, 2> &order_map) {
NDView<uint64_t, 2> order_map_view(order_map.data(), order_map.shape());
return reorder(order_map_view);
}
static NDArray<uint64_t, 2> generate_moench_order_map() {
std::array<int, 32> const adc_nr = {300, 325, 350, 375, 300, 325, 350, 375, 200, 225, 250,
275, 200, 225, 250, 275, 100, 125, 150, 175, 100, 125,
150, 175, 0, 25, 50, 75, 0, 25, 50, 75};
int const sc_width = 25;
int const nadc = 32;
int const pixels_per_sc = 5000;
NDArray<uint64_t, 2> order_map({400, 400});
int pixel = 0;
for (int i = 0; i != pixels_per_sc; ++i) {
for (int i_adc = 0; i_adc != nadc; ++i_adc) {
int const col = adc_nr[i_adc] + (i % sc_width);
int row = 0;
if ((i_adc / 4) % 2 == 0)
row = 199 - (i / sc_width);
else
row = 200 + (i / sc_width);
order_map(row, col) = pixel;
pixel++;
}
}
return order_map;
}
static std::function<Frame &(Frame &)> reorder_moench() {
return reorder(MOENCH_ORDER_MAP);
}
};
NDArray<uint64_t,2> Transforms::MOENCH_ORDER_MAP = Transforms::generate_moench_order_map();
} // namespace aare

6
src/core/include/aare/core/VariableSizeClusterFinder.hpp
View File

@ -28,7 +28,7 @@ template <typename T> class ClusterFinder {
};
private:
const std::array<ssize_t, 2> shape_;
const std::array<int64_t, 2> shape_;
NDView<T, 2> original_;
NDArray<int, 2> labeled_;
NDArray<int, 2> peripheral_labeled_;
@ -250,7 +250,7 @@ template <typename T> void ClusterFinder<T>::first_pass() {
template <typename T> void ClusterFinder<T>::second_pass() {
for (ssize_t i = 0; i != labeled_.size(); ++i) {
for (int64_t i = 0; i != labeled_.size(); ++i) {
auto current_label = labeled_(i);
if (current_label != 0) {
auto it = child.find(current_label);
@ -274,7 +274,7 @@ template <typename T> void ClusterFinder<T>::store_clusters() {
std::unordered_map<int, Hit> h_size;
for (int i = 0; i < shape_[0]; ++i) {
for (int j = 0; j < shape_[1]; ++j) {
if (labeled_(i, j) != 0 or false
if (labeled_(i, j) != 0 || false
// (i-1 >= 0 and labeled_(i-1, j) != 0) or // another circle of peripheral pixels
// (j-1 >= 0 and labeled_(i, j-1) != 0) or
// (i+1 < shape_[0] and labeled_(i+1, j) != 0) or

30
src/core/include/aare/core/defs.hpp
View File

@ -1,6 +1,6 @@
#pragma once
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/utils/logger.hpp"
#include <array>
@ -22,13 +22,13 @@ class Cluster {
int cluster_sizeY;
int16_t x;
int16_t y;
DType dt;
Dtype dt;
private:
std::byte *m_data;
public:
Cluster(int cluster_sizeX_, int cluster_sizeY_, DType dt_ = DType(typeid(int32_t)))
Cluster(int cluster_sizeX_, int cluster_sizeY_, Dtype dt_ = Dtype(typeid(int32_t)))
: cluster_sizeX(cluster_sizeX_), cluster_sizeY(cluster_sizeY_), dt(dt_) {
m_data = new std::byte[cluster_sizeX * cluster_sizeY * dt.bytes()]{};
}
@ -38,7 +38,7 @@ class Cluster {
return;
x = other.x;
y = other.y;
memcpy(m_data, other.m_data, other.size());
memcpy(m_data, other.m_data, other.bytes());
}
Cluster &operator=(const Cluster &other) {
if (this == &other)
@ -51,7 +51,7 @@ class Cluster {
: cluster_sizeX(other.cluster_sizeX), cluster_sizeY(other.cluster_sizeY), x(other.x), y(other.y), dt(other.dt),
m_data(other.m_data) {
other.m_data = nullptr;
other.dt = DType(DType::TypeIndex::ERROR);
other.dt = Dtype(Dtype::TypeIndex::ERROR);
}
~Cluster() { delete[] m_data; }
template <typename T> T get(int idx) {
@ -79,7 +79,8 @@ class Cluster {
/**
* @brief size of the cluster in bytes when saved to a file
*/
size_t size() const { return cluster_sizeX * cluster_sizeY * dt.bytes(); }
size_t size() const { return cluster_sizeX * cluster_sizeY ; }
size_t bytes() const { return cluster_sizeX * cluster_sizeY * dt.bytes(); }
auto begin() const { return m_data; }
auto end() const { return m_data + cluster_sizeX * cluster_sizeY * dt.bytes(); }
std::byte *data() { return m_data; }
@ -103,6 +104,21 @@ struct sls_detector_header {
uint8_t detType;
uint8_t version;
std::array<uint8_t, 64> packetMask;
std::string to_string() {
std::string packetMaskStr = "[";
for (auto &i : packetMask) {
packetMaskStr += std::to_string(i) + ", ";
}
packetMaskStr += "]";
return "frameNumber: " + std::to_string(frameNumber) + "\n" + "expLength: " + std::to_string(expLength) + "\n" +
"packetNumber: " + std::to_string(packetNumber) + "\n" + "bunchId: " + std::to_string(bunchId) + "\n" +
"timestamp: " + std::to_string(timestamp) + "\n" + "modId: " + std::to_string(modId) + "\n" +
"row: " + std::to_string(row) + "\n" + "column: " + std::to_string(column) + "\n" +
"reserved: " + std::to_string(reserved) + "\n" + "debug: " + std::to_string(debug) + "\n" +
"roundRNumber: " + std::to_string(roundRNumber) + "\n" + "detType: " + std::to_string(detType) + "\n" +
"version: " + std::to_string(version) + "\n" + "packetMask: " + packetMaskStr + "\n";
}
};
template <typename T> struct t_xy {
@ -114,7 +130,7 @@ template <typename T> struct t_xy {
};
using xy = t_xy<uint32_t>;
using dynamic_shape = std::vector<ssize_t>;
using dynamic_shape = std::vector<int64_t>;
enum class DetectorType { Jungfrau, Eiger, Mythen3, Moench, ChipTestBoard, Unknown };

46
src/core/src/DType.cpp → src/core/src/Dtype.cpp
View File

@ -1,5 +1,5 @@
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/utils/logger.hpp"
#include <fmt/core.h>
@ -12,7 +12,7 @@ namespace aare {
* @note supported types are: int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t, float, double
* @note the type_info object is obtained using typeid (e.g. typeid(int))
*/
DType::DType(const std::type_info &t) {
Dtype::Dtype(const std::type_info &t) {
if (t == typeid(int8_t))
m_type = TypeIndex::INT8;
else if (t == typeid(uint8_t))
@ -25,7 +25,7 @@ DType::DType(const std::type_info &t) {
m_type = TypeIndex::INT32;
else if (t == typeid(uint32_t))
m_type = TypeIndex::UINT32;
else if (t == typeid(int64_t) || t == typeid(long)) // NOLINT
else if (t == typeid(int64_t)) // NOLINT
m_type = TypeIndex::INT64;
else if (t == typeid(uint64_t))
m_type = TypeIndex::UINT64;
@ -41,7 +41,7 @@ DType::DType(const std::type_info &t) {
* @brief Get the bitdepth of the data type
* @return bitdepth
*/
uint8_t DType::bitdepth() const {
uint8_t Dtype::bitdepth() const {
switch (m_type) {
case TypeIndex::INT8:
case TypeIndex::UINT8:
@ -59,7 +59,7 @@ uint8_t DType::bitdepth() const {
return 32;
case TypeIndex::DOUBLE:
return 64;
case TypeIndex::ERROR:
case TypeIndex::NONE:
return 0;
default:
throw std::runtime_error(LOCATION + "Could not get bitdepth. Type not supported.");
@ -69,14 +69,14 @@ uint8_t DType::bitdepth() const {
/**
* @brief Get the number of bytes of the data type
*/
uint8_t DType::bytes() const { return bitdepth() / 8; }
size_t Dtype::bytes() const { return bitdepth() / 8; }
/**
* @brief Construct a DType object from a TypeIndex
* @param ti TypeIndex
*
*/
DType::DType(DType::TypeIndex ti) : m_type(ti) {}
Dtype::Dtype(Dtype::TypeIndex ti) : m_type(ti) {}
/**
* @brief Construct a DType object from a string
@ -85,7 +85,7 @@ DType::DType(DType::TypeIndex ti) : m_type(ti) {}
* @note example strings: "<i4", "u8", "f4"
* @note the endianess is checked and only native endianess is supported
*/
DType::DType(std::string_view sv) {
Dtype::Dtype(std::string_view sv) {
// Check if the file is using our native endianess
if (auto pos = sv.find_first_of("<>"); pos != std::string_view::npos) {
@ -123,14 +123,28 @@ DType::DType(std::string_view sv) {
else if (sv == "f8")
m_type = TypeIndex::DOUBLE;
else
throw std::runtime_error("Could not construct data type. Type no supported.");
throw std::runtime_error("Cannot construct data type from string.");
}
Dtype Dtype::from_bitdepth(uint8_t bitdepth) {
switch (bitdepth) {
case 8:
return Dtype(TypeIndex::UINT8);
case 16:
return Dtype(TypeIndex::UINT16);
case 32:
return Dtype(TypeIndex::UINT32);
case 64:
return Dtype(TypeIndex::UINT64);
default:
throw std::runtime_error("Could not construct data type from bitdepth.");
}
}
/**
* @brief Get the string representation of the data type
* @return string representation
*/
std::string DType::to_string() const {
std::string Dtype::to_string() const {
char ec{};
if (endian::native == endian::little)
@ -160,15 +174,17 @@ std::string DType::to_string() const {
case TypeIndex::DOUBLE:
return "f8";
case TypeIndex::ERROR:
return "ERROR";
throw std::runtime_error("Could not get string representation. Type not supported.");
case TypeIndex::NONE:
throw std::runtime_error("Could not get string representation. Type not supported.");
}
return {};
}
bool DType::operator==(const DType &other) const noexcept { return m_type == other.m_type; }
bool DType::operator!=(const DType &other) const noexcept { return !(*this == other); }
bool Dtype::operator==(const Dtype &other) const noexcept { return m_type == other.m_type; }
bool Dtype::operator!=(const Dtype &other) const noexcept { return !(*this == other); }
bool DType::operator==(const std::type_info &t) const { return DType(t) == *this; }
bool DType::operator!=(const std::type_info &t) const { return DType(t) != *this; }
bool Dtype::operator==(const std::type_info &t) const { return Dtype(t) == *this; }
bool Dtype::operator!=(const std::type_info &t) const { return Dtype(t) != *this; }
} // namespace aare

77
src/core/src/Frame.cpp
View File

@ -13,10 +13,10 @@ namespace aare {
* @param cols number of columns
* @param bitdepth bitdepth of the pixels
*/
Frame::Frame(std::byte *bytes, size_t rows, size_t cols, size_t bitdepth)
: m_rows(rows), m_cols(cols), m_bitdepth(bitdepth), m_data(new std::byte[rows * cols * bitdepth / 8]) {
Frame::Frame(const std::byte *bytes, uint32_t rows, uint32_t cols, Dtype dtype)
: m_rows(rows), m_cols(cols), m_dtype(dtype), m_data(new std::byte[rows * cols * m_dtype.bytes()]) {
std::memcpy(m_data, bytes, rows * cols * bitdepth / 8);
std::memcpy(m_data, bytes, rows * cols * m_dtype.bytes());
}
/**
@ -26,12 +26,20 @@ Frame::Frame(std::byte *bytes, size_t rows, size_t cols, size_t bitdepth)
* @param bitdepth bitdepth of the pixels
* @note the data is initialized to zero
*/
Frame::Frame(size_t rows, size_t cols, size_t bitdepth)
: m_rows(rows), m_cols(cols), m_bitdepth(bitdepth), m_data(new std::byte[rows * cols * bitdepth / 8]) {
Frame::Frame(uint32_t rows, uint32_t cols, Dtype dtype)
: m_rows(rows), m_cols(cols), m_dtype(dtype), m_data(new std::byte[rows * cols * dtype.bytes()]) {
std::memset(m_data, 0, rows * cols * bitdepth / 8);
std::memset(m_data, 0, rows * cols * dtype.bytes());
}
uint32_t Frame::rows() const { return m_rows; }
uint32_t Frame::cols() const { return m_cols; }
size_t Frame::bitdepth() const { return m_dtype.bitdepth(); }
Dtype Frame::dtype() const { return m_dtype; }
uint64_t Frame::size() const { return m_rows * m_cols; }
size_t Frame::bytes() const { return m_rows * m_cols * m_dtype.bytes(); }
std::byte *Frame::data() const { return m_data; }
/**
* @brief Get the pointer to the pixel at the given row and column
* @param row row index
@ -39,12 +47,63 @@ Frame::Frame(size_t rows, size_t cols, size_t bitdepth)
* @return pointer to the pixel
* @note the user should cast the pointer to the appropriate type
*/
std::byte *Frame::get(size_t row, size_t col) {
if (row >= m_rows or col >= m_cols) {
std::byte *Frame::get(uint32_t row, uint32_t col) {
if ((row >= m_rows) || (col >= m_cols)) {
std::cerr << "Invalid row or column index" << '\n';
return nullptr;
}
return m_data + (row * m_cols + col) * (m_bitdepth / 8);
return m_data + (row * m_cols + col) * (m_dtype.bytes());
}
// Frame &Frame::operator=(const Frame &other) {
// if (this == &other) {
// return *this;
// }
// m_rows = other.rows();
// m_cols = other.cols();
// m_dtype = other.dtype();
// m_data = new std::byte[m_rows * m_cols * m_dtype.bytes()];
// if (m_data == nullptr) {
// throw std::bad_alloc();
// }
// std::memcpy(m_data, other.m_data, m_rows * m_cols * m_dtype.bytes());
// return *this;
// }
Frame &Frame::operator=(Frame &&other) noexcept {
if (this == &other) {
return *this;
}
m_rows = other.rows();
m_cols = other.cols();
m_dtype = other.dtype();
if (m_data != nullptr) {
delete[] m_data;
}
m_data = other.m_data;
other.m_data = nullptr;
other.m_rows = other.m_cols = 0;
other.m_dtype = Dtype(Dtype::TypeIndex::ERROR);
return *this;
}
Frame::Frame(Frame &&other) noexcept
: m_rows(other.rows()), m_cols(other.cols()), m_dtype(other.dtype()), m_data(other.m_data) {
other.m_data = nullptr;
other.m_rows = other.m_cols = 0;
other.m_dtype = Dtype(Dtype::TypeIndex::ERROR);
}
// Frame::Frame(const Frame &other)
// : m_rows(other.rows()), m_cols(other.cols()), m_dtype(other.dtype()),
// m_data(new std::byte[m_rows * m_cols * m_dtype.bytes()]) {
// std::memcpy(m_data, other.m_data, m_rows * m_cols * m_dtype.bytes());
// }
Frame Frame::copy() const {
Frame frame(m_rows, m_cols, m_dtype);
std::memcpy(frame.m_data, m_data, m_rows * m_cols * m_dtype.bytes());
return frame;
}
Frame::~Frame() noexcept { delete[] m_data; }
} // namespace aare

54
src/core/test/DType.test.cpp
View File

@ -1,54 +0,0 @@
#include "aare/core/DType.hpp"
#include <catch2/catch_test_macros.hpp>
using aare::DType;
using aare::endian;
TEST_CASE("Construct from typeid") {
REQUIRE(DType(typeid(int)) == typeid(int));
REQUIRE(DType(typeid(int)) != typeid(double));
}
TEST_CASE("Construct from string") {
if (endian::native == endian::little) {
REQUIRE(DType("<i1") == typeid(int8_t));
REQUIRE(DType("<u1") == typeid(uint8_t));
REQUIRE(DType("<i2") == typeid(int16_t));
REQUIRE(DType("<u2") == typeid(uint16_t));
REQUIRE(DType("<i4") == typeid(int));
REQUIRE(DType("<u4") == typeid(unsigned));
REQUIRE(DType("<i4") == typeid(int32_t));
REQUIRE(DType("<i8") == typeid(long));
REQUIRE(DType("<i8") == typeid(int64_t));
REQUIRE(DType("<u4") == typeid(uint32_t));
REQUIRE(DType("<u8") == typeid(uint64_t));
REQUIRE(DType("f4") == typeid(float));
REQUIRE(DType("f8") == typeid(double));
}
if (endian::native == endian::big) {
REQUIRE(DType(">i1") == typeid(int8_t));
REQUIRE(DType(">u1") == typeid(uint8_t));
REQUIRE(DType(">i2") == typeid(int16_t));
REQUIRE(DType(">u2") == typeid(uint16_t));
REQUIRE(DType(">i4") == typeid(int));
REQUIRE(DType(">u4") == typeid(unsigned));
REQUIRE(DType(">i4") == typeid(int32_t));
REQUIRE(DType(">i8") == typeid(long));
REQUIRE(DType(">i8") == typeid(int64_t));
REQUIRE(DType(">u4") == typeid(uint32_t));
REQUIRE(DType(">u8") == typeid(uint64_t));
REQUIRE(DType("f4") == typeid(float));
REQUIRE(DType("f8") == typeid(double));
}
}
TEST_CASE("Construct from string with endianess") {
// TODO! handle big endian system in test!
REQUIRE(DType("<i4") == typeid(int32_t));
REQUIRE_THROWS(DType(">i4") == typeid(int32_t));
}
TEST_CASE("Convert to string") { REQUIRE(DType(typeid(int)).to_string() == "<i4"); }

54
src/core/test/Dtype.test.cpp Normal file
View File

@ -0,0 +1,54 @@
#include "aare/core/Dtype.hpp"
#include <catch2/catch_test_macros.hpp>
using aare::Dtype;
using aare::endian;
TEST_CASE("Construct from typeid") {
REQUIRE(Dtype(typeid(int)) == typeid(int));
REQUIRE(Dtype(typeid(int)) != typeid(double));
}
TEST_CASE("Construct from string") {
if (endian::native == endian::little) {
REQUIRE(Dtype("<i1") == typeid(int8_t));
REQUIRE(Dtype("<u1") == typeid(uint8_t));
REQUIRE(Dtype("<i2") == typeid(int16_t));
REQUIRE(Dtype("<u2") == typeid(uint16_t));
REQUIRE(Dtype("<i4") == typeid(int));
REQUIRE(Dtype("<u4") == typeid(unsigned));
REQUIRE(Dtype("<i4") == typeid(int32_t));
// REQUIRE(Dtype("<i8") == typeid(long));
REQUIRE(Dtype("<i8") == typeid(int64_t));
REQUIRE(Dtype("<u4") == typeid(uint32_t));
REQUIRE(Dtype("<u8") == typeid(uint64_t));
REQUIRE(Dtype("f4") == typeid(float));
REQUIRE(Dtype("f8") == typeid(double));
}
if (endian::native == endian::big) {
REQUIRE(Dtype(">i1") == typeid(int8_t));
REQUIRE(Dtype(">u1") == typeid(uint8_t));
REQUIRE(Dtype(">i2") == typeid(int16_t));
REQUIRE(Dtype(">u2") == typeid(uint16_t));
REQUIRE(Dtype(">i4") == typeid(int));
REQUIRE(Dtype(">u4") == typeid(unsigned));
REQUIRE(Dtype(">i4") == typeid(int32_t));
// REQUIRE(Dtype(">i8") == typeid(long));
REQUIRE(Dtype(">i8") == typeid(int64_t));
REQUIRE(Dtype(">u4") == typeid(uint32_t));
REQUIRE(Dtype(">u8") == typeid(uint64_t));
REQUIRE(Dtype("f4") == typeid(float));
REQUIRE(Dtype("f8") == typeid(double));
}
}
TEST_CASE("Construct from string with endianess") {
// TODO! handle big endian system in test!
REQUIRE(Dtype("<i4") == typeid(int32_t));
REQUIRE_THROWS(Dtype(">i4") == typeid(int32_t));
}
TEST_CASE("Convert to string") { REQUIRE(Dtype(typeid(int)).to_string() == "<i4"); }

69
src/core/test/Frame.test.cpp
View File

@ -1,19 +1,20 @@
#include "aare/core/Frame.hpp"
#include "aare/core/Dtype.hpp"
#include <catch2/catch_test_macros.hpp>
using aare::Frame;
using namespace aare;
TEST_CASE("Construct a frame") {
size_t rows = 10;
size_t cols = 10;
size_t bitdepth = 8;
Frame frame(rows, cols, bitdepth);
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
REQUIRE(frame.rows() == rows);
REQUIRE(frame.cols() == cols);
REQUIRE(frame.bitdepth() == bitdepth);
REQUIRE(frame.size() == rows * cols * bitdepth / 8);
REQUIRE(frame.bytes() == rows * cols * bitdepth / 8);
// data should be initialized to 0
for (size_t i = 0; i < rows; i++) {
@ -30,7 +31,7 @@ TEST_CASE("Set a value in a 8 bit frame") {
size_t cols = 10;
size_t bitdepth = 8;
Frame frame(rows, cols, bitdepth);
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
// set a value
uint8_t value = 255;
@ -55,7 +56,7 @@ TEST_CASE("Set a value in a 64 bit frame") {
size_t cols = 10;
size_t bitdepth = 64;
Frame frame(rows, cols, bitdepth);
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
// set a value
uint64_t value = 255;
@ -80,7 +81,7 @@ TEST_CASE("Move construct a frame") {
size_t cols = 10;
size_t bitdepth = 8;
Frame frame(rows, cols, bitdepth);
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
std::byte *data = frame.data();
Frame frame2(std::move(frame));
@ -88,14 +89,64 @@ TEST_CASE("Move construct a frame") {
// state of the moved from object
REQUIRE(frame.rows() == 0);
REQUIRE(frame.cols() == 0);
REQUIRE(frame.bitdepth() == 0);
REQUIRE(frame.size() == 0);
REQUIRE(frame.dtype() == Dtype(Dtype::TypeIndex::ERROR));
REQUIRE(frame.data() == nullptr);
// state of the moved to object
REQUIRE(frame2.rows() == rows);
REQUIRE(frame2.cols() == cols);
REQUIRE(frame2.bitdepth() == bitdepth);
REQUIRE(frame2.size() == rows * cols * bitdepth / 8);
REQUIRE(frame2.bytes() == rows * cols * bitdepth / 8);
REQUIRE(frame2.data() == data);
}
TEST_CASE("Move assign a frame") {
size_t rows = 10;
size_t cols = 10;
size_t bitdepth = 8;
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
std::byte *data = frame.data();
Frame frame2(5, 5, Dtype::from_bitdepth(16));
frame2 = std::move(frame);
// state of the moved from object
REQUIRE(frame.rows() == 0);
REQUIRE(frame.cols() == 0);
REQUIRE(frame.dtype() == Dtype(Dtype::TypeIndex::ERROR));
REQUIRE(frame.data() == nullptr);
// state of the moved to object
REQUIRE(frame2.rows() == rows);
REQUIRE(frame2.cols() == cols);
REQUIRE(frame2.bitdepth() == bitdepth);
REQUIRE(frame2.bytes() == rows * cols * bitdepth / 8);
REQUIRE(frame2.data() == data);
}
TEST_CASE("test explicit copy constructor") {
size_t rows = 10;
size_t cols = 10;
size_t bitdepth = 8;
Frame frame(rows, cols, Dtype::from_bitdepth(bitdepth));
std::byte *data = frame.data();
Frame frame2 = frame.copy();
// state of the original object
REQUIRE(frame.rows() == rows);
REQUIRE(frame.cols() == cols);
REQUIRE(frame.bitdepth() == bitdepth);
REQUIRE(frame.bytes() == rows * cols * bitdepth / 8);
REQUIRE(frame.data() == data);
// state of the copied object
REQUIRE(frame2.rows() == rows);
REQUIRE(frame2.cols() == cols);
REQUIRE(frame2.bitdepth() == bitdepth);
REQUIRE(frame2.bytes() == rows * cols * bitdepth / 8);
REQUIRE(frame2.data() != data);
}

82
src/core/test/NDArray.test.cpp
View File

@ -22,19 +22,19 @@ TEST_CASE("Construct from a DataSpan") {
REQUIRE(image.size() == view.size());
REQUIRE(image.data() != view.data());
for (int i = 0; i < image.size(); ++i) {
for (uint32_t i = 0; i < image.size(); ++i) {
REQUIRE(image(i) == view(i));
}
// Changing the image doesn't change the view
image = 43;
for (int i = 0; i < image.size(); ++i) {
for (uint32_t i = 0; i < image.size(); ++i) {
REQUIRE(image(i) != view(i));
}
}
TEST_CASE("1D image") {
std::array<ssize_t, 1> shape{{20}};
std::array<int64_t, 1> shape{{20}};
NDArray<short, 1> img(shape, 3);
REQUIRE(img.size() == 20);
REQUIRE(img(5) == 3);
@ -51,13 +51,13 @@ TEST_CASE("Accessing a const object") {
}
TEST_CASE("Indexing of a 2D image") {
std::array<ssize_t, 2> shape{{3, 7}};
std::array<int64_t, 2> shape{{3, 7}};
NDArray<long> img(shape, 5);
for (int i = 0; i != img.size(); ++i) {
for (uint32_t i = 0; i != img.size(); ++i) {
REQUIRE(img(i) == 5);
}
for (int i = 0; i != img.size(); ++i) {
for (uint32_t i = 0; i != img.size(); ++i) {
img(i) = i;
}
REQUIRE(img(0, 0) == 0);
@ -67,14 +67,14 @@ TEST_CASE("Indexing of a 2D image") {
TEST_CASE("Indexing of a 3D image") {
NDArray<float, 3> img{{{3, 4, 2}}, 5.0f};
for (int i = 0; i != img.size(); ++i) {
for (uint32_t i = 0; i != img.size(); ++i) {
REQUIRE(img(i) == 5.0f);
}
// Double check general properties
REQUIRE(img.size() == 3 * 4 * 2);
for (int i = 0; i != img.size(); ++i) {
for (uint32_t i = 0; i != img.size(); ++i) {
img(i) = float(i);
}
REQUIRE(img(0, 0, 0) == 0);
@ -94,10 +94,10 @@ TEST_CASE("Divide double by int") {
}
TEST_CASE("Elementwise multiplication of 3D image") {
std::array<ssize_t, 3> shape{3, 4, 2};
std::array<int64_t, 3> shape{3, 4, 2};
NDArray<double, 3> a{shape};
NDArray<double, 3> b{shape};
for (int i = 0; i != a.size(); ++i) {
for (uint32_t i = 0; i != a.size(); ++i) {
a(i) = i;
b(i) = i;
}
@ -112,31 +112,31 @@ TEST_CASE("Elementwise multiplication of 3D image") {
TEST_CASE("Compare two images") {
NDArray<int> a;
NDArray<int> b;
CHECK(a == b);
CHECK((a == b));
a = NDArray<int>{{5, 10}, 0};
CHECK(a != b);
CHECK((a != b));
b = NDArray<int>{{5, 10}, 0};
CHECK(a == b);
CHECK((a == b));
b(3, 3) = 7;
CHECK(a != b);
CHECK((a != b));
}
TEST_CASE("Size and shape matches") {
ssize_t w = 15;
ssize_t h = 75;
std::array<ssize_t, 2> shape{w, h};
int64_t w = 15;
int64_t h = 75;
std::array<int64_t, 2> shape{w, h};
NDArray<double> a{shape};
REQUIRE(a.size() == w * h);
REQUIRE(a.size() == static_cast<uint64_t>(w * h));
REQUIRE(a.shape() == shape);
}
TEST_CASE("Initial value matches for all elements") {
double v = 4.35;
NDArray<double> a{{5, 5}, v};
for (int i = 0; i < a.size(); ++i) {
for (uint32_t i = 0; i < a.size(); ++i) {
REQUIRE(a(i) == v);
}
}
@ -171,7 +171,7 @@ TEST_CASE("Bitwise and on data") {
// TEST_CASE("Benchmarks")
// {
// NDArray<double> img;
// std::array<ssize_t, 2> shape{ 512, 1024 };
// std::array<int64_t, 2> shape{ 512, 1024 };
// BENCHMARK("Allocate 500k double image")
// {
// NDArray<double>im{ shape };
@ -204,7 +204,7 @@ TEST_CASE("Bitwise and on data") {
// }
TEST_CASE("Elementwise operatios on images") {
std::array<ssize_t, 2> shape{5, 5};
std::array<int64_t, 2> shape{5, 5};
double a_val = 3.0;
double b_val = 8.0;
@ -215,17 +215,17 @@ TEST_CASE("Elementwise operatios on images") {
auto C = A + B;
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val + b_val);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
// Value of B is not changed
for (int i = 0; i < B.size(); ++i) {
for (uint32_t i = 0; i < B.size(); ++i) {
REQUIRE(B(i) == b_val);
}
@ -239,17 +239,17 @@ TEST_CASE("Elementwise operatios on images") {
auto C = A - B;
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val - b_val);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
// Value of B is not changed
for (int i = 0; i < B.size(); ++i) {
for (uint32_t i = 0; i < B.size(); ++i) {
REQUIRE(B(i) == b_val);
}
@ -263,17 +263,17 @@ TEST_CASE("Elementwise operatios on images") {
auto C = A * B;
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val * b_val);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
// Value of B is not changed
for (int i = 0; i < B.size(); ++i) {
for (uint32_t i = 0; i < B.size(); ++i) {
REQUIRE(B(i) == b_val);
}
@ -287,17 +287,17 @@ TEST_CASE("Elementwise operatios on images") {
auto C = A / B;
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val / b_val);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
// Value of B is not changed
for (int i = 0; i < B.size(); ++i) {
for (uint32_t i = 0; i < B.size(); ++i) {
REQUIRE(B(i) == b_val);
}
@ -314,12 +314,12 @@ TEST_CASE("Elementwise operatios on images") {
REQUIRE(C.data() != A.data());
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val - v);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
}
@ -331,12 +331,12 @@ TEST_CASE("Elementwise operatios on images") {
REQUIRE(C.data() != A.data());
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val + v);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
}
@ -348,12 +348,12 @@ TEST_CASE("Elementwise operatios on images") {
REQUIRE(C.data() != A.data());
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val / v);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
}
@ -365,12 +365,12 @@ TEST_CASE("Elementwise operatios on images") {
REQUIRE(C.data() != A.data());
// Value of C matches
for (int i = 0; i < C.size(); ++i) {
for (uint32_t i = 0; i < C.size(); ++i) {
REQUIRE(C(i) == a_val / v);
}
// Value of A is not changed
for (int i = 0; i < A.size(); ++i) {
for (uint32_t i = 0; i < A.size(); ++i) {
REQUIRE(A(i) == a_val);
}
}

8
src/core/test/NDView.test.cpp
View File

@ -147,7 +147,7 @@ TEST_CASE("iterators") {
// for (int i = 0; i != 12; ++i) {
// vec.push_back(i);
// }
// std::vector<ssize_t> shape{3, 4};
// std::vector<int64_t> shape{3, 4};
// NDView<int, 2> data(vec.data(), shape);
// }
@ -156,8 +156,8 @@ TEST_CASE("divide with another span") {
std::vector<int> vec1{3, 2, 1};
std::vector<int> result{3, 6, 3};
NDView<int, 1> data0(vec0.data(), Shape<1>{static_cast<ssize_t>(vec0.size())});
NDView<int, 1> data1(vec1.data(), Shape<1>{static_cast<ssize_t>(vec1.size())});
NDView<int, 1> data0(vec0.data(), Shape<1>{static_cast<int64_t>(vec0.size())});
NDView<int, 1> data1(vec1.data(), Shape<1>{static_cast<int64_t>(vec1.size())});
data0 /= data1;
@ -189,5 +189,5 @@ TEST_CASE("compare two views") {
}
NDView<int, 2> view2(vec2.data(), Shape<2>{3, 4});
REQUIRE(view1 == view2);
REQUIRE((view1 == view2));
}

209
src/core/test/Transforms.test.cpp Normal file
View File

@ -0,0 +1,209 @@
#include "aare/core/Frame.hpp"
#include "aare/core/Transforms.hpp"
#include <catch2/catch_test_macros.hpp>
using namespace aare;
TEST_CASE("test identity transformation") {
Frame f(10, 10, Dtype::INT32);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i + j);
}
}
Frame &f2 = Transforms::identity()(f);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f2.get_t<int32_t>(i, j) == i + j));
}
}
}
TEST_CASE("test zero transformation") {
Frame f(1, 10, Dtype::DOUBLE);
for (int j = 0; j < 10; j++) {
f.set<double>(0, j, j);
}
Frame &f2 = Transforms::zero()(f);
for (int j = 0; j < 10; j++) {
REQUIRE((f.get_t<double>(0, j) == 0));
}
for (int i = 0; i < 10; i++) {
REQUIRE((f2.get_t<double>(0, i) == 0));
}
}
TEST_CASE("test flip_horizental transformation") {
Frame f(10, 10, Dtype::INT32);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i * 10 + j);
}
}
Frame &f2 = Transforms::flip_horizental()(f);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f.get_t<int32_t>(i, j) == (9 - i) * 10 + j));
}
}
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f2.get_t<int32_t>(i, j) == (9 - i) * 10 + j));
}
}
}
TEST_CASE("test horizontal flip odd rows") {
Frame f(5, 10, Dtype::INT32);
for (int i = 0; i < 5; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i * 10 + j);
}
}
Frame &f2 = Transforms::flip_horizental()(f);
for (int i = 0; i < 5; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f2.get_t<int32_t>(i, j) == (4 - i) * 10 + j));
}
}
}
TEST_CASE("test that two horizontal flips are identity") {
Frame f(79, 9, Dtype::INT32);
for (int i = 0; i < 79; i++) {
for (int j = 0; j < 9; j++) {
f.set<int32_t>(i, j, i * 79 + j);
}
}
Frame &f2 = Transforms::flip_horizental()(f);
Frame &f3 = Transforms::flip_horizental()(f2);
for (int i = 0; i < 79; i++) {
for (int j = 0; j < 9; j++) {
REQUIRE(f3.get_t<int32_t>(i, j) == i * 79 + j);
}
}
}
TEST_CASE("test reorder") {
SECTION("test with int32") {
Frame f(10, 10, Dtype::INT32);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i * 10 + j);
}
}
SECTION("change only two elements in the order map") {
Frame tmp = f.copy();
NDArray<uint64_t, 2> order_map({10, 10});
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
order_map(i, j) = i * 10 + j;
}
}
order_map(0, 0) = 99;
order_map(9, 9) = 0;
Frame &f2 = Transforms::reorder(order_map)(tmp);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
if (i == 0 && j == 0) {
REQUIRE(f2.get_t<int32_t>(i, j) == 99);
} else if (i == 9 && j == 9) {
REQUIRE(f2.get_t<int32_t>(i, j) == 0);
} else {
REQUIRE(f2.get_t<int32_t>(i, j) == i * 10 + j);
}
}
}
}
SECTION("test flip_horizental and reorder") {
Frame tmp = f.copy();
// flip frame horizontally
NDArray<uint64_t, 2> order_map({10, 10});
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
order_map(i, j) = (9 - i) * 10 + j;
}
}
Frame &f2 = Transforms::reorder(order_map)(tmp);
Frame &f3 = Transforms::flip_horizental()(f2);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE(f3.get_t<int32_t>(i, j) == f.get_t<int32_t>(i, j));
}
}
}
}
SECTION("test with double") {
Frame f(10, 10, Dtype::DOUBLE);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<double>(i, j, i * 10 + j);
}
}
// flip vertically
Frame tmp = f.copy();
NDArray<uint64_t, 2> order_map({10, 10});
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
order_map(i, j) = i * 10 + (9 - j);
}
}
Frame &f2 = Transforms::reorder(order_map)(tmp);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE(f2.get_t<double>(i, j) == f.get_t<double>(i, 9 - j));
}
}
}
}
TEST_CASE("test chain transformation") {
Frame f(10, 10, Dtype::INT32);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i * 10 + j);
}
}
Transforms transforms({
Transforms::zero(),
});
Frame &f2 = transforms(f);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f2.get_t<int32_t>(i, j) == 0));
}
}
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE(f.get_t<int32_t>(i, j) == 0);
}
}
}
TEST_CASE("test chain transformation with multiple transformations") {
Frame f(10, 10, Dtype::INT32);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
f.set<int32_t>(i, j, i * 10 + j);
}
}
Transforms transforms({
Transforms::identity(),
Transforms::flip_horizental(),
Transforms::identity(),
});
Frame &f2 = transforms(f);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE((f2.get_t<int32_t>(i, j) == (9 - i) * 10 + j));
}
}
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
REQUIRE(f.get_t<int32_t>(i, j) == f2.get_t<int32_t>(i, j));
}
}
}

8
src/core/test/defs.test.cpp
View File

@ -12,22 +12,22 @@ TEST_CASE("Cluster creation") {
aare::Cluster c(13, 15);
REQUIRE(c.cluster_sizeX == 13);
REQUIRE(c.cluster_sizeY == 15);
REQUIRE(c.dt == aare::DType(typeid(int32_t)));
REQUIRE(c.dt == aare::Dtype(typeid(int32_t)));
REQUIRE(c.data() != nullptr);
aare::Cluster c2(c);
REQUIRE(c2.cluster_sizeX == 13);
REQUIRE(c2.cluster_sizeY == 15);
REQUIRE(c2.dt == aare::DType(typeid(int32_t)));
REQUIRE(c2.dt == aare::Dtype(typeid(int32_t)));
REQUIRE(c2.data() != nullptr);
}
TEST_CASE("cluster set and get data") {
aare::Cluster c2(33, 44, aare::DType(typeid(double)));
aare::Cluster c2(33, 44, aare::Dtype(typeid(double)));
REQUIRE(c2.cluster_sizeX == 33);
REQUIRE(c2.cluster_sizeY == 44);
REQUIRE(c2.dt == aare::DType::DOUBLE);
REQUIRE(c2.dt == aare::Dtype::DOUBLE);
double v = 3.14;
c2.set<double>(0, v);
double v2 = c2.get<double>(0);

10
src/core/test/wrappers.test.cpp
View File

@ -1,4 +1,5 @@
#include <aare/core/Frame.hpp>
#include <aare/core/Dtype.hpp>
#include <aare/core/NDView.hpp>
#include <catch2/catch_test_macros.hpp>
#include <cstdint>
@ -6,13 +7,14 @@
using aare::Frame;
using aare::NDArray;
using aare::NDView;
using aare::Dtype;
TEST_CASE("Frame") {
auto data = new uint16_t[100];
for (int i = 0; i < 100; i++) {
data[i] = i;
}
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, 16);
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, Dtype::UINT16);
for (int i = 0; i < 100; i++) {
REQUIRE((uint16_t)*f.get(i / 10, i % 10) == data[i]);
}
@ -32,13 +34,13 @@ TEST_CASE("NDView") {
data[i] = i;
}
SECTION("constructors") {
NDView<uint16_t, 2> ds(data, std::array<ssize_t, 2>({10, 10}));
NDView<uint16_t, 2> ds(data, std::array<int64_t, 2>({10, 10}));
for (int i = 0; i < 100; i++) {
REQUIRE(ds(i / 10, i % 10) == data[i]);
}
}
SECTION("from Frame") {
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, 16);
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, Dtype::UINT16);
NDView<uint16_t> ds = f.view<uint16_t>();
for (int i = 0; i < 100; i++) {
REQUIRE(ds(i / 10, i % 10) == data[i]);
@ -58,7 +60,7 @@ TEST_CASE("NDArray") {
data[i] = i;
}
SECTION("from Frame") {
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, 16);
Frame f(reinterpret_cast<std::byte *>(data), 10, 10, Dtype::UINT16);
NDArray<uint16_t> img = f.image<uint16_t>();
for (int i = 0; i < 100; i++) {
REQUIRE(img(i / 10, i % 10) == data[i]);

2
src/file_io/CMakeLists.txt
View File

@ -15,7 +15,6 @@ set(SourceFiles
${CMAKE_CURRENT_SOURCE_DIR}/src/SubFile.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/NumpyFile.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/NumpyHelpers.cpp
${CMAKE_CURRENT_SOURCE_DIR}/src/ClusterFile.cpp
)
add_library(file_io STATIC ${SourceFiles})
@ -32,7 +31,6 @@ if(AARE_TESTS)
${CMAKE_CURRENT_SOURCE_DIR}/test/NumpyFile.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/NumpyHelpers.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/RawFile.test.cpp
${CMAKE_CURRENT_SOURCE_DIR}/test/ClusterFile.test.cpp
)
target_sources(tests PRIVATE ${TestSources} )
target_link_libraries(tests PRIVATE core file_io)

85
src/file_io/include/aare/file_io/ClusterFile.hpp
View File

@ -1,85 +0,0 @@
#pragma once
#include "aare/core/defs.hpp"
#include <filesystem>
#include <string>
/**
* cluster file format:
* header: [int32 frame_number][int32 n_clusters]
* data: [clusters....]
* where each cluster is of the form:
* typedef struct {
* int16_t x;
* int16_t y;
* int32_t data[n];
*} Cluster ;
*
*/
namespace aare {
/**
* @brief Configuration of the ClusterFile
* can be use as the header of the cluster file
*/
struct ClusterFileConfig {
int32_t frame_number;
int32_t n_clusters;
int cluster_sizeX;
int cluster_sizeY;
DType dt;
ClusterFileConfig(int32_t frame_number_, int32_t n_clusters_, int cluster_sizeX_ = 3, int cluster_sizeY_ = 3,
DType dt_ = DType::INT32)
: frame_number{frame_number_}, n_clusters{n_clusters_}, cluster_sizeX{cluster_sizeX_},
cluster_sizeY{cluster_sizeY_}, dt{dt_} {}
ClusterFileConfig() : ClusterFileConfig(0, 0) {}
bool operator==(const ClusterFileConfig &other) const {
return frame_number == other.frame_number && n_clusters == other.n_clusters && dt == other.dt &&
cluster_sizeX == other.cluster_sizeX && cluster_sizeY == other.cluster_sizeY;
}
bool operator!=(const ClusterFileConfig &other) const { return !(*this == other); }
std::string to_string() const {
return "frame_number: " + std::to_string(frame_number) + ", n_clusters: " + std::to_string(n_clusters) +
", dt: " + dt.to_string() + "\n cluster_sizeX: " + std::to_string(cluster_sizeX) +
", cluster_sizeY: " + std::to_string(cluster_sizeY);
}
};
struct ClusterFileHeader {
int32_t frame_number;
int32_t n_clusters;
};
/**
* @brief Class to read and write clusters to a file
*/
class ClusterFile {
public:
ClusterFile(const std::filesystem::path &fname, const std::string &mode, ClusterFileConfig config = {});
void write(std::vector<Cluster> &clusters);
void write(Cluster &cluster);
Cluster read();
Cluster iread(size_t cluster_number);
std::vector<Cluster> read(size_t n_clusters);
void seek(size_t cluster_number);
size_t tell() const;
size_t count() noexcept;
int32_t frame() const;
void update_header() /* throws */;
~ClusterFile() noexcept;
private:
FILE *fp = nullptr;
// size_t current_cluster{};
std::filesystem::path fname{};
std::string mode{};
int32_t frame_number{};
int32_t n_clusters{};
static const int HEADER_BYTES = 8;
DType dt;
size_t m_cluster_size{};
};
} // namespace aare

80
src/file_io/include/aare/file_io/ClusterFileV2.hpp
View File

@ -7,19 +7,41 @@ namespace aare {
struct ClusterHeader {
int32_t frame_number;
int32_t n_clusters;
std::string to_string() const {
return "frame_number: " + std::to_string(frame_number) + ", n_clusters: " + std::to_string(n_clusters);
}
};
struct ClusterV2_ {
int16_t x;
int16_t y;
std::array<int32_t, 9> data;
std::string to_string(bool detailed = false) const {
if (detailed) {
std::string data_str = "[";
for (auto &d : data) {
data_str += std::to_string(d) + ", ";
}
data_str += "]";
return "x: " + std::to_string(x) + ", y: " + std::to_string(y) + ", data: " + data_str;
}
return "x: " + std::to_string(x) + ", y: " + std::to_string(y);
}
};
struct ClusterV2 {
ClusterV2_ cluster;
int16_t frame_number;
int32_t frame_number;
std::string to_string() const {
return "frame_number: " + std::to_string(frame_number) + ", " + cluster.to_string();
}
};
/**
* @brief
* importtant not: fp always points to the clutsers header and does not point to individual clusters
*
*/
class ClusterFileV2 {
private:
bool m_closed = true;
@ -29,9 +51,24 @@ class ClusterFileV2 {
public:
ClusterFileV2(std::filesystem::path const &fpath, std::string const &mode) {
if (mode != "r" && mode != "w")
throw std::invalid_argument("mode must be 'r' or 'w'");
if (mode == "r" && !std::filesystem::exists(fpath))
throw std::invalid_argument("File does not exist");
m_fpath = fpath;
m_mode = mode;
fp = fopen(fpath.c_str(), "rb");
if (mode == "r") {
fp = fopen(fpath.string().c_str(), "rb");
} else if (mode == "w") {
if (std::filesystem::exists(fpath)) {
fp = fopen(fpath.string().c_str(), "r+b");
} else {
fp = fopen(fpath.string().c_str(), "wb");
}
}
if (fp == nullptr) {
throw std::runtime_error("Failed to open file");
}
m_closed = false;
}
~ClusterFileV2() { close(); }
@ -58,6 +95,45 @@ class ClusterFileV2 {
return clusters;
}
size_t write(std::vector<ClusterV2> const &clusters) {
if (m_mode != "w") {
throw std::runtime_error("File not opened in write mode");
}
if (clusters.empty()) {
return 0;
}
ClusterHeader header;
header.frame_number = clusters[0].frame_number;
header.n_clusters = clusters.size();
fwrite(&header, sizeof(ClusterHeader), 1, fp);
for (auto &c : clusters) {
fwrite(&c.cluster, sizeof(ClusterV2_), 1, fp);
}
return clusters.size();
}
size_t write(std::vector<std::vector<ClusterV2>> const &clusters) {
if (m_mode != "w") {
throw std::runtime_error("File not opened in write mode");
}
size_t n_clusters = 0;
for (auto &c : clusters) {
n_clusters += write(c);
}
return n_clusters;
}
int seek_to_begin() { return fseek(fp, 0, SEEK_SET); }
int seek_to_end() { return fseek(fp, 0, SEEK_END); }
int32_t frame_number() {
auto pos = ftell(fp);
ClusterHeader header;
fread(&header, sizeof(ClusterHeader), 1, fp);
fseek(fp, pos, SEEK_SET);
return header.frame_number;
}
void close() {
if (!m_closed) {
fclose(fp);

5
src/file_io/include/aare/file_io/File.hpp
View File

@ -1,5 +1,6 @@
#pragma once
#include "aare/file_io/FileInterface.hpp"
#include "aare/file_io/RawFile.hpp"
namespace aare {
@ -11,7 +12,7 @@ namespace aare {
class File {
private:
FileInterface *file_impl;
bool is_npy = true;
bool is_npy;
public:
/**
@ -40,6 +41,8 @@ class File {
size_t cols() const;
size_t bitdepth() const;
void set_total_frames(size_t total_frames);
DetectorType detector_type() const;
xy geometry() const;
/**
* @brief Move constructor

13
src/file_io/include/aare/file_io/FileInterface.hpp
View File

@ -1,5 +1,5 @@
#pragma once
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/core/Frame.hpp"
#include "aare/core/defs.hpp"
#include <filesystem>
@ -15,7 +15,7 @@ namespace aare {
* geometry: geometry of the file
*/
struct FileConfig {
aare::DType dtype{typeid(uint16_t)};
aare::Dtype dtype{typeid(uint16_t)};
uint64_t rows{};
uint64_t cols{};
bool operator==(const FileConfig &other) const {
@ -30,6 +30,12 @@ struct FileConfig {
DetectorType detector_type{DetectorType::Unknown};
int max_frames_per_file{};
size_t total_frames{};
std::string to_string() const {
return "{ dtype: " + dtype.to_string() + ", rows: " + std::to_string(rows) + ", cols: " + std::to_string(cols) +
", geometry: " + geometry.to_string() + ", detector_type: " + toString(detector_type) +
", max_frames_per_file: " + std::to_string(max_frames_per_file) +
", total_frames: " + std::to_string(total_frames) + " }";
}
};
/**
@ -161,6 +167,7 @@ class FileInterface {
seek(old_pos);
return tmp;
}
DetectorType detector_type() const { return m_type; }
// function to query the data type of the file
/*virtual DataType dtype = 0; */
@ -178,7 +185,7 @@ class FileInterface {
size_t m_total_frames{};
size_t max_frames_per_file{};
std::string version{};
DetectorType m_type{};
DetectorType m_type{DetectorType::Unknown};
size_t m_rows{};
size_t m_cols{};
size_t m_bitdepth{};

4
src/file_io/include/aare/file_io/NumpyFile.hpp
View File

@ -1,5 +1,5 @@
#pragma once
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/core/defs.hpp"
#include "aare/file_io/FileInterface.hpp"
#include "aare/file_io/NumpyHelpers.hpp"
@ -47,7 +47,7 @@ class NumpyFile : public FileInterface {
* @brief get the data type of the numpy file
* @return DType
*/
DType dtype() const { return m_header.dtype; }
Dtype dtype() const { return m_header.dtype; }
/**
* @brief get the shape of the numpy file

6
src/file_io/include/aare/file_io/NumpyHelpers.hpp
View File

@ -11,7 +11,7 @@
#include <unordered_map>
#include <vector>
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/core/defs.hpp"
namespace aare {
@ -19,7 +19,7 @@ namespace aare {
using shape_t = std::vector<size_t>;
struct NumpyHeader {
DType dtype{aare::DType::ERROR};
Dtype dtype{aare::Dtype::ERROR};
bool fortran_order{false};
shape_t shape{};
@ -50,7 +50,7 @@ template <typename T, size_t N> bool in_array(T val, const std::array<T, N> &arr
}
bool is_digits(const std::string &str);
aare::DType parse_descr(std::string typestring);
aare::Dtype parse_descr(std::string typestring);
size_t write_header(const std::filesystem::path &fname, const NumpyHeader &header);
size_t write_header(std::ostream &out, const NumpyHeader &header);

3
src/file_io/include/aare/file_io/RawFile.hpp
View File

@ -116,6 +116,7 @@ class RawFile : public FileInterface {
size_t rows() const override { return m_rows; }
size_t cols() const override { return m_cols; }
size_t bitdepth() const override { return m_bitdepth; }
xy geometry() { return m_geometry; }
private:
void write_master_file();
@ -175,7 +176,7 @@ class RawFile : public FileInterface {
size_t n_subfile_parts{};
std::vector<std::vector<SubFile *>> subfiles;
size_t subfile_rows{}, subfile_cols{};
xy geometry{};
xy m_geometry{};
std::vector<xy> positions;
ModuleConfig cfg{0, 0};
TimingMode timing_mode{};

24
src/file_io/include/aare/file_io/SubFile.hpp
View File

@ -12,30 +12,6 @@ namespace aare {
* @brief Class to read a subfile from a RawFile
*/
class SubFile {
protected:
/**
* @brief type of the read_impl function pointer
* @param buffer pointer to the buffer to read the data into
* @return number of bytes read
*/
using pfunc = size_t (SubFile::*)(std::byte *);
pfunc read_impl = nullptr;
/**
* @brief map to store the read_impl functions for different detectors
* @note the key is a pair of DetectorType and bitdepth
* @note the value is a pointer to the read_impl function specific for the detector
* @note the read_impl function will be set to the appropriate function in the constructor
*/
std::map<std::pair<DetectorType, int>, pfunc> read_impl_map = {
{{DetectorType::Moench, 16}, &SubFile::read_impl_reorder<uint16_t>},
{{DetectorType::Jungfrau, 16}, &SubFile::read_impl_normal},
{{DetectorType::ChipTestBoard, 16}, &SubFile::read_impl_normal},
{{DetectorType::Mythen3, 32}, &SubFile::read_impl_normal},
{{DetectorType::Eiger, 32}, &SubFile::read_impl_normal},
{{DetectorType::Eiger, 16}, &SubFile::read_impl_normal}
};
public:
size_t write_part(std::byte *buffer, sls_detector_header header, size_t frame_index);
/**

241
src/file_io/src/ClusterFile.cpp
View File

@ -1,241 +0,0 @@
#include "aare/file_io/ClusterFile.hpp"
#include "aare/core/defs.hpp"
#include "aare/utils/logger.hpp"
#include "fmt/core.h"
namespace aare {
/**
* @brief Constructor for the ClusterFile class.
*
* Opens the file with the given mode ('r' for read, 'w' for write).
* Throws an exception if the mode is not 'r' or 'w', or if the file cannot be opened.
*
* @param fname_ The name of the file to open.
* @param mode_ The mode to open the file in.
* @param config Configuration for the file header.
*/
ClusterFile::ClusterFile(const std::filesystem::path &fname_, const std::string &mode_, ClusterFileConfig config)
: fname{fname_}, mode{mode_}, frame_number{config.frame_number}, n_clusters{config.n_clusters}, dt{config.dt} {
// check if the file has the .clust extension
if (fname.extension() != ".clust") {
aare::logger::warn("file", fname, "does not have .clust extension");
}
if (mode == "r") {
// check if the file exists and is a regular file
if (not std::filesystem::exists(fname)) {
throw std::invalid_argument(fmt::format("file {} does not exist", fname.c_str()));
}
if (not std::filesystem::is_regular_file(fname)) {
throw std::invalid_argument(fmt::format("file {} is not a regular file", fname.c_str()));
}
if (dt == DType(DType::TypeIndex::ERROR)) {
throw std::invalid_argument("data type not set in ClusterFileConfig");
}
// open file
fp = fopen(fname.c_str(), "rb");
if (fp == nullptr) {
throw std::runtime_error(fmt::format("could not open file {}", fname.c_str()));
}
// read header
ClusterFileHeader header{};
const size_t rc = fread(&header, sizeof(header), 1, fp);
if (rc != 1) {
throw std::runtime_error(fmt::format("could not read header from file {}", fname.c_str()));
}
frame_number = header.frame_number;
n_clusters = header.n_clusters;
} else if (mode == "w") {
// open file
fp = fopen(fname.c_str(), "wb");
if (fp == nullptr) {
throw std::runtime_error(fmt::format("could not open file {}", fname.c_str()));
}
// write header
ClusterFileHeader header{config.frame_number, config.n_clusters};
if (fwrite(&header, sizeof(header), 1, fp) != 1) {
throw std::runtime_error(fmt::format("could not write header to file {}", fname.c_str()));
}
} else {
throw std::invalid_argument("mode must be 'r' or 'w'");
}
m_cluster_size =
2 * sizeof(int16_t) + static_cast<size_t>(config.cluster_sizeX * config.cluster_sizeY * dt.bytes());
}
/**
* @brief Writes a vector of clusters to the file.
*
* @param clusters The vector of clusters to write to the file.
*/
void ClusterFile::write(std::vector<Cluster> &clusters) {
if (clusters.empty()) {
return;
}
assert(clusters[0].dt == dt && "cluster data type mismatch");
// prepare buffer to write to file
auto *buffer = new std::byte[m_cluster_size * clusters.size()];
for (size_t i = 0; i < clusters.size(); i++) {
auto &cluster = clusters[i];
memcpy(buffer + i * m_cluster_size, &cluster.x, sizeof(int16_t));
memcpy(buffer + i * m_cluster_size + sizeof(int16_t), &cluster.y, sizeof(int16_t));
memcpy(buffer + i * m_cluster_size + 2 * sizeof(int16_t), cluster.data(), cluster.size());
}
fwrite(buffer, m_cluster_size * clusters.size(), 1, fp);
delete[] buffer;
}
/**
* @brief Writes a single cluster to the file.
*
* @param cluster The cluster to write to the file.
*/
void ClusterFile::write(Cluster &cluster) {
// prepare buffer to write to file
auto *buffer = new std::byte[m_cluster_size];
memcpy(buffer, &cluster.x, sizeof(int16_t));
memcpy(buffer + sizeof(int16_t), &cluster.y, sizeof(int16_t));
memcpy(buffer + 2 * sizeof(int16_t), cluster.data(), cluster.size());
fwrite(buffer, m_cluster_size, 1, fp);
delete[] buffer;
}
/**
* @brief Reads a single cluster from the file.
*
* @return The cluster read from the file.
*/
Cluster ClusterFile::read() {
if (tell() >= count()) {
throw std::runtime_error("cluster number out of range");
}
Cluster cluster(3, 3, DType::INT32);
auto *tmp = new std::byte[cluster.size() + 2 * sizeof(int16_t)];
fread(tmp, cluster.size() + 2 * sizeof(int16_t), 1, fp);
memcpy(&cluster.x, tmp, sizeof(int16_t));
memcpy(&cluster.y, tmp + sizeof(int16_t), sizeof(int16_t));
memcpy(cluster.data(), tmp + 2 * sizeof(int16_t), cluster.size());
delete[] tmp;
return cluster;
}
/**
* @brief Reads a specific cluster from the file.
*
* The file pointer is moved to the specific cluster
*
* @param cluster_number The number of the cluster to read from the file.
* @return The cluster read from the file.
*/
Cluster ClusterFile::iread(size_t cluster_number) {
if (cluster_number >= count()) {
throw std::runtime_error("cluster number out of range");
}
auto old_pos = ftell(fp);
this->seek(cluster_number);
Cluster cluster = read();
fseek(fp, old_pos, SEEK_SET); // restore the file position
return cluster;
}
/**
* @brief Reads a specific number of clusters from the file.
*
* Each cluster is read as a binary block of size sizeof(Cluster).
*
* @param n_clusters The number of clusters to read from the file.
* @return A vector of clusters read from the file.
*/
std::vector<Cluster> ClusterFile::read(size_t n_clusters_) {
if (n_clusters_ + tell() > count()) {
throw std::runtime_error("cluster number out of range");
}
std::vector<Cluster> clusters(n_clusters_, Cluster(3, 3, DType::INT32));
// TODO: read all clusters at once if possible
for (size_t i = 0; i < n_clusters_; i++) {
clusters[i] = read();
}
return clusters;
}
/**
* @brief Moves the file pointer to a specific cluster.
*
* The file pointer is moved to the start of the specific cluster, based on the size of a cluster.
*
* @param cluster_number The number of the cluster to move the file pointer to.
*/
void ClusterFile::seek(size_t cluster_number) {
if (cluster_number > count()) {
throw std::runtime_error("cluster number out of range");
}
const auto offset = static_cast<int64_t>(sizeof(ClusterFileHeader) + cluster_number * m_cluster_size);
fseek(fp, offset, SEEK_SET);
}
/**
* @brief Gets the current position of the file pointer in terms of clusters.
*
* The position is calculated as the number of clusters from the beginning of the file to the current position of
* the file pointer.
*
* @return The current position of the file pointer in terms of clusters.
*/
size_t ClusterFile::tell() const { return (ftell(fp) - sizeof(ClusterFileHeader)) / m_cluster_size; }
/**
* @brief Counts the number of clusters in the file.
*
* The count is calculated as the size of the file divided by the size of a cluster.
*
* @return The number of clusters in the file.
*/
size_t ClusterFile::count() noexcept {
if (mode == "r") {
return n_clusters;
}
// save the current position
auto old_pos = ftell(fp);
fseek(fp, 0, SEEK_END);
const size_t n_clusters_ = (ftell(fp) - sizeof(ClusterFileHeader)) / m_cluster_size;
// restore the file position
fseek(fp, old_pos, SEEK_SET);
return n_clusters_;
}
int32_t ClusterFile::frame() const { return frame_number; }
void ClusterFile::update_header() {
if (mode == "r") {
throw std::runtime_error("update header is not implemented for read mode");
}
// update the header with the correct number of clusters
aare::logger::debug("updating header with correct number of clusters", count());
auto tmp_n_clusters = count();
fseek(fp, 0, SEEK_SET);
ClusterFileHeader header{frame_number, static_cast<int32_t>(tmp_n_clusters)};
if (fwrite(&header, sizeof(ClusterFileHeader), 1, fp) != 1) {
throw std::runtime_error("could not write header to file");
}
if (fflush(fp) != 0) {
throw std::runtime_error("could not flush file");
}
}
ClusterFile::~ClusterFile() noexcept {
if (mode == "w") {
try {
update_header();
} catch (std::runtime_error &e) {
aare::logger::error("error updating header", e.what());
}
}
if (fp != nullptr) {
fclose(fp);
}
}
} // namespace aare

18
src/file_io/src/ClusterFileV2.cpp
View File

@ -1,18 +0,0 @@
#include "aare/file_io/ClusterFileV2.hpp"
namespace aare {
// ClusterFileV2::ClusterFileV2(std::filesystem::path const &fpath, std::string const &mode,
// ClusterFileConfig const &config)
// : m_fpath(fpath), m_mode(mode) {
// if (mode == "w") {
// fp = fopen(fpath.c_str(), "wb");
// write_header(config);
// } else if (mode == "r") {
// fp = fopen(fpath.c_str(), "rb");
// read_header();
// } else {
// throw std::runtime_error("invalid mode");
// }
// }
} // namespace aare

16
src/file_io/src/File.cpp
View File

@ -6,13 +6,14 @@
namespace aare {
File::File(const std::filesystem::path &fname, const std::string &mode, const FileConfig &cfg) {
File::File(const std::filesystem::path &fname, const std::string &mode, const FileConfig &cfg)
: file_impl(nullptr), is_npy(true) {
if (mode != "r" && mode != "w" && mode != "a") {
throw std::invalid_argument("Unsupported file mode");
}
if ((mode == "r" or mode == "a") and not std::filesystem::exists(fname)) {
throw std::runtime_error(fmt::format("File does not exist: {}", fname.c_str()));
if ((mode == "r" || mode == "a") && !std::filesystem::exists(fname)) {
throw std::runtime_error(fmt::format("File does not exist: {}", fname.string()));
}
if (fname.extension() == ".raw" || fname.extension() == ".json") {
@ -52,10 +53,17 @@ size_t File::cols() const { return file_impl->cols(); }
size_t File::bitdepth() const { return file_impl->bitdepth(); }
void File::set_total_frames(size_t total_frames) { return file_impl->set_total_frames(total_frames); }
File::~File() { delete file_impl; }
DetectorType File::detector_type() const { return file_impl->detector_type(); }
xy File::geometry() const {
if (is_npy) {
return {1, 1};
}
return reinterpret_cast<RawFile *>(file_impl)->geometry();
}
Frame File::iread(size_t frame_number) { return file_impl->iread(frame_number); }
File::File(File &&other) noexcept : file_impl(other.file_impl) { other.file_impl = nullptr; }
File::File(File &&other) noexcept : file_impl(other.file_impl), is_npy(other.is_npy) { other.file_impl = nullptr; }
// write move assignment operator

18
src/file_io/src/NumpyFile.cpp
View File

@ -9,9 +9,9 @@ NumpyFile::NumpyFile(const std::filesystem::path &fname, const std::string &mode
m_fname = fname;
m_mode = mode;
if (mode == "r") {
fp = fopen(m_fname.c_str(), "rb");
fp = fopen(m_fname.string().c_str(), "rb");
if (!fp) {
throw std::runtime_error(fmt::format("Could not open: {} for reading", m_fname.c_str()));
throw std::runtime_error(fmt::format("Could not open: {} for reading", m_fname.string()));
}
load_metadata();
} else if (mode == "w") {
@ -20,9 +20,9 @@ NumpyFile::NumpyFile(const std::filesystem::path &fname, const std::string &mode
m_cols = cfg.cols;
m_header = {cfg.dtype, false, {cfg.rows, cfg.cols}};
m_header.shape = {0, cfg.rows, cfg.cols};
fp = fopen(m_fname.c_str(), "wb");
fp = fopen(m_fname.string().c_str(), "wb");
if (!fp) {
throw std::runtime_error(fmt::format("Could not open: {} for reading", m_fname.c_str()));
throw std::runtime_error(fmt::format("Could not open: {} for reading", m_fname.string()));
}
initial_header_len = aare::NumpyHelpers::write_header(std::filesystem::path(m_fname.c_str()), m_header);
}
@ -30,13 +30,13 @@ NumpyFile::NumpyFile(const std::filesystem::path &fname, const std::string &mode
m_bytes_per_frame = m_header.dtype.bitdepth() / 8 * m_pixels_per_frame;
}
void NumpyFile::write(Frame &frame) { write_impl(frame.data(), frame.size()); }
void NumpyFile::write(Frame &frame) { write_impl(frame.data(), frame.bytes()); }
void NumpyFile::write_impl(void *data, uint64_t size) {
if (fp == nullptr) {
throw std::runtime_error("File not open");
}
if (not(m_mode == "w" or m_mode == "a")) {
if (!(m_mode == "w" || m_mode == "a")) {
throw std::invalid_argument("File not open for writing");
}
if (fseek(fp, 0, SEEK_END))
@ -50,7 +50,7 @@ void NumpyFile::write_impl(void *data, uint64_t size) {
}
Frame NumpyFile::get_frame(size_t frame_number) {
Frame frame(m_header.shape[1], m_header.shape[2], m_header.dtype.bitdepth());
Frame frame(m_header.shape[1], m_header.shape[2], m_header.dtype);
get_frame_into(frame_number, frame.data());
return frame;
}
@ -91,7 +91,7 @@ void NumpyFile::read_into(std::byte *image_buf, size_t n_frames) {
}
NumpyFile::~NumpyFile() noexcept {
if (m_mode == "w" or m_mode == "a") {
if (m_mode == "w" || m_mode == "a") {
// determine number of frames
if (fseek(fp, 0, SEEK_END)) {
aare::logger::error("Could not seek to end of file");
@ -182,7 +182,7 @@ void NumpyFile::load_metadata() {
std::string const shape_s = dict_map["shape"];
std::string const descr = aare::NumpyHelpers::parse_str(descr_s);
aare::DType const dtype = aare::NumpyHelpers::parse_descr(descr);
aare::Dtype const dtype = aare::NumpyHelpers::parse_descr(descr);
// convert literal Python bool to C++ bool
bool const fortran_order = aare::NumpyHelpers::parse_bool(fortran_s);

6
src/file_io/src/NumpyHelpers.cpp
View File

@ -90,7 +90,7 @@ std::unordered_map<std::string, std::string> parse_dict(std::string in, const st
return map;
}
aare::DType parse_descr(std::string typestring) {
aare::Dtype parse_descr(std::string typestring) {
if (typestring.length() < 3) {
throw std::runtime_error("invalid typestring (length)");
@ -118,7 +118,7 @@ aare::DType parse_descr(std::string typestring) {
throw std::runtime_error("invalid typestring (itemsize)");
}
return aare::DType(typestring);
return aare::Dtype(typestring);
}
bool parse_bool(const std::string &in) {
@ -245,7 +245,7 @@ size_t write_header(std::ostream &out, const NumpyHeader &header) {
write_magic(out, version_major, version_minor);
// write header length
if (version_major == 1 and version_minor == 0) {
if (version_major == 1 && version_minor == 0) {
auto header_len = static_cast<uint16_t>(header_dict.length() + padding.length() + 1);
std::array<uint8_t, 2> header_len_le16{static_cast<uint8_t>((header_len >> 0) & 0xff),

48
src/file_io/src/RawFile.cpp
View File

@ -12,7 +12,7 @@ namespace aare {
RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode, const FileConfig &config) {
m_mode = mode;
m_fname = fname;
if (mode == "r" or mode == "r+") {
if (mode == "r" || mode == "r+") {
if (config != FileConfig()) {
aare::logger::warn(
"In read mode it is not necessary to provide a config, the provided config will be ignored");
@ -23,7 +23,7 @@ RawFile::RawFile(const std::filesystem::path &fname, const std::string &mode, co
find_geometry();
open_subfiles();
} else if (mode == "w" or mode == "w+") {
} else if (mode == "w" || mode == "w+") {
if (std::filesystem::exists(fname)) {
// handle mode w as w+ (no overrwriting)
@ -51,12 +51,12 @@ void RawFile::parse_config(const FileConfig &config) {
m_cols = config.cols;
m_type = config.detector_type;
max_frames_per_file = config.max_frames_per_file;
geometry = config.geometry;
m_geometry = config.geometry;
version = config.version;
subfile_rows = config.geometry.row;
subfile_cols = config.geometry.col;
if (geometry != aare::xy{1, 1}) {
if (m_geometry != aare::xy{1, 1}) {
throw std::runtime_error(LOCATION + "Only geometry {1,1} files are supported for writing");
}
}
@ -74,10 +74,10 @@ void RawFile::write_master_file() {
ss += "\n\t";
aare::write_str(ss, "Detector Type", toString(m_type));
ss += "\n\t";
aare::write_str(ss, "Geometry", geometry.to_string());
aare::write_str(ss, "Geometry", m_geometry.to_string());
ss += "\n\t";
uint64_t img_size = (m_cols * m_rows) / (static_cast<size_t>(geometry.col * geometry.row));
uint64_t img_size = (m_cols * m_rows) / (static_cast<size_t>(m_geometry.col * m_geometry.row));
img_size *= m_bitdepth;
aare::write_digit(ss, "Image Size in bytes", img_size);
ss += "\n\t";
@ -85,8 +85,8 @@ void RawFile::write_master_file() {
ss += "\n\t";
aare::write_digit(ss, "Dynamic Range", m_bitdepth);
ss += "\n\t";
const aare::xy pixels = {static_cast<uint32_t>(m_rows / geometry.row),
static_cast<uint32_t>(m_cols / geometry.col)};
const aare::xy pixels = {static_cast<uint32_t>(m_rows / m_geometry.row),
static_cast<uint32_t>(m_cols / m_geometry.col)};
aare::write_str(ss, "Pixels", pixels.to_string());
ss += "\n\t";
aare::write_digit(ss, "Number of rows", m_rows);
@ -132,9 +132,9 @@ void RawFile::open_subfiles() {
sls_detector_header RawFile::read_header(const std::filesystem::path &fname) {
sls_detector_header h{};
FILE *fp = fopen(fname.c_str(), "r");
FILE *fp = fopen(fname.string().c_str(), "r");
if (!fp)
throw std::runtime_error(fmt::format("Could not open: {} for reading", fname.c_str()));
throw std::runtime_error(fmt::format("Could not open: {} for reading", fname.string()));
size_t const rc = fread(reinterpret_cast<char *>(&h), sizeof(h), 1, fp);
if (rc != 1)
@ -146,7 +146,7 @@ sls_detector_header RawFile::read_header(const std::filesystem::path &fname) {
return h;
}
bool RawFile::is_master_file(const std::filesystem::path &fpath) {
std::string const stem = fpath.stem();
std::string const stem = fpath.stem().string();
return stem.find("_master_") != std::string::npos;
}
@ -203,7 +203,7 @@ void RawFile::parse_metadata() {
} else {
throw std::runtime_error(LOCATION + "Unsupported file type");
}
n_subfile_parts = static_cast<size_t>(geometry.row) * geometry.col;
n_subfile_parts = static_cast<size_t>(m_geometry.row) * m_geometry.col;
}
void RawFile::parse_json_metadata() {
@ -228,7 +228,7 @@ void RawFile::parse_json_metadata() {
quad = (j["Quad"] == 1);
}
geometry = {j["Geometry"]["y"], j["Geometry"]["x"]};
m_geometry = {j["Geometry"]["y"], j["Geometry"]["x"]};
}
void RawFile::parse_raw_metadata() {
std::ifstream ifs(master_fname());
@ -267,8 +267,8 @@ void RawFile::parse_raw_metadata() {
max_frames_per_file = std::stoi(value);
} else if (key == "Geometry") {
pos = value.find(',');
geometry = {static_cast<uint32_t>(std::stoi(value.substr(1, pos))),
static_cast<uint32_t>(std::stoi(value.substr(pos + 1)))};
m_geometry = {static_cast<uint32_t>(std::stoi(value.substr(1, pos))),
static_cast<uint32_t>(std::stoi(value.substr(pos + 1)))};
}
}
}
@ -276,9 +276,9 @@ void RawFile::parse_raw_metadata() {
void RawFile::parse_fname() {
bool wrong_format = false;
m_base_path = m_fname.parent_path();
m_base_name = m_fname.stem();
m_ext = m_fname.extension();
m_base_path = m_fname.parent_path().string();
m_base_name = m_fname.stem().string();
m_ext = m_fname.extension().string();
try {
auto pos = m_base_name.rfind('_');
m_findex = std::stoi(m_base_name.substr(pos + 1));
@ -291,14 +291,14 @@ void RawFile::parse_fname() {
m_base_name.erase(pos);
wrong_format = true;
}
if (wrong_format and (m_mode == "w+" or m_mode == "w")) {
if (wrong_format && (m_mode == "w+" || m_mode == "w")) {
aare::logger::warn("Master Filename", m_fname, "is not in the correct format");
aare::logger::warn("using", master_fname(), "as the master file");
}
}
Frame RawFile::get_frame(size_t frame_index) {
auto f = Frame(this->m_rows, this->m_cols, this->m_bitdepth);
auto f = Frame(this->m_rows, this->m_cols, Dtype::from_bitdepth(this->m_bitdepth));
std::byte *frame_buffer = f.data();
get_frame_into(frame_index, frame_buffer);
return f;
@ -335,7 +335,7 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer) {
}
}
if (this->geometry.col == 1) {
if (this->m_geometry.col == 1) {
// get the part from each subfile and copy it to the frame
for (size_t part_idx = 0; part_idx != this->n_subfile_parts; ++part_idx) {
auto corrected_idx = frame_indices[part_idx];
@ -357,8 +357,8 @@ void RawFile::get_frame_into(size_t frame_index, std::byte *frame_buffer) {
this->subfiles[subfile_id][part_idx]->get_part(part_buffer, corrected_idx % this->max_frames_per_file);
for (size_t cur_row = 0; cur_row < (this->subfile_rows); cur_row++) {
auto irow = cur_row + (part_idx / this->geometry.col) * this->subfile_rows;
auto icol = (part_idx % this->geometry.col) * this->subfile_cols;
auto irow = cur_row + (part_idx / this->m_geometry.col) * this->subfile_rows;
auto icol = (part_idx % this->m_geometry.col) * this->subfile_cols;
auto dest = (irow * this->m_cols + icol);
dest = dest * this->m_bitdepth / 8;
memcpy(frame_buffer + dest, part_buffer + cur_row * this->subfile_cols * this->m_bitdepth / 8,
@ -410,7 +410,7 @@ size_t RawFile::frame_number(size_t frame_index) {
RawFile::~RawFile() noexcept {
// update master file
if (m_mode == "w" or m_mode == "w+" or m_mode == "r+") {
if (m_mode == "w" || m_mode == "w+" || m_mode == "r+") {
try {
write_master_file();
} catch (...) {

71
src/file_io/src/SubFile.cpp
View File

@ -11,12 +11,6 @@ SubFile::SubFile(const std::filesystem::path &fname, DetectorType detector, size
const std::string &mode)
: m_bitdepth(bitdepth), m_fname(fname), m_rows(rows), m_cols(cols), m_mode(mode) {
if (read_impl_map.find({detector, bitdepth}) == read_impl_map.end()) {
auto error_msg = LOCATION + "No read_impl function found for detector: " + toString(detector) +
" and bitdepth: " + std::to_string(bitdepth);
throw std::invalid_argument(error_msg);
}
this->read_impl = read_impl_map.at({detector, bitdepth});
if (std::filesystem::exists(fname)) {
n_frames = std::filesystem::file_size(fname) / (sizeof(sls_detector_header) + rows * cols * bitdepth / 8);
} else {
@ -24,14 +18,14 @@ SubFile::SubFile(const std::filesystem::path &fname, DetectorType detector, size
}
if (mode == "r") {
fp = fopen(m_fname.c_str(), "rb");
fp = fopen(m_fname.string().c_str(), "rb");
} else {
// if file exists, open in read/write mode (without truncating the file)
// if file does not exist, open in write mode
if (std::filesystem::exists(fname)) {
fp = fopen(m_fname.c_str(), "r+b");
fp = fopen(m_fname.string().c_str(), "r+b");
} else {
fp = fopen(m_fname.c_str(), "wb");
fp = fopen(m_fname.string().c_str(), "wb");
}
}
if (fp == nullptr) {
@ -43,76 +37,21 @@ size_t SubFile::get_part(std::byte *buffer, size_t frame_index) {
if (frame_index >= n_frames) {
throw std::runtime_error("Frame number out of range");
}
// TODO: find a way to avoid opening and closing the file for each frame
aare::logger::debug(LOCATION, "frame:", frame_index, "file:", m_fname.c_str());
fseek(fp, sizeof(sls_detector_header) + (sizeof(sls_detector_header) + bytes_per_part()) * frame_index, // NOLINT
SEEK_SET);
auto ret = (this->*read_impl)(buffer);
return ret;
return fread(buffer, this->bytes_per_part(), 1, this->fp);
}
size_t SubFile::write_part(std::byte *buffer, sls_detector_header header, size_t frame_index) {
if (frame_index > n_frames) {
throw std::runtime_error("Frame number out of range");
}
fseek(fp, static_cast<ssize_t>((sizeof(sls_detector_header) + bytes_per_part()) * frame_index), SEEK_SET);
fseek(fp, static_cast<int64_t>((sizeof(sls_detector_header) + bytes_per_part()) * frame_index), SEEK_SET);
auto wc = fwrite(reinterpret_cast<char *>(&header), sizeof(header), 1, fp);
wc += fwrite(buffer, bytes_per_part(), 1, fp);
return wc;
}
size_t SubFile::read_impl_normal(std::byte *buffer) { return fread(buffer, this->bytes_per_part(), 1, this->fp); }
template <typename DataType> size_t SubFile::read_impl_reorder(std::byte *buffer) {
std::vector<DataType> tmp(this->pixels_per_part());
size_t const rc = fread(reinterpret_cast<char *>(tmp.data()), this->bytes_per_part(), 1, this->fp);
std::array<int, 32> const adc_nr = {300, 325, 350, 375, 300, 325, 350, 375, 200, 225, 250, 275, 200, 225, 250, 275,
100, 125, 150, 175, 100, 125, 150, 175, 0, 25, 50, 75, 0, 25, 50, 75};
int const sc_width = 25;
int const nadc = 32;
int const pixels_per_sc = 5000;
auto dst = reinterpret_cast<DataType *>(buffer);
int pixel = 0;
for (int i = 0; i != pixels_per_sc; ++i) {
for (int i_adc = 0; i_adc != nadc; ++i_adc) {
int const col = adc_nr[i_adc] + (i % sc_width);
int row = 0;
if ((i_adc / 4) % 2 == 0)
row = 199 - (i / sc_width);
else
row = 200 + (i / sc_width);
dst[col + row * 400] = tmp[pixel];
pixel++;
}
}
return rc;
};
template <typename DataType> size_t SubFile::read_impl_flip(std::byte *buffer) {
// read to temporary buffer
// TODO! benchmark direct reads
std::vector<std::byte> tmp(this->bytes_per_part());
size_t const rc = fread(reinterpret_cast<char *>(tmp.data()), this->bytes_per_part(), 1, this->fp);
// copy to place
const size_t start = this->m_cols * (this->m_rows - 1) * sizeof(DataType);
const size_t row_size = this->m_cols * sizeof(DataType);
auto *dst = buffer + start;
auto *src = tmp.data();
for (size_t i = 0; i != this->m_rows; ++i) {
std::memcpy(dst, src, row_size);
dst -= row_size;
src += row_size;
}
return rc;
};
size_t SubFile::frame_number(size_t frame_index) {
sls_detector_header h{};
fseek(fp, (sizeof(sls_detector_header) + bytes_per_part()) * frame_index, SEEK_SET); // NOLINT

114
src/file_io/test/ClusterFile.test.cpp
View File

@ -1,114 +0,0 @@
#include "aare/file_io/ClusterFile.hpp"
#include "aare/utils/compare_files.hpp"
#include "test_config.hpp"
#include <catch2/catch_test_macros.hpp>
#include <iostream>
#include <random>
using aare::Cluster;
using aare::ClusterFile;
using aare::ClusterFileConfig;
TEST_CASE("Read a cluster file") {
auto fpath = test_data_path() / "clusters" / "single_frame_97_clustrers.clust";
REQUIRE(std::filesystem::exists(fpath));
ClusterFile cf(fpath, "r");
SECTION("Read the header of the file") {
REQUIRE(cf.count() == 97);
REQUIRE(cf.frame() == 135);
}
SECTION("Read a single cluster") {
Cluster c = cf.read();
REQUIRE(c.x == 1);
REQUIRE(c.y == 200);
for (int i = 0; i < 9; i++) {
REQUIRE(c.get<int32_t>(i) == i);
}
}
SECTION("Read a single cluster using iread") {
Cluster c = cf.iread(0);
REQUIRE(c.x == 1);
REQUIRE(c.y == 200);
for (int i = 0; i < 9; i++) {
REQUIRE(c.get<int32_t>(i) == i);
}
}
SECTION("Read a cluster using seek") {
cf.seek(0);
Cluster c = cf.read();
REQUIRE(c.x == 1);
REQUIRE(c.y == 200);
for (int i = 0; i < 9; i++) {
REQUIRE(c.get<int32_t>(i) == i);
}
c = cf.read();
REQUIRE(c.x == 2);
REQUIRE(c.y == 201);
for (int i = 0; i < 9; i++) {
REQUIRE(c.get<int32_t>(i) == i + 9);
}
}
SECTION("check out of bound reading") {
REQUIRE_THROWS_AS(cf.iread(97), std::runtime_error);
REQUIRE_NOTHROW(cf.seek(97));
REQUIRE_THROWS_AS(cf.read(), std::runtime_error);
REQUIRE_THROWS_AS(cf.read(1), std::runtime_error);
REQUIRE_NOTHROW(cf.seek(0));
REQUIRE_NOTHROW(cf.read(97));
}
SECTION("test read multiple clusters") {
std::vector<Cluster> cluster = cf.read(97);
REQUIRE(cluster.size() == 97);
int offset = 0;
int data_offset = 0;
for (auto c : cluster) {
REQUIRE(c.x == offset + 1);
REQUIRE(c.y == offset + 200);
for (int i = 0; i < 9; i++) {
REQUIRE(c.get<int32_t>(i) == data_offset + i);
}
offset++;
data_offset += 9;
}
}
}
TEST_CASE("write a cluster file") {
auto const FRAME_NUMBER = 1461041991;
auto const TOTAL_CLUSTERS = 214748;
std::filesystem::path const fpath_out("/tmp/file.clust");
ClusterFile cf_out(fpath_out, "w", ClusterFileConfig(FRAME_NUMBER, TOTAL_CLUSTERS));
REQUIRE(cf_out.count() == 0);
REQUIRE(cf_out.frame() == FRAME_NUMBER);
// write file with random close to bounds values
int32_t offset = 0;
std::vector<Cluster> clusters(TOTAL_CLUSTERS);
for (int32_t i = 0; i < TOTAL_CLUSTERS; i++) {
Cluster c;
c.x = (INT16_MAX - offset);
c.y = (INT16_MAX - (offset + 200));
for (int32_t j = 0; j < 9; j++) {
if (j % 2 == 0)
c.set<int32_t>(j, -(offset * 2));
else
c.set<int32_t>(j, (offset * 2));
}
clusters[i] = c;
offset++;
offset %= INT16_MAX - 200;
}
cf_out.write(clusters);
REQUIRE(cf_out.count() == TOTAL_CLUSTERS);
REQUIRE(cf_out.frame() == FRAME_NUMBER);
cf_out.update_header();
REQUIRE(cf_out.count() == TOTAL_CLUSTERS);
REQUIRE(cf_out.frame() == FRAME_NUMBER);
auto data_file = test_data_path() / "clusters" / "test_writing.clust";
REQUIRE(aare::compare_files(fpath_out, data_file));
}

6
src/file_io/test/NumpyFile.test.cpp
View File

@ -4,7 +4,7 @@
#include "test_config.hpp"
using aare::DType;
using aare::Dtype;
using aare::NumpyFile;
TEST_CASE("Read a 1D numpy file with int32 data type") {
@ -14,7 +14,7 @@ TEST_CASE("Read a 1D numpy file with int32 data type") {
NumpyFile f(fpath);
// we know the file contains 10 elements of np.int32 containing values 0-9
REQUIRE(f.dtype() == DType::INT32);
REQUIRE(f.dtype() == Dtype::INT32);
REQUIRE(f.shape() == std::vector<size_t>{10});
// use the load function to read the full file into a NDArray
@ -32,7 +32,7 @@ TEST_CASE("Read a 3D numpy file with np.double data type") {
NumpyFile f(fpath);
// we know the file contains 10 elements of np.int32 containing values 0-9
REQUIRE(f.dtype() == DType::DOUBLE);
REQUIRE(f.dtype() == Dtype::DOUBLE);
REQUIRE(f.shape() == std::vector<size_t>{3, 2, 5});
// use the load function to read the full file into a NDArray

20
src/file_io/test/NumpyHelpers.test.cpp
View File

@ -31,18 +31,18 @@ TEST_CASE("trim whitespace") {
}
TEST_CASE("parse data type descriptions") {
REQUIRE(parse_descr("<i1") == aare::DType::INT8);
REQUIRE(parse_descr("<i2") == aare::DType::INT16);
REQUIRE(parse_descr("<i4") == aare::DType::INT32);
REQUIRE(parse_descr("<i8") == aare::DType::INT64);
REQUIRE(parse_descr("<i1") == aare::Dtype::INT8);
REQUIRE(parse_descr("<i2") == aare::Dtype::INT16);
REQUIRE(parse_descr("<i4") == aare::Dtype::INT32);
REQUIRE(parse_descr("<i8") == aare::Dtype::INT64);
REQUIRE(parse_descr("<u1") == aare::DType::UINT8);
REQUIRE(parse_descr("<u2") == aare::DType::UINT16);
REQUIRE(parse_descr("<u4") == aare::DType::UINT32);
REQUIRE(parse_descr("<u8") == aare::DType::UINT64);
REQUIRE(parse_descr("<u1") == aare::Dtype::UINT8);
REQUIRE(parse_descr("<u2") == aare::Dtype::UINT16);
REQUIRE(parse_descr("<u4") == aare::Dtype::UINT32);
REQUIRE(parse_descr("<u8") == aare::Dtype::UINT64);
REQUIRE(parse_descr("<f4") == aare::DType::FLOAT);
REQUIRE(parse_descr("<f8") == aare::DType::DOUBLE);
REQUIRE(parse_descr("<f4") == aare::Dtype::FLOAT);
REQUIRE(parse_descr("<f8") == aare::Dtype::DOUBLE);
}
TEST_CASE("is element in array") {

4
src/file_io/test/RawFile.test.cpp
View File

@ -77,7 +77,7 @@ TEST_CASE("Compare reading from a numpy file with a raw file") {
for (size_t i = 0; i < 10; ++i) {
auto raw_frame = raw.read();
auto npy_frame = npy.read();
CHECK(raw_frame.view<uint16_t>() == npy_frame.view<uint16_t>());
CHECK((raw_frame.view<uint16_t>() == npy_frame.view<uint16_t>()));
}
}
@ -110,5 +110,5 @@ TEST_CASE("Read file with unordered frames") {
auto fpath = test_data_path() / "mythen" / "scan242_master_3.raw";
REQUIRE(std::filesystem::exists(fpath));
File f(fpath, "r");
REQUIRE_THROWS(f.read());
REQUIRE_THROWS((f.read()));
}

19
src/network_io/include/aare/network_io/defs.hpp
View File

@ -12,13 +12,16 @@ namespace aare {
* wrapper class to contain a ZmqHeader and a Frame
*/
struct ZmqFrame {
ZmqHeader header;
Frame frame;
ZmqFrame(const ZmqHeader& header_, const Frame& frame_) : header(header_), frame(frame_) {}
const ZmqHeader& header;
const Frame& frame;
std::string to_string() const {
return "ZmqFrame{header: " + header.to_string() + ", frame:\nrows: " + std::to_string(frame.rows()) +
", cols: " + std::to_string(frame.cols()) + ", bitdepth: " + std::to_string(frame.bitdepth()) + "\n}";
}
size_t size() const { return frame.size() + header.size; }
size_t size() const { return frame.bytes() + header.size; }
};
struct Task {
@ -58,8 +61,15 @@ struct Task {
PEDESTAL_AND_CLUSTER_AND_SAVE,
COUNT,
};
} __attribute__((packed));
#ifdef AARE_MSVC
} __declspec(align(1));
// msvc does not support packed attribute
// TODO: check if this is the correct way to do this in msvc
// maybe use #pragma pack(push, 1) and #pragma pack(pop)
#else
} __attribute__((packed));
#endif
namespace network_io {
/**
* @brief NetworkError exception class
@ -73,7 +83,6 @@ class NetworkError : public std::runtime_error {
explicit NetworkError(const std::string &msg) : std::runtime_error(msg), m_msg(strdup(msg.c_str())) {}
const char *what() const noexcept override { return m_msg; }
};
} // namespace network_io
} // namespace aare

6
src/network_io/src/ZmqMultiReceiver.cpp
View File

@ -51,7 +51,7 @@ ZmqFrame ZmqMultiReceiver::receive_zmqframe_(std::unordered_map<uint64_t, std::v
if (items[i].revents & ZMQ_POLLIN) {
auto new_frame = m_receivers[i]->receive_zmqframe();
if (frames_map.find(new_frame.header.frameNumber) == frames_map.end()) {
frames_map[new_frame.header.frameNumber] = {};
frames_map[new_frame.header.frameNumber] = std::vector<ZmqFrame>();
}
ret_frames = frames_map.find(new_frame.header.frameNumber);
@ -66,7 +66,7 @@ ZmqFrame ZmqMultiReceiver::receive_zmqframe_(std::unordered_map<uint64_t, std::v
}
std::vector<ZmqFrame> &frames = ret_frames->second;
if (!frames[0].header.data) {
return ZmqFrame{frames[0].header, Frame(0, 0, 0)};
return ZmqFrame{frames[0].header, Frame(0, 0, Dtype::NONE)};
}
// check that all frames have the same shape
auto shape = frames[0].header.shape;
@ -86,7 +86,7 @@ ZmqFrame ZmqMultiReceiver::receive_zmqframe_(std::unordered_map<uint64_t, std::v
for (auto &zmq_frame : frames) {
part_buffers.push_back(zmq_frame.frame.data());
}
Frame const f(shape.row, shape.col, bitdepth);
Frame const f(shape.row, shape.col, Dtype::from_bitdepth(bitdepth));
merge_frames(part_buffers, part_size, f.data(), m_geometry, shape.row, shape.col, bitdepth);
ZmqFrame zmq_frame = {std::move(frames[0].header), f};
return zmq_frame;

7
src/network_io/src/ZmqSocketReceiver.cpp
View File

@ -102,7 +102,8 @@ ZmqFrame ZmqSocketReceiver::receive_zmqframe() {
if (!header.data) {
// no data following header
return {header, Frame(0, 0, 0)};
Frame frame(0, 0, Dtype::NONE);
return {header, frame};
}
// receive frame data
@ -112,8 +113,8 @@ ZmqFrame ZmqSocketReceiver::receive_zmqframe() {
if (header.bitmode == 0) {
header.bitmode = 16;
}
Frame frame(header.shape.row, header.shape.col, header.bitmode);
int bytes_received = receive_data(frame.data(), frame.size());
Frame frame(header.shape.row, header.shape.col, Dtype::from_bitdepth(header.bitmode));
int bytes_received = receive_data(frame.data(), frame.bytes());
if (bytes_received == -1) {
throw network_io::NetworkError(LOCATION + "Error receiving frame");
}

6
src/network_io/src/ZmqSocketSender.cpp
View File

@ -84,7 +84,7 @@ size_t ZmqSocketSender::send(const ZmqHeader &header, const void *data, size_t s
size_t ZmqSocketSender::send(const ZmqFrame &zmq_frame) {
const Frame &frame = zmq_frame.frame;
// send frame
size_t const rc = send(zmq_frame.header, frame.data(), frame.size());
size_t const rc = send(zmq_frame.header, frame.data(), frame.bytes());
// send end of message header
ZmqHeader end_header = zmq_frame.header;
end_header.data = false;
@ -106,10 +106,10 @@ size_t ZmqSocketSender::send(const std::vector<ZmqFrame> &zmq_frames) {
// send header and frame
if (i < zmq_frames.size() - 1) {
// send header and frame
rc += send(header, frame.data(), frame.size());
rc += send(header, frame.data(), frame.bytes());
} else {
// send header, frame and end of message header
rc += send({header, frame});
rc += send(ZmqFrame(header, frame));
}
}
return rc;

2
src/network_io/src/ZmqWorker.cpp
View File

@ -8,7 +8,7 @@ namespace aare {
ZmqWorker::ZmqWorker(const std::string &ventilator_endpoint, const std::string &sink_endpoint)
: m_receiver(new ZmqSocketReceiver(ventilator_endpoint, ZMQ_PULL)), m_sender(nullptr) {
m_receiver->connect();
if (not sink_endpoint.empty()) {
if (!sink_endpoint.empty()) {
m_sender = new ZmqSocketSender(sink_endpoint, ZMQ_PUSH);
m_sender->connect();
}

41
src/processing/include/aare/processing/ClusterFinder.hpp
View File

@ -1,5 +1,5 @@
#pragma once
#include "aare/core/DType.hpp"
#include "aare/core/Dtype.hpp"
#include "aare/core/NDArray.hpp"
#include "aare/core/NDView.hpp"
#include "aare/processing/Pedestal.hpp"
@ -28,7 +28,15 @@ class ClusterFinder {
};
template <typename FRAME_TYPE, typename PEDESTAL_TYPE>
std::vector<Cluster> find_clusters(NDView<FRAME_TYPE, 2> frame, Pedestal<PEDESTAL_TYPE> &pedestal) {
std::vector<Cluster> find_clusters_without_threshold(NDView<FRAME_TYPE, 2> frame, Pedestal<PEDESTAL_TYPE> &pedestal,
bool late_update = false) {
struct pedestal_update {
int x;
int y;
FRAME_TYPE value;
};
std::vector<pedestal_update> pedestal_updates;
std::vector<Cluster> clusters;
std::vector<std::vector<eventType>> eventMask;
for (int i = 0; i < frame.shape(0); i++) {
@ -57,7 +65,7 @@ class ClusterFinder {
}
auto rms = pedestal.standard_deviation(iy, ix);
if (frame(iy, ix) < -m_nSigma * rms) {
if (frame(iy, ix) - pedestal.mean(iy, ix) < -m_nSigma * rms) {
eventMask[iy][ix] = NEGATIVE_PEDESTAL;
continue;
} else if (max > m_nSigma * rms) {
@ -65,21 +73,27 @@ class ClusterFinder {
} else if (total > c3 * m_nSigma * rms) {
eventMask[iy][ix] = PHOTON;
} else {
pedestal.push(iy, ix, frame(iy, ix));
} else{
if (late_update) {
pedestal_updates.push_back({ix, iy, frame(iy, ix)});
} else {
pedestal.push(iy, ix, frame(iy, ix));
}
continue;
}
if (eventMask[iy][ix] == PHOTON and frame(iy, ix) - pedestal.mean(iy, ix) >= max) {
if (eventMask[iy][ix] == PHOTON && (frame(iy, ix) - pedestal.mean(iy, ix)) >= max) {
eventMask[iy][ix] = PHOTON_MAX;
Cluster cluster(m_cluster_sizeX, m_cluster_sizeY, DType(typeid(FRAME_TYPE)));
Cluster cluster(m_cluster_sizeX, m_cluster_sizeY, Dtype(typeid(PEDESTAL_TYPE)));
cluster.x = ix;
cluster.y = iy;
short i = 0;
for (short ir = -(m_cluster_sizeY / 2); ir < (m_cluster_sizeY / 2) + 1; ir++) {
for (short ic = -(m_cluster_sizeX / 2); ic < (m_cluster_sizeX / 2) + 1; ic++) {
if (ix + ic >= 0 && ix + ic < frame.shape(1) && iy + ir >= 0 && iy + ir < frame.shape(0)) {
FRAME_TYPE tmp = frame(iy + ir, ix + ic) - pedestal.mean(iy + ir, ix + ic);
cluster.set<FRAME_TYPE>(i, tmp);
PEDESTAL_TYPE tmp = static_cast<PEDESTAL_TYPE>(frame(iy + ir, ix + ic)) -
pedestal.mean(iy + ir, ix + ic);
cluster.set<PEDESTAL_TYPE>(i, tmp);
i++;
}
}
@ -88,6 +102,11 @@ class ClusterFinder {
}
}
}
if (late_update) {
for (auto &update : pedestal_updates) {
pedestal.push(update.y, update.x, update.value);
}
}
return clusters;
}
@ -163,9 +182,9 @@ class ClusterFinder {
pedestal.push(iy, ix, frame(iy, ix));
continue;
}
if (eventMask[iy][ix] == PHOTON and frame(iy, ix) - pedestal.mean(iy, ix) == max) {
if (eventMask[iy][ix] == PHOTON && frame(iy, ix) - pedestal.mean(iy, ix) >= max) {
eventMask[iy][ix] = PHOTON_MAX;
Cluster cluster(m_cluster_sizeX, m_cluster_sizeY, DType(typeid(FRAME_TYPE)));
Cluster cluster(m_cluster_sizeX, m_cluster_sizeY, Dtype(typeid(FRAME_TYPE)));
cluster.x = ix;
cluster.y = iy;
short i = 0;

34
src/processing/include/aare/processing/Pedestal.hpp
View File

@ -8,14 +8,14 @@ namespace aare {
template <typename SUM_TYPE = double> class Pedestal {
public:
Pedestal(int rows, int cols, int n_samples = 1000);
Pedestal(uint32_t rows, uint32_t cols, uint32_t n_samples = 1000);
~Pedestal();
// frame level operations
template <typename T> void push(NDView<T, 2> frame) {
assert(frame.size() == m_rows * m_cols);
// TODO: test the effect of #pragma omp parallel for
for (int index = 0; index < m_rows * m_cols; index++) {
for (uint32_t index = 0; index < m_rows * m_cols; index++) {
push<T>(index / m_cols, index % m_cols, frame(index));
}
}
@ -29,17 +29,19 @@ template <typename SUM_TYPE = double> class Pedestal {
void clear();
// getter functions
inline int rows() const { return m_rows; }
inline int cols() const { return m_cols; }
inline int n_samples() const { return m_samples; }
inline uint32_t rows() const { return m_rows; }
inline uint32_t cols() const { return m_cols; }
inline uint32_t n_samples() const { return m_samples; }
inline uint32_t *cur_samples() const { return m_cur_samples; }
inline NDArray<SUM_TYPE, 2> get_sum() const { return m_sum; }
inline NDArray<SUM_TYPE, 2> get_sum2() const { return m_sum2; }
// pixel level operations (should be refactored to allow users to implement their own pixel level operations)
template <typename T> inline void push(const int row, const int col, const T val) {
const int idx = index(row, col);
template <typename T> inline void push(const uint32_t row, const uint32_t col, const T val) {
if (m_freeze) {
return;
}
const uint32_t idx = index(row, col);
if (m_cur_samples[idx] < m_samples) {
m_sum(idx) += val;
m_sum2(idx) += val * val;
@ -49,15 +51,17 @@ template <typename SUM_TYPE = double> class Pedestal {
m_sum2(idx) += val * val - m_sum2(idx) / m_cur_samples[idx];
}
}
SUM_TYPE mean(const int row, const int col) const;
SUM_TYPE variance(const int row, const int col) const;
SUM_TYPE standard_deviation(const int row, const int col) const;
inline int index(const int row, const int col) const { return row * m_cols + col; };
void clear(const int row, const int col);
SUM_TYPE mean(const uint32_t row, const uint32_t col) const;
SUM_TYPE variance(const uint32_t row, const uint32_t col) const;
SUM_TYPE standard_deviation(const uint32_t row, const uint32_t col) const;
inline uint32_t index(const uint32_t row, const uint32_t col) const { return row * m_cols + col; };
void clear(const uint32_t row, const uint32_t col);
void set_freeze(bool freeze) { m_freeze = freeze; }
private:
int m_rows;
int m_cols;
uint32_t m_rows;
uint32_t m_cols;
bool m_freeze;
uint32_t m_samples;
uint32_t *m_cur_samples{nullptr};
NDArray<SUM_TYPE, 2> m_sum;

22
src/processing/src/Pedestal.cpp
View File

@ -4,9 +4,9 @@
namespace aare {
template <typename SUM_TYPE>
Pedestal<SUM_TYPE>::Pedestal(int rows, int cols, int n_samples)
: m_rows(rows), m_cols(cols), m_samples(n_samples), m_sum(NDArray<SUM_TYPE, 2>({cols, rows})),
m_sum2(NDArray<SUM_TYPE, 2>({cols, rows})), m_cur_samples(new uint32_t[static_cast<uint64_t>(rows) * cols]{}) {
Pedestal<SUM_TYPE>::Pedestal(uint32_t rows, uint32_t cols, uint32_t n_samples)
: m_freeze(false), m_rows(rows), m_cols(cols), m_samples(n_samples), m_sum(NDArray<SUM_TYPE, 2>({rows, cols})),
m_sum2(NDArray<SUM_TYPE, 2>({rows, cols})), m_cur_samples(new uint32_t[rows * cols]{}) {
assert(rows > 0 && cols > 0 && n_samples > 0);
m_sum = 0;
m_sum2 = 0;
@ -14,7 +14,7 @@ Pedestal<SUM_TYPE>::Pedestal(int rows, int cols, int n_samples)
template <typename SUM_TYPE> NDArray<SUM_TYPE, 2> Pedestal<SUM_TYPE>::mean() {
NDArray<SUM_TYPE, 2> mean_array({m_rows, m_cols});
for (int i = 0; i < m_rows * m_cols; i++) {
for (uint32_t i = 0; i < m_rows * m_cols; i++) {
mean_array(i / m_cols, i % m_cols) = mean(i / m_cols, i % m_cols);
}
return mean_array;
@ -22,7 +22,7 @@ template <typename SUM_TYPE> NDArray<SUM_TYPE, 2> Pedestal<SUM_TYPE>::mean() {
template <typename SUM_TYPE> NDArray<SUM_TYPE, 2> Pedestal<SUM_TYPE>::variance() {
NDArray<SUM_TYPE, 2> variance_array({m_rows, m_cols});
for (int i = 0; i < m_rows * m_cols; i++) {
for (uint32_t i = 0; i < m_rows * m_cols; i++) {
variance_array(i / m_cols, i % m_cols) = variance(i / m_cols, i % m_cols);
}
return variance_array;
@ -30,14 +30,14 @@ template <typename SUM_TYPE> NDArray<SUM_TYPE, 2> Pedestal<SUM_TYPE>::variance()
template <typename SUM_TYPE> NDArray<SUM_TYPE, 2> Pedestal<SUM_TYPE>::standard_deviation() {
NDArray<SUM_TYPE, 2> standard_deviation_array({m_rows, m_cols});
for (int i = 0; i < m_rows * m_cols; i++) {
for (uint32_t i = 0; i < m_rows * m_cols; i++) {
standard_deviation_array(i / m_cols, i % m_cols) = standard_deviation(i / m_cols, i % m_cols);
}
return standard_deviation_array;
}
template <typename SUM_TYPE> void Pedestal<SUM_TYPE>::clear() {
for (int i = 0; i < m_rows * m_cols; i++) {
for (uint32_t i = 0; i < m_rows * m_cols; i++) {
clear(i / m_cols, i % m_cols);
}
}
@ -45,23 +45,23 @@ template <typename SUM_TYPE> void Pedestal<SUM_TYPE>::clear() {
/*
* index level operations
*/
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::mean(const int row, const int col) const {
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::mean(const uint32_t row, const uint32_t col) const {
if (m_cur_samples[index(row, col)] == 0) {
return 0.0;
}
return m_sum(row, col) / m_cur_samples[index(row, col)];
}
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::variance(const int row, const int col) const {
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::variance(const uint32_t row, const uint32_t col) const {
if (m_cur_samples[index(row, col)] == 0) {
return 0.0;
}
return m_sum2(row, col) / m_cur_samples[index(row, col)] - mean(row, col) * mean(row, col);
}
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::standard_deviation(const int row, const int col) const {
template <typename SUM_TYPE> SUM_TYPE Pedestal<SUM_TYPE>::standard_deviation(const uint32_t row, const uint32_t col) const {
return std::sqrt(variance(row, col));
}
template <typename SUM_TYPE> void Pedestal<SUM_TYPE>::clear(const int row, const int col) {
template <typename SUM_TYPE> void Pedestal<SUM_TYPE>::clear(const uint32_t row, const uint32_t col) {
m_sum(row, col) = 0;
m_sum2(row, col) = 0;
m_cur_samples[index(row, col)] = 0;

4
src/processing/test/ClusterFinder.test.cpp
View File

@ -37,12 +37,12 @@ TEST_CASE("test cluster finder") {
frame = 0;
ClusterFinder clusterFinder(3, 3, 1, 1); // 3x3 cluster, 1 nSigma, 1 threshold
auto clusters = clusterFinder.find_clusters(frame.span(), pedestal);
auto clusters = clusterFinder.find_clusters_without_threshold(frame.span(), pedestal);
REQUIRE(clusters.size() == 0);
frame(5, 5) = 10;
clusters = clusterFinder.find_clusters(frame.span(), pedestal);
clusters = clusterFinder.find_clusters_without_threshold(frame.span(), pedestal);
REQUIRE(clusters.size() == 1);
REQUIRE(clusters[0].x == 5);
REQUIRE(clusters[0].y == 5);

10
src/processing/test/Pedestal.test.cpp
View File

@ -22,7 +22,7 @@ TEST_CASE("test pedestal constructor") {
TEST_CASE("test pedestal push") {
aare::Pedestal pedestal(10, 10, 5);
aare::Frame frame(10, 10, 16);
aare::Frame frame(10, 10, Dtype::UINT16);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
frame.set<uint16_t>(i, j, i + j);
@ -50,10 +50,10 @@ TEST_CASE("test pedestal push") {
}
// test number of samples after multiple push
for (int k = 0; k < 50; k++) {
for (uint32_t k = 0; k < 50; k++) {
pedestal.push<uint16_t>(frame);
for (int i = 0; i < 10; i++) {
for (int j = 0; j < 10; j++) {
for (uint32_t i = 0; i < 10; i++) {
for (uint32_t j = 0; j < 10; j++) {
if (k < 5) {
REQUIRE(pedestal.cur_samples()[pedestal.index(i, j)] == k + 1);
REQUIRE(pedestal.get_sum()(i, j) == (k + 1) * (i + j));
@ -80,7 +80,7 @@ TEST_CASE("test pedestal with normal distribution") {
aare::Pedestal pedestal(3, 5, 10000);
for (int i = 0; i < 10000; i++) {
aare::Frame frame(3, 5, 64);
aare::Frame frame(3, 5, Dtype::DOUBLE);
for (int j = 0; j < 3; j++) {
for (int k = 0; k < 5; k++) {
frame.set<double>(j, k, distribution(generator));

15
src/python/CMakeLists.txt
View File

@ -18,8 +18,19 @@ endif()
pybind11_add_module(_aare src/bindings.cpp)
pybind11_add_module(_aare cpp/bindings.cpp)
set_target_properties(_aare PROPERTIES
LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}
)
target_link_libraries(_aare PRIVATE aare aare_compiler_flags)
target_link_libraries(_aare PRIVATE aare aare_compiler_flags)
if(AARE_TESTS)
set(TestSources
${CMAKE_CURRENT_SOURCE_DIR}/test/DtypeFormatDescr.test.cpp
)
target_sources(tests PRIVATE ${TestSources} )
target_link_libraries(tests PRIVATE core pybind11::embed)
endif()

68
src/python/aare/File.py
View File

@ -1,68 +0,0 @@
import json
from typing import Any
import _aare
import os
class File:
"""
File class. uses proxy pattern to wrap around the pybinding class
abstracts the python binding class that is requires type and detector information
(e.g. _FileHandler_Jungfrau_16)
"""
def __init__(self, path):
"""
opens the master file and checks the dynamic range and detector
"""
self.path = path
# check if file exists
if not os.path.exists(path):
raise FileNotFoundError(f"File not found: {path}")
ext = os.path.splitext(path)[1]
if ext not in (".raw", ".json", ".npy"):
raise ValueError(f"Invalid file extension: {ext}")
if ext == ".json":
# read the master file and get the detector and bitdepth
master_data = json.load(open(path))
detector = master_data["Detector Type"]
bitdepth = None
if 'Dynamic Range' not in master_data and detector == "Jungfrau":
bitdepth = 16
else:
bitdepth = master_data["Dynamic Range"]
elif ext == ".npy":
# TODO: find solution for this. maybe add a None detector type
detector = "Jungfrau"
with open(path, "rb") as fobj:
import numpy as np
version = np.lib.format.read_magic(fobj)
# find what function to call based on the version
func_name = 'read_array_header_' + '_'.join(str(v) for v in version)
func = getattr(np.lib.format, func_name)
header = func(fobj)
bitdepth = header[2].itemsize * 8
else:
NotImplementedError("Raw file not implemented yet")
# class_name is of the form _FileHandler_Jungfrau_16...
class_name = f"_FileHandler_{detector}_{bitdepth}"
# this line is the equivalent of:
# self._file = _FileHandler_Jungfrau_16(path)
self._file = getattr(_aare, class_name)(path)
def __getattribute__(self, __name: str) -> Any:
"""
Proxy pattern to call the methods of the _file
"""
return getattr(object.__getattribute__(self, "_file"), __name)

16
src/python/aare/Frame.py
View File

@ -1,16 +0,0 @@
from typing import Any
class Frame:
"""
Frame class. uses proxy pattern to wrap around the pybinding class
the intention behind it is to only use one class for frames in python (not Frame_8, Frame_16, etc)
"""
def __init__(self, frameImpl):
self._frameImpl = frameImpl
def __getattribute__(self, __name: str) -> Any:
"""
Proxy pattern to call the methods of the frameImpl
"""
return getattr(object.__getattribute__(self, "_frameImpl"), __name)

3
src/python/aare/__init__.py
View File

@ -1,3 +0,0 @@
from _aare import *
from .Frame import Frame
from .File import File

61
src/python/cpp/NDArray_bindings.hpp Normal file
View File

@ -0,0 +1,61 @@
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "aare/core/Dtype.hpp"
#include "aare/core/NDArray.hpp"
#include "aare/core/NDView.hpp"
namespace py = pybind11;
using namespace aare;
using namespace std;
template <typename ArrayType, int64_t Ndim> void define_NDArray_bindings(py::module_ &m) {
std::string name = "NDArray_" + Dtype(typeid(ArrayType)).to_string() + "_" + to_string(Ndim);
py::class_<NDArray<ArrayType, Ndim>>(m, name.c_str(), py::buffer_protocol())
.def(py::init<array<int64_t, Ndim>>())
.def(py::init([](py::array_t<ArrayType, py::array::c_style | py::array::forcecast> &np_array) {
py::buffer_info info = np_array.request();
if (info.format != py::format_descriptor<ArrayType>::format())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != Ndim)
throw std::runtime_error("Incompatible dimension: expected a" + to_string(Ndim) + " array!");
std::array<int64_t, Ndim> arr_shape;
std::move(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDArray<ArrayType, Ndim> a(arr_shape);
std::memcpy(a.data(), info.ptr, info.size * sizeof(ArrayType));
return a;
}))
.def("__getitem__",
[](NDArray<ArrayType, Ndim> &a, py::tuple index) {
if (index.size() != 2) {
throw std::runtime_error("Index must be a tuple of size " + to_string(Ndim));
}
auto offset = 0;
for (size_t i = 0; i < Ndim; i++) {
offset += index[i].cast<int64_t>() * a.strides()[i];
}
return a(offset);
})
.def_property_readonly("shape", [](NDArray<ArrayType, Ndim> &a) { return a.shape(); })
.def_property_readonly("size", &NDArray<ArrayType, Ndim>::size)
.def_property_readonly("bitdepth", &NDArray<ArrayType, Ndim>::bitdepth)
.def_property_readonly("strides", &NDArray<ArrayType, Ndim>::strides)
.def_property_readonly("byte_strides", &NDArray<ArrayType, Ndim>::byte_strides)
.def("__add__", [](NDArray<ArrayType, Ndim> &a, NDArray<ArrayType, Ndim> &b) { return a + b; })
.def_buffer([](NDArray<ArrayType, Ndim> &a) -> py::buffer_info {
return py::buffer_info(
a.data(), /* Pointer to buffer */
sizeof(ArrayType), /* Size of one scalar */
py::format_descriptor<ArrayType>::format(), /* Python struct-style format descriptor */
Ndim, /* Number of dimensions */
a.shape(), /* Buffer dimensions */
a.byte_strides() /* Strides (in bytes) for each index */
);
});
}

54
src/python/cpp/NDView_bindings.hpp Normal file
View File

@ -0,0 +1,54 @@
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "aare/core/Dtype.hpp"
#include "aare/core/NDView.hpp"
namespace py = pybind11;
using namespace aare;
using namespace std;
template <typename ArrayType, int64_t Ndim> void define_NDView_bindings(py::module_ &m) {
std::string name = "NDView_" + Dtype(typeid(ArrayType)).to_string() + "_" + to_string(Ndim);
py::class_<NDView<ArrayType, Ndim>>(m, name.c_str(), py::buffer_protocol())
.def(py::init([](py::array_t<ArrayType, py::array::c_style | py::array::forcecast> &np_array) {
py::buffer_info info = np_array.request();
if (info.format != py::format_descriptor<ArrayType>::format())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != Ndim)
throw std::runtime_error("Incompatible dimension: expected a" + to_string(Ndim) + " array!");
std::array<int64_t, Ndim> arr_shape;
std::move(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDView<ArrayType, Ndim> a(static_cast<ArrayType *>(info.ptr), arr_shape);
return a;
}))
.def("__getitem__",
[](NDView<ArrayType, Ndim> &a, py::tuple index) {
if (index.size() != 2) {
throw std::runtime_error("Index must be a tuple of size " + to_string(Ndim));
}
auto offset = 0;
for (size_t i = 0; i < Ndim; i++) {
offset += index[i].cast<int64_t>() * a.strides()[i];
}
return a(offset);
})
.def_property_readonly("shape", [](NDView<ArrayType, Ndim> &a) { return a.shape(); })
.def_property_readonly("size", &NDView<ArrayType, Ndim>::size)
.def_buffer([](NDView<ArrayType, Ndim> &a) -> py::buffer_info {
return py::buffer_info(
a.data(), /* Pointer to buffer */
sizeof(ArrayType), /* Size of one scalar */
py::format_descriptor<ArrayType>::format(), /* Python struct-style format descriptor */
Ndim, /* Number of dimensions */
a.shape(), /* Buffer dimensions */
a.strides() /* Strides (in bytes) for each index */
);
});
}

49
src/python/cpp/bindings.cpp Normal file
View File

@ -0,0 +1,49 @@
#include <cstdint>
#include <filesystem>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include "aare/core/Frame.hpp"
#include "aare/core/defs.hpp"
#include "aare/file_io/File.hpp"
#include "NDArray_bindings.hpp"
#include "NDView_bindings.hpp"
#include "core.hpp"
#include "file_io.hpp"
#include "processing.hpp"
namespace py = pybind11;
PYBIND11_MODULE(_aare, m) {
// helps to convert from std::string to std::filesystem::path
py::class_<std::filesystem::path>(m, "Path").def(py::init<std::string>());
py::implicitly_convertible<std::string, std::filesystem::path>();
define_core_bindings(m);
define_file_io_bindings(m);
define_processing_bindings(m);
define_NDArray_bindings<double, 2>(m);
define_NDArray_bindings<float, 2>(m);
define_NDArray_bindings<uint8_t, 2>(m);
define_NDArray_bindings<uint16_t, 2>(m);
define_NDArray_bindings<uint32_t, 2>(m);
define_NDArray_bindings<uint64_t, 2>(m);
define_NDArray_bindings<int8_t, 2>(m);
define_NDArray_bindings<int16_t, 2>(m);
define_NDArray_bindings<int32_t, 2>(m);
define_NDArray_bindings<int64_t, 2>(m);
define_NDView_bindings<double, 2>(m);
define_NDView_bindings<float, 2>(m);
define_NDView_bindings<uint8_t, 2>(m);
define_NDView_bindings<uint16_t, 2>(m);
define_NDView_bindings<uint32_t, 2>(m);
define_NDView_bindings<uint64_t, 2>(m);
define_NDView_bindings<int8_t, 2>(m);
define_NDView_bindings<int16_t, 2>(m);
define_NDView_bindings<int32_t, 2>(m);
define_NDView_bindings<int64_t, 2>(m);
}

159
src/python/cpp/core.hpp Normal file
View File

@ -0,0 +1,159 @@
#include <cstdint>
#include <filesystem>
#include <pybind11/functional.h>
#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include "aare/core/Frame.hpp"
#include "aare/core/Transforms.hpp"
#include "aare/core/defs.hpp"
template <typename T> void define_to_frame(py::module &m) {
m.def("to_frame", [](py::array_t<T> &np_array) {
py::buffer_info info = np_array.request();
if (info.format != py::format_descriptor<T>::format())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != 2)
throw std::runtime_error("Incompatible dimension: expected a 2D array!");
Frame f(info.shape[0], info.shape[1], Dtype(typeid(T)));
std::memcpy(f.data(), info.ptr, f.bytes());
return f;
});
}
void define_core_bindings(py::module &m) {
py::class_<Frame>(m, "Frame", py::buffer_protocol())
.def(py::init<std::byte *, int64_t, int64_t, Dtype>())
.def(py::init<int64_t, int64_t, Dtype>())
.def_property_readonly("rows", &Frame::rows)
.def_property_readonly("cols", &Frame::cols)
.def_property_readonly("bitdepth", &Frame::bitdepth)
.def_property_readonly("size", &Frame::bytes)
.def_property_readonly("data", &Frame::data, py::return_value_policy::reference)
.def_buffer([](Frame &f) -> py::buffer_info {
Dtype dt = f.dtype();
return {
f.data(), /* Pointer to buffer */
static_cast<int64_t>(dt.bytes()), /* Size of one scalar */
dt.format_descr(), /* Python struct-style format descriptor */
2, /* Number of dimensions */
{f.rows(), f.cols()}, /* Buffer dimensions */
{f.cols() * dt.bytes(), dt.bytes()} /* Strides (in bytes) for each index */
};
});
py::class_<xy>(m, "xy")
.def(py::init<>())
.def(py::init<uint32_t, uint32_t>())
.def_readwrite("row", &xy::row)
.def_readwrite("col", &xy::col)
.def("__eq__", &xy::operator==)
.def("__ne__", &xy::operator!=)
.def("__repr__",
[](const xy &a) { return "<xy: row=" + std::to_string(a.row) + ", col=" + std::to_string(a.col) + ">"; });
py::enum_<DetectorType>(m, "DetectorType")
.value("Jungfrau", DetectorType::Jungfrau)
.value("Eiger", DetectorType::Eiger)
.value("Mythen3", DetectorType::Mythen3)
.value("Moench", DetectorType::Moench)
.value("ChipTestBoard", DetectorType::ChipTestBoard)
.value("Unknown", DetectorType::Unknown)
.export_values();
py::enum_<TimingMode>(m, "TimingMode")
.value("Auto", TimingMode::Auto)
.value("Trigger", TimingMode::Trigger)
.export_values();
py::enum_<endian>(m, "endian")
.value("big", endian::big)
.value("little", endian::little)
.value("native", endian::native)
.export_values();
py::enum_<Dtype::TypeIndex>(m, "DtypeIndex")
.value("INT8", Dtype::TypeIndex::INT8)
.value("UINT8", Dtype::TypeIndex::UINT8)
.value("INT16", Dtype::TypeIndex::INT16)
.value("UINT16", Dtype::TypeIndex::UINT16)
.value("INT32", Dtype::TypeIndex::INT32)
.value("UINT32", Dtype::TypeIndex::UINT32)
.value("INT64", Dtype::TypeIndex::INT64)
.value("UINT64", Dtype::TypeIndex::UINT64)
.value("FLOAT", Dtype::TypeIndex::FLOAT)
.value("DOUBLE", Dtype::TypeIndex::DOUBLE)
.value("ERROR", Dtype::TypeIndex::ERROR)
.export_values();
py::class_<Dtype>(m, "Dtype")
.def(py::init<std::string_view>())
.def(py::init<Dtype::TypeIndex>())
.def("bitdepth", &Dtype::bitdepth)
.def("bytes", &Dtype::bytes)
.def("to_string", &Dtype::to_string);
py::class_<sls_detector_header>(m, "sls_detector_header")
.def(py::init<>())
.def_readwrite("frameNumber", &sls_detector_header::frameNumber)
.def_readwrite("expLength", &sls_detector_header::expLength)
.def_readwrite("packetNumber", &sls_detector_header::packetNumber)
.def_readwrite("bunchId", &sls_detector_header::bunchId)
.def_readwrite("timestamp", &sls_detector_header::timestamp)
.def_readwrite("modId", &sls_detector_header::modId)
.def_readwrite("row", &sls_detector_header::row)
.def_readwrite("column", &sls_detector_header::column)
.def_readwrite("reserved", &sls_detector_header::reserved)
.def_readwrite("debug", &sls_detector_header::debug)
.def_readwrite("roundRNumber", &sls_detector_header::roundRNumber)
.def_readwrite("detType", &sls_detector_header::detType)
.def_readwrite("version", &sls_detector_header::version)
.def_readwrite("packetMask", &sls_detector_header::packetMask)
.def("__repr__", &sls_detector_header::to_string);
py::class_<Transforms>(m, "Transforms")
.def(py::init<>())
.def_static("identity", &Transforms::identity)
.def_static("zero", &Transforms::zero)
.def_static("reorder_moench", &Transforms::reorder_moench)
.def_static("reorder", py::overload_cast<NDView<uint64_t, 2> &>(&Transforms::reorder))
.def_static("reorder", py::overload_cast<NDArray<uint64_t, 2> &>(&Transforms::reorder))
.def_static("reorder", py::overload_cast<std::vector<uint64_t> &>(&Transforms::reorder))
.def_static(
// only accept order_map of type uint64_t
"reorder",
[](py::array_t<uint64_t> &np_array) {
py::buffer_info info = np_array.request();
if (info.format != Dtype(Dtype::UINT64).numpy_descr())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != 2)
throw std::runtime_error("Incompatible dimension: expected a 2D array!");
std::array<int64_t, 2> arr_shape;
std::move(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDView<uint64_t, 2> a(static_cast<uint64_t *>(info.ptr), arr_shape);
return Transforms::reorder(a);
})
.def_static("flip_horizental", &Transforms::flip_horizental)
.def("add", [](Transforms &self, std::function<Frame &(Frame &)> transformation) { self.add(transformation); })
.def("add", [](Transforms &self,
std::vector<std::function<Frame &(Frame &)>> transformations) { self.add(transformations); })
.def("__call__", &Transforms::apply);
define_to_frame<uint8_t>(m);
define_to_frame<uint16_t>(m);
define_to_frame<uint32_t>(m);
define_to_frame<uint64_t>(m);
define_to_frame<int8_t>(m);
define_to_frame<int16_t>(m);
define_to_frame<int32_t>(m);
define_to_frame<int64_t>(m);
define_to_frame<float>(m);
define_to_frame<double>(m);
}

95
src/python/cpp/file_io.hpp Normal file
View File

@ -0,0 +1,95 @@
#include "aare/core/Frame.hpp"
#include "aare/core/defs.hpp"
#include "aare/file_io/ClusterFileV2.hpp"
#include "aare/file_io/File.hpp"
#include <cstdint>
#include <filesystem>
#include <pybind11/iostream.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
namespace py = pybind11;
void define_file_io_bindings(py::module &m) {
py::class_<File>(m, "File")
.def(py::init([](const std::filesystem::path &fname) { return File(fname, "r", {}); }))
.def(
py::init([](const std::filesystem::path &fname, const std::string &mode) { return File(fname, mode, {}); }))
.def(py::init<const std::filesystem::path &, const std::string &, const FileConfig &>())
.def("read", py::overload_cast<>(&File::read))
.def("read", py::overload_cast<size_t>(&File::read))
.def("iread", py::overload_cast<size_t>(&File::iread))
.def("frame_number", &File::frame_number)
.def_property_readonly("bytes_per_frame", &File::bytes_per_frame)
.def_property_readonly("pixels_per_frame", &File::pixels_per_frame)
.def("seek", &File::seek)
.def("tell", &File::tell)
.def_property_readonly("total_frames", &File::total_frames)
.def_property_readonly("rows", &File::rows)
.def_property_readonly("cols", &File::cols)
.def_property_readonly("bitdepth", &File::bitdepth)
.def_property_readonly("detector_type", &File::detector_type)
.def_property_readonly("geometry", &File::geometry,
py::call_guard<py::scoped_ostream_redirect, py::scoped_estream_redirect>())
.def("set_total_frames", &File::set_total_frames);
py::class_<FileConfig>(m, "FileConfig")
.def(py::init<>())
.def_readwrite("rows", &FileConfig::rows)
.def_readwrite("cols", &FileConfig::cols)
.def_readwrite("version", &FileConfig::version)
.def_readwrite("geometry", &FileConfig::geometry)
.def_readwrite("detector_type", &FileConfig::detector_type)
.def_readwrite("max_frames_per_file", &FileConfig::max_frames_per_file)
.def_readwrite("total_frames", &FileConfig::total_frames)
.def_readwrite("dtype", &FileConfig::dtype)
.def("__eq__", &FileConfig::operator==)
.def("__ne__", &FileConfig::operator!=)
.def("__repr__", [](const FileConfig &a) { return "<FileConfig: " + a.to_string() + ">"; });
py::class_<ClusterHeader>(m, "ClusterHeader")
.def(py::init<>())
.def_readwrite("frame_number", &ClusterHeader::frame_number)
.def_readwrite("n_clusters", &ClusterHeader::n_clusters)
.def("__repr__", [](const ClusterHeader &a) { return "<ClusterHeader: " + a.to_string() + ">"; });
py::class_<ClusterV2_>(m, "ClusterV2_")
.def(py::init<>())
.def_readwrite("x", &ClusterV2_::x)
.def_readwrite("y", &ClusterV2_::y)
.def_readwrite("data", &ClusterV2_::data)
.def("__repr__", [](const ClusterV2_ &a) { return "<ClusterV2_: " + a.to_string(false) + ">"; });
py::class_<ClusterV2>(m, "ClusterV2")
.def(py::init<>())
.def_readwrite("cluster", &ClusterV2::cluster)
.def_readwrite("frame_number", &ClusterV2::frame_number)
.def("__repr__", [](const ClusterV2 &a) { return "<ClusterV2: " + a.to_string() + ">"; });
py::class_<ClusterFileV2>(m, "ClusterFileV2")
.def(py::init<const std::filesystem::path &, const std::string &>())
.def("read", py::overload_cast<>(&ClusterFileV2::read))
.def("read", py::overload_cast<int>(&ClusterFileV2::read))
.def("frame_number", &ClusterFileV2::frame_number)
.def("write", py::overload_cast<std::vector<ClusterV2> const &>(&ClusterFileV2::write))
.def("close", &ClusterFileV2::close);
m.def("to_clustV2", [](std::vector<Cluster> &clusters, const int frame_number) {
std::vector<ClusterV2> clusters_;
for (auto &c : clusters) {
ClusterV2 cluster;
cluster.cluster.x = c.x;
cluster.cluster.y = c.y;
int i=0;
for(auto &d : cluster.cluster.data) {
d=c.get<double>(i++);
}
cluster.frame_number = frame_number;
clusters_.push_back(cluster);
}
return clusters_;
});
}

160
src/python/cpp/processing.hpp Normal file
View File

@ -0,0 +1,160 @@
#include <cstdint>
#include <filesystem>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <string>
#include "aare/core/Frame.hpp"
#include "aare/core/NDView.hpp"
#include "aare/core/defs.hpp"
#include "aare/file_io/ClusterFileV2.hpp"
#include "aare/file_io/File.hpp"
#include "aare/processing/ClusterFinder.hpp"
#include "aare/processing/Pedestal.hpp"
namespace py = pybind11;
template <typename T, typename SUM_TYPE> void define_pedestal_push_bindings(py::class_<Pedestal<SUM_TYPE>> &p) {
p.def("push", [](Pedestal<SUM_TYPE> &pedestal, py::array_t<T> &np_array) {
py::buffer_info info = np_array.request();
if (info.format != py::format_descriptor<T>::format())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != 2)
throw std::runtime_error("Incompatible dimension: expected a 2D array!");
std::array<int64_t, 2> arr_shape;
std::move(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDView<T, 2> a(static_cast<T *>(info.ptr), arr_shape);
pedestal.push(a);
});
p.def("push", [](Pedestal<SUM_TYPE> &pedestal, const int row, const int col, const T val) {
pedestal.push(row, col, val);
});
}
template <typename SUM_TYPE> void define_pedestal_bindings(py::module &m) {
auto p = py::class_<Pedestal<SUM_TYPE>>(m, "Pedestal");
// TODO: add DType to Frame so that we can define def_buffer()
// and we can know what type of values are stored in the frame
p.def(py::init<int, int, int>())
.def(py::init<int, int>())
.def("set_freeze", &Pedestal<SUM_TYPE>::set_freeze)
.def("mean", py::overload_cast<>(&Pedestal<SUM_TYPE>::mean))
.def("mean", [](Pedestal<SUM_TYPE> &pedestal, const uint32_t row, const uint32_t col) { return pedestal.mean(row, col); })
.def("variance", py::overload_cast<>(&Pedestal<SUM_TYPE>::variance))
.def("variance",
[](Pedestal<SUM_TYPE> &pedestal, const uint32_t row, const uint32_t col) { return pedestal.variance(row, col); })
.def("standard_deviation", py::overload_cast<>(&Pedestal<SUM_TYPE>::standard_deviation))
.def("standard_deviation", [](Pedestal<SUM_TYPE> &pedestal, const int row,
const int col) { return pedestal.standard_deviation(row, col); })
.def("clear", py::overload_cast<>(&Pedestal<SUM_TYPE>::clear))
.def("clear", py::overload_cast<const uint32_t, const uint32_t>(&Pedestal<SUM_TYPE>::clear))
.def_property_readonly("rows", &Pedestal<SUM_TYPE>::rows)
.def_property_readonly("cols", &Pedestal<SUM_TYPE>::cols)
.def_property_readonly("n_samples", &Pedestal<SUM_TYPE>::n_samples)
.def_property_readonly("index", &Pedestal<SUM_TYPE>::index)
.def_property_readonly("sum", &Pedestal<SUM_TYPE>::get_sum)
.def_property_readonly("sum2", &Pedestal<SUM_TYPE>::get_sum2);
p.def("push", [](Pedestal<SUM_TYPE> &pedestal, Frame &f) {
if (f.bitdepth() == 8) {
pedestal.template push<uint8_t>(f);
} else if (f.bitdepth() == 16) {
pedestal.template push<uint16_t>(f);
} else if (f.bitdepth() == 32) {
pedestal.template push<uint32_t>(f);
} else if (f.bitdepth() == 64) {
pedestal.template push<uint64_t>(f);
} else {
throw std::runtime_error("Unsupported bitdepth");
}
});
define_pedestal_push_bindings<uint8_t>(p);
define_pedestal_push_bindings<uint16_t>(p);
define_pedestal_push_bindings<uint32_t>(p);
define_pedestal_push_bindings<uint64_t>(p);
define_pedestal_push_bindings<int8_t>(p);
define_pedestal_push_bindings<int16_t>(p);
define_pedestal_push_bindings<int32_t>(p);
define_pedestal_push_bindings<int64_t>(p);
define_pedestal_push_bindings<float>(p);
define_pedestal_push_bindings<double>(p);
}
template <typename VIEW_TYPE, typename PEDESTAL_TYPE = double>
void define_cluster_finder_template_bindings(py::class_<ClusterFinder> &cf) {
cf.def("find_clusters_without_threshold",
py::overload_cast<NDView<VIEW_TYPE, 2>, Pedestal<PEDESTAL_TYPE> &, bool>(
&ClusterFinder::find_clusters_without_threshold<VIEW_TYPE, PEDESTAL_TYPE>));
cf.def("find_clusters_with_threshold", py::overload_cast<NDView<VIEW_TYPE, 2>, Pedestal<PEDESTAL_TYPE> &>(
&ClusterFinder::find_clusters_with_threshold<VIEW_TYPE, PEDESTAL_TYPE>));
cf.def("find_clusters_without_threshold", [](ClusterFinder &self, py::array_t<VIEW_TYPE> &np_array,
Pedestal<PEDESTAL_TYPE> &pedestal, bool late_update) {
py::buffer_info info = np_array.request();
if (info.format != Dtype(typeid(VIEW_TYPE)).numpy_descr())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != 2)
throw std::runtime_error("Incompatible dimension: expected a 2D array!");
std::array<int64_t, 2> arr_shape;
std::copy(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDView<VIEW_TYPE, 2> a(static_cast<VIEW_TYPE *>(info.ptr), arr_shape);
return self.find_clusters_without_threshold(a, pedestal, late_update);
});
cf.def("find_clusters_with_threshold",
[](ClusterFinder &self, py::array_t<VIEW_TYPE> &np_array, Pedestal<PEDESTAL_TYPE> &pedestal) {
py::buffer_info info = np_array.request();
if (info.format != py::format_descriptor<VIEW_TYPE>::format())
throw std::runtime_error(
"Incompatible format: different formats! (Are you sure the arrays are of the same type?)");
if (info.ndim != 2)
throw std::runtime_error("Incompatible dimension: expected a 2D array!");
std::array<int64_t, 2> arr_shape;
std::copy(info.shape.begin(), info.shape.end(), arr_shape.begin());
NDView<VIEW_TYPE, 2> a(static_cast<VIEW_TYPE *>(info.ptr), arr_shape);
return self.find_clusters_with_threshold(a, pedestal);
});
}
void define_cluster_finder_bindings(py::module &m) {
py::class_<ClusterFinder> cf(m, "ClusterFinder");
cf.def(py::init<int, int, double, double>());
define_cluster_finder_template_bindings<uint8_t>(cf);
define_cluster_finder_template_bindings<uint16_t>(cf);
define_cluster_finder_template_bindings<uint32_t>(cf);
define_cluster_finder_template_bindings<uint64_t>(cf);
define_cluster_finder_template_bindings<int8_t>(cf);
define_cluster_finder_template_bindings<int16_t>(cf);
define_cluster_finder_template_bindings<int32_t>(cf);
define_cluster_finder_template_bindings<int64_t>(cf);
define_cluster_finder_template_bindings<float>(cf);
define_cluster_finder_template_bindings<double>(cf);
}
void define_processing_bindings(py::module &m) {
define_pedestal_bindings<double>(m);
py::class_<Cluster>(m, "Cluster",py::buffer_protocol())
.def(py::init<int, int, Dtype>())
.def("size", &Cluster::size)
.def("begin", &Cluster::begin)
.def("end", &Cluster::end)
.def_readwrite("x", &Cluster::x)
.def_readwrite("y", &Cluster::y)
.def_buffer([](Cluster &c) -> py::buffer_info {
return py::buffer_info(c.data(), c.dt.bytes(), c.dt.format_descr(), 1, {c.size()}, {c.dt.bytes()});
})
.def("__repr__", [](const Cluster &a) {
return "<Cluster: x: " + std::to_string(a.x) + ", y: " + std::to_string(a.y) + ">";
});
define_cluster_finder_bindings(m);
}

0
src/python/example/.ipynb_checkpoints/__init__-checkpoint.py Normal file
View File

2473
src/python/example/.ipynb_checkpoints/bindings_walkthrough-checkpoint.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

6
src/python/example/.ipynb_checkpoints/cluster_finder-checkpoint.ipynb Normal file
View File

@ -0,0 +1,6 @@
{
"cells": [],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}

6
src/python/example/.ipynb_checkpoints/comparing_cluster_finder-checkpoint.ipynb Normal file
View File

@ -0,0 +1,6 @@
{
"cells": [],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}

6
src/python/example/.ipynb_checkpoints/transforms-checkpoint.ipynb Normal file
View File

@ -0,0 +1,6 @@
{
"cells": [],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}

BIN
src/python/example/aare_hist.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

637
src/python/example/bindings_walkthrough.ipynb Normal file
View File

@ -0,0 +1,637 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "ce9b0c7b-09ae-4334-b131-c7097c308ad5",
"metadata": {},
"outputs": [],
"source": [
"#!pip install numpy maptlotlib mpld3"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "bad2f0df-bcfa-4f61-94cd-c4d790ffadf7",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import sys\n",
"%matplotlib inline\n",
"import mpld3\n",
"mpld3.enable_notebook()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2af45495-267d-423f-b80d-ed57afc43249",
"metadata": {},
"outputs": [],
"source": [
"PROJECT_ROOT_DIR=Path(\"../../../\").resolve()\n",
"assert PROJECT_ROOT_DIR.exists()\n",
"DATA_PATH = PROJECT_ROOT_DIR / 'data'\n",
"assert DATA_PATH.exists()\n",
"DATA_PATH_STRING = str(DATA_PATH)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b3937211-1c01-4055-ae07-638c32d7d5dd",
"metadata": {},
"outputs": [],
"source": [
"# you need to compile the project with -DAARE_PYTHON_BINDINGS=\"ON\"\n",
"sys.path.append(str((PROJECT_ROOT_DIR / 'build').resolve()))\n",
"# check if _aare can be imported\n",
"import importlib.util\n",
"_aare_module = importlib.util.find_spec(\"_aare\")\n",
"assert _aare_module is not None\n"
]
},
{
"cell_type": "markdown",
"id": "bdc318d8-d519-4c31-a196-4cb89bbfeb95",
"metadata": {},
"source": [
"# File"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c77ec29c-1042-421c-a965-96e7c597fbe3",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 5,
"id": "c8e08f34-5b88-4f13-8973-749c67a75854",
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "ImportError",
"evalue": "To be able to register buffer protocol support for the type '_aare.Frame' the associated class<>(..) invocation must include the pybind11::buffer_protocol() annotation!",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[5], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01m_aare\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m File\n",
"\u001b[0;31mImportError\u001b[0m: To be able to register buffer protocol support for the type '_aare.Frame' the associated class<>(..) invocation must include the pybind11::buffer_protocol() annotation!"
]
}
],
"source": [
"from _aare import File"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c6c5750a-56d5-4120-b687-ef61f1a73078",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "dc525593-0c63-4e51-bf41-edda59fd4244",
"metadata": {},
"outputs": [],
"source": [
"# eiger json master file\n",
"file = File(DATA_PATH_STRING + \"/eiger/eiger_500k_32bit_master_0.json\")\n",
"print(file.rows,file.cols,file.bitdepth)\n",
"print(file.detector_type)\n",
"print(file.total_frames)\n",
"print(\"geometry\",file.geometry)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a42f4d35-afee-4029-b3eb-a43fd024f5e2",
"metadata": {},
"outputs": [],
"source": [
"from _aare import DetectorType\n",
"list(DetectorType.__entries) # DetectorType is an enum"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "df6ae154-69a3-486a-96e2-c87e7532ef23",
"metadata": {},
"outputs": [],
"source": [
"# Frame class"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "614e9252-d595-46f2-891a-1473d6785586",
"metadata": {},
"outputs": [],
"source": [
"frame = file.read() # read frame from current position\n",
"frame"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06ce776e-6e96-4aa5-ac3f-b390e4fdfea0",
"metadata": {},
"outputs": [],
"source": [
"frame.rows,frame.cols,frame.bitdepth,frame.size"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4d377def-8628-4d02-9ca4-0bb2f522fb76",
"metadata": {},
"outputs": [],
"source": [
"frame.array() # returns a numpy array of type unsigned int "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "213beb5c-3153-4b48-8036-70701db14abe",
"metadata": {},
"outputs": [],
"source": [
"file.iread(1) # read frame at index 1 (only complete frames are returned)\n",
"pos = file.tell() # get the index file is currently pointed to\n",
"print(\"pos\",pos)\n",
"file.seek(0) # go to index 0\n",
"file.read(3) # read three frames. returns list of frames\n",
"# file.write(file.iread(0))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b99f72d6-3c16-4182-9034-79805770ecee",
"metadata": {},
"outputs": [],
"source": [
"# plot frame\n",
"plt.imshow(frame.array())\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "fded0b6d-02e8-4eb5-aacc-51f70663a664",
"metadata": {},
"source": [
"# ClusterFile"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "86674099-fd17-4ebb-af11-e86b7665a254",
"metadata": {},
"outputs": [],
"source": [
"from _aare import ClusterFileV2"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "416394ee-1251-4373-acc4-49ca1fa2637a",
"metadata": {},
"outputs": [],
"source": [
"file = ClusterFileV2(DATA_PATH_STRING + \"/clusters/beam_En700eV_-40deg_300V_10us_d0_f0_100.clust\",\"r\")\n",
"file"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "baef30ee-e2de-4052-9b5b-438af78f51e3",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"file.read() # returns a list of the clusters of the current frame number"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b979f805-0058-428b-b580-28e228fe0b4e",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"file.read(2) # returns a list of clusters for two frames"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ee58e574-2a44-455e-93a1-2344aefe74f9",
"metadata": {},
"outputs": [],
"source": [
"file.frame_number() # get the frame number where clusterFile currently points to"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7889a1f3-4292-4369-aebb-a19262bd6daf",
"metadata": {},
"outputs": [],
"source": [
"clusters = file.read()\n",
"len(clusters)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3077f5c7-7c4d-43db-b460-bfc77c6733d3",
"metadata": {},
"outputs": [],
"source": [
"cluster = clusters[0]\n",
"print(cluster.frame_number, cluster.cluster.x,cluster.cluster.y) # get cluster information\n",
"np.array(cluster.cluster.data,np.int32) # view cluster data "
]
},
{
"cell_type": "markdown",
"id": "b24afe47-0e93-49ed-871d-c32a06d3f975",
"metadata": {},
"source": [
"# Pedestal"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1653184c-4d27-4de1-b400-2ca8eb80bf9f",
"metadata": {},
"outputs": [],
"source": [
"from _aare import Pedestal\n",
"file = File(\"/mnt/sls_det_storage/moench_data/testNewFW20230714/cu_half_speed_master_4.json\")\n",
"file.rows,file.cols,file.detector_type"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e78505b4-cd25-440a-9996-ea7e2104f5ee",
"metadata": {},
"outputs": [],
"source": [
"pedestal = Pedestal(file.rows,file.cols)\n",
"pedestal.rows,pedestal.cols,pedestal.n_samples"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8e172b20-ed12-4c8c-8090-881b7cc9b1a0",
"metadata": {},
"outputs": [],
"source": [
"ndarray = pedestal.mean() # pedestal mean returns an internal NDArray object of type float8 and dimensions 2\n",
"ndarray"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2c9313bb-efca-4cd1-8c1e-fdd1dd3e5f76",
"metadata": {},
"outputs": [],
"source": [
"np.array(ndarray) # to convert NDArray to numpy array with copying the data\n",
"np.array(ndarray,copy=False) # to convert NDArray to numpy array without copying the data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "190d1fa0-9e16-4706-9523-287bd6456767",
"metadata": {},
"outputs": [],
"source": [
"ndarray.shape,ndarray.size,ndarray.bitdepth,ndarray.strides,ndarray.byte_strides"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f6352bc7-147b-4421-8049-273bb3070d88",
"metadata": {},
"outputs": [],
"source": [
"ndarray[0,300]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c951ba2-8a95-4046-81f9-c38b64bc0f08",
"metadata": {},
"outputs": [],
"source": [
"for i in range(2000):\n",
" frame = file.iread(i)\n",
" pedestal.push(frame) # push 2000 frames into the pedestal\n",
" "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "15af0d52-cf8d-4b5b-8f59-cec78f73fbad",
"metadata": {},
"outputs": [],
"source": [
"mean = np.array(pedestal.mean())\n",
"plt.imshow(mean)\n",
"plt.show()\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "64c686db-ad1b-42a8-a926-7d3653b1b6a5",
"metadata": {},
"outputs": [],
"source": [
"frame=file.iread(3000) # read frame at index 3000\n",
"plt.imshow(frame.array())\n",
"plt.title(\"Frame\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3e5261a5-b311-4d3a-ac7c-81535d0b91a3",
"metadata": {},
"outputs": [],
"source": [
"frame=file.iread(3000) # read frame at index 3000\n",
"denoised_frame = frame.array()-pedestal.mean()\n",
"plt.imshow(denoised_frame)\n",
"plt.title(\"denoised frame\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ba40185a-2996-4b33-b797-fd9a27ba0269",
"metadata": {},
"outputs": [],
"source": [
"#other utilities in pedestal\n",
"pedestal.variance() # get NDArray of the variance for each pixel \n",
"pedestal.standard_deviation() # get NDArray of the standard deviation for each pixel \n",
"pedestal.sum # get NDArray of the sum squared for each pixel \n",
"pedestal.sum2 # get NDArray of the sum squared for each pixel \n",
"pedestal.clear() # resets the pedestal\n",
"# mean, variance, standard_deviation and clear can accept indexes to apply only specific pixels\n",
"row,col=0,0\n",
"pedestal.mean(row,col)\n",
"pedestal.variance(row,col)\n",
"pedestal.standard_deviation(row,col)\n",
"pedestal.clear(row,col)"
]
},
{
"cell_type": "markdown",
"id": "74eeb261-374b-41f8-a707-ca95441321ef",
"metadata": {},
"source": [
"# ClusterFinder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33fed84c-dc7b-4a3e-9078-5530c8234234",
"metadata": {},
"outputs": [],
"source": [
"from _aare import ClusterFinder"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5b87588f-1749-49db-b2d1-209a99d95518",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"cf = ClusterFinder(3,3,1.0,1) # cluster_sizeX, cluster_sizeY, nSigma, threshold (can be ignored depending on the function called later)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c74573f2-f693-499d-82a4-493f0df91b8f",
"metadata": {},
"outputs": [],
"source": [
"clusters = cf.find_clusters_without_threshold(denoised_frame,pedestal)\n",
"len(clusters)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b5338147-202a-4872-bf7f-54d279485d84",
"metadata": {},
"outputs": [],
"source": [
"# x_arr,y_arr=[],[]\n",
"# for cluster in clusters:\n",
"# denoised_frame[cluster.x,cluster.y]=-1000\n",
"# plt.imshow(denoised_frame)\n",
"# plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d93fbc4c-afbf-4ab8-9e35-94a6c57e8b6b",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "e2d19bc5-e024-4b36-b025-d8f37f6b6ba1",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "9b0f7ab7-4d00-40d1-8d6d-9e261658a718",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "1683c4aa-eea8-43ec-b623-8a7c37262ec0",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b7aa2b9-3b16-48b2-a3fb-f6968ead8c4a",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "a56219c2-2deb-4469-8762-c86c00e9a2c7",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "ec117dcd-cab5-48fd-ae67-095ff818dc1a",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "1b8ed8fb-f30f-47fe-a52a-5ffea364543d",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f6e595d-9c86-4a15-ba58-c007a6593ac8",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "d62db3b3-997e-46ba-ae25-05a4bdfa16b2",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "a5d7a57e-49b5-4027-830c-d3b9487da440",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "1acab559-9168-418d-a476-78ff90a1360f",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "20df5451-508e-4925-afd5-a0309c30ed3d",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "a3bb1bc6-909a-467d-ae7f-4009882fd0af",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "bb3ced8f-c6d0-495d-a446-5300b7a5293e",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"id": "2de814a2-0c5e-4ffe-a3ef-f5ce6436f63d",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

BIN
src/python/example/cluster sum for each (x,y).png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

3891
src/python/example/cluster_finder.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

BIN
src/python/example/cluster_sum_per_pixel.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 180 KiB

1890
src/python/example/comparing_cluster_finder.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

32
src/python/example/read_frame.py
View File

@ -1,32 +0,0 @@
import os
from pathlib import Path
from aare import File, Frame
import numpy as np
if __name__ == "__main__":
#get env variable
root_dir = Path(os.environ.get("PROJECT_ROOT_DIR"))
# read JSON master file
data_path = str(root_dir / "data"/"jungfrau_single_master_0.json")
file = File(data_path)
frame = file.get_frame(0)
print(frame.rows, frame.cols)
print(frame.get(0,0))
# read Numpy file
data_path = str(root_dir / "data"/"test_numpy_file.npy")
file = File(data_path)
frame = file.get_frame(0)
print(frame.rows, frame.cols)
print(frame.get(0,0))
arr = np.array(frame.get_array())
print(arr)
print(arr.shape)
print(np.array_equal(arr, np.load(data_path)[0]))

BIN
src/python/example/simple_density.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 176 KiB

374
src/python/example/transforms.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More