try to write root files

This commit is contained in:
ext-muller_m committed 2026-11-12 16:51:08 +01:00
1 parent 78bede6781
commit f131777610
24 files changed
+880 -40

No files matched your search

+1
View File
@@ -221,6 +221,7 @@ else()
endif()
option(SLS_USE_HDF5 "HDF5 File format" OFF)
option(SLS_USE_ROOT "ROOT File format" OFF)
option(SLS_BUILD_SHARED_LIBRARIES "Build shared libaries" OFF)
option(SLS_USE_TEXTCLIENT "Text Client" ON)
option(SLS_USE_DETECTOR "Detector libs" ON)
+1 -1
View File
@@ -3,7 +3,7 @@
Binary File Format
====================
This is the default file format that can be configured using command `fformat <commandline.html#term-fformat-binary-hdf5>`_.
This is the default file format that can be configured using command `fformat <commandline.html#term-fformat-binary-hdf5-root>`_.
.. code-block:: bash
+4 -3
View File
@@ -55,12 +55,13 @@ Some file name examples:
Formats
--------
There are 2 file formats supported by the receiver:
There are 3 file formats supported by the receiver:
* Binary - extension .json (master file) or .raw (data files)
* HDF5 - extension .h5
* ROOT - extension .json (master file) or .root (data files)
The default is binary. HDF5 can be enabled by compiling the package with HDF5 option enabled. The file format is set using the command `fformat <commandline.html#term-fformat-binary-hdf5>`_.
The default is binary. HDF5 can be enabled by compiling the package with HDF5 option enabled and ROOT by compiling the package with ROOT option enabled. The file format is set using the command `fformat <commandline.html#term-fformat-binary-hdf5-root>`_.
Content
@@ -72,4 +73,4 @@ Content
| `Data File`: Contains the metadata for each image (:ref:`slsReceiverHeader<sls receiver header format>`) and the `data of the image` transmitted over that port. The image data is described in :ref:`Detector Image Size and Format <data format>` section.
| More content and examples are found in the :ref:`HDF5 file format <hdf5 file format>` and :ref:`Binary file format <binary file format>` sections.
| More content and examples are found in the :ref:`HDF5 file format <hdf5 file format>`, :ref:`Binary file format <binary file format>` and :ref:`ROOT file format <root file format>` sections.
+2 -2
View File
@@ -20,7 +20,7 @@ Setup
#. Load config file
#. Set file format using command `fformat <commandline.html#term-fformat-binary-hdf5>`_.
#. Set file format using command `fformat <commandline.html#term-fformat-binary-hdf5-root>`_.
.. code-block:: bash
@@ -30,7 +30,7 @@ Setup
Filters (Compression)
----------------------
HDF5 filters (eg. compression) for the image dataset in the data files are set using command `fformatoptions <commandline.html#term-fformatoptions-filter-id-param-1-param-2-...-or-none>`_ after setting the file format to hdf5. The default is no filters.
HDF5 filters (eg. compression) for the image dataset in the data files are set using command `fformatoptions <commandline.html#term-fformatoptions-filter-id-param-1-param-2-...-or-TTree-TH2I-or-none>`_ after setting the file format to hdf5. The default is no filters.
* Each filter is the HDF5 filter id followed by its parameters (cd_values), all unsigned integers.
+1
View File
@@ -113,6 +113,7 @@ slsDetectorPackage
masterfileattributes
binaryfileformat
hdf5fileformat
rootfileformat
.. toctree::
:caption: Receiver ZMQ Stream
+1
View File
@@ -182,6 +182,7 @@ Example cmake options Comment
-DPython_FIND_VIRTUALENV=ONLY Python from the conda env
-DSLS_USE_GUI=ON GUI
-DSLS_USE_HDF5=ON HDF5
-DSLS_USE_ROOT=ON ROOT
-DSLS_USE_SIMULATOR=ON Simulator
=============================== ===============================
+95
View File
@@ -0,0 +1,95 @@
.. _root file format:
ROOT File Format
================================
Compilation
-------------
#. Compile the package with ROOT option enabled using cmake option **-DSLS_USE_ROOT=ON**. ROOT must be found by cmake (eg. by sourcing thisroot.sh of the ROOT installation).
Setup
-------
#. Start Receiver process
#. Load config file
#. Set file format using command `fformat <commandline.html#term-fformat-binary-hdf5-root>`_.
.. code-block:: bash
sls_detector_put fformat root
#. Choose how the images are saved using command `fformatoptions <commandline.html#term-fformatoptions-filter-id-param-1-param-2-...-or-TTree-TH2I-or-none>`_. The default is TTree. Changing the file format resets it to the default.
.. code-block:: bash
# every image is an entry in a tree (default)
sls_detector_put fformatoptions TTree
# every image is a histogram
sls_detector_put fformatoptions TH2I
Master File
-------------
* File Name: [fpath]/[fname]_master_[findex].json :ref:`Details here<file name format>`
* It is the same master file as for the :ref:`binary file format <binary file format>` and contains :ref:`attributes<master file attributes>` relevant to the acquisition.
Data File
-----------
* File Name: [fpath]/[fname]_dx_fy_[findex].root :ref:`Details here<file name format>`
* Every data file has a tree called **frames** with one entry per image. Every member of the :ref:`sls_detector_header<sls receiver header format>` of the image is a branch of this tree.
================= =========================== =====================
Branch Type Content
================= =========================== =====================
frameNumber 64 bit unsigned integer frame number
expLength 32 bit unsigned integer exposure length or sub exposure time
packetNumber 32 bit unsigned integer packets caught
detSpec1 64 bit unsigned integer detector specific 1
timestamp 64 bit unsigned integer timestamp
modId 16 bit unsigned integer module id
row 16 bit unsigned integer row
column 16 bit unsigned integer column
detSpec2 16 bit unsigned integer detector specific 2
detSpec3 32 bit unsigned integer detector specific 3
detSpec4 16 bit unsigned integer detector specific 4
detType 8 bit unsigned integer detector type
version 8 bit unsigned integer detector header version
packetsMask 64 x 8 bit unsigned integer packets caught bit mask
================= =========================== =====================
* The image itself is saved depending on the file format option:
* **TTree** (default): The image is the branch **data** of the tree **frames**. It is a two dimensional array [rows][columns] of unsigned integers of the size of the dynamic range. As in the HDF5 file format, 12 bit is saved as 16 bit and 4 bit is saved as it is sent by the detector (2 pixels in every 8 bit unsigned integer, so half the number of columns).
* **TH2I**: The image is a histogram called **frame_[frame number]** with one bin per pixel (x axis: column, y axis: row). The bins are signed 32 bit integers, so larger values are saved as the maximum of a signed 32 bit integer (2147483647). The file has one histogram per image.
Reading
---------
.. code-block:: cpp
TFile file("run_d0_f0_0.root");
TTree *frames = (TTree *)file.Get("frames");
ULong64_t frameNumber;
frames->SetBranchAddress("frameNumber", &frameNumber);
// TTree: [rows][columns] in the type of the dynamic range (here 16 bit)
UShort_t data[512][1024];
frames->SetBranchAddress("data", data);
frames->GetEntry(0);
// TH2I: histogram of the image with the frame number of the entry
frames->GetEntry(0);
TH2I *image = (TH2I *)file.Get(Form("frame_%llu", frameNumber));
image->Draw("colz");
+26 -5
View File
@@ -1065,9 +1065,9 @@ class Detector(CppDetectorApi):
Note
-----
Options: BINARY, HDF5
Options: BINARY, HDF5, ROOT
Default: BINARY
For HDF5, package must be compiled with HDF5 flags. Default is binary.
For HDF5, package must be compiled with HDF5 flags. For ROOT, package must be compiled with ROOT flags. Default is binary.
Example
--------
@@ -1088,8 +1088,9 @@ class Detector(CppDetectorApi):
Note
-----
Default is no options. They are reset when the file format is changed. \n
[HDF5] List of HDF5 filters (eg. compression) applied in the given order to the image dataset. Each filter is a list with the HDF5 filter id followed by its parameters (cd_values). Filters that are not part of the HDF5 library (eg. 32008 for bitshuffle) must be available to the receiver as plugins found in HDF5_PLUGIN_PATH. \n
No options ([]) sets the default of the file format. They are reset to the default when the file format is changed. \n
[HDF5] List of HDF5 filters (eg. compression) applied in the given order to the image dataset. Each filter is a list with the HDF5 filter id followed by its parameters (cd_values). Filters that are not part of the HDF5 library (eg. 32008 for bitshuffle) must be available to the receiver as plugins found in HDF5_PLUGIN_PATH. Default is no filters. \n
[ROOT] How the images are saved. Enum: rootImageType. ROOT_TTREE (default): every image is an entry in the tree 'frames'. ROOT_TH2I: every image is a histogram called 'frame_[frame number]'. \n
[Binary] No options available.
Example
@@ -1100,11 +1101,31 @@ class Detector(CppDetectorApi):
>>> d.fformatoptions
[[2], [1, 6]]
>>> d.fformatoptions = [] # no filters
>>> d.fformat = fileFormat.ROOT
>>> d.fformatoptions
rootImageType.ROOT_TTREE
>>> d.fformatoptions = rootImageType.ROOT_TH2I
"""
return self.getFileFormatOptions()
options = self.getFileFormatOptions()
# [ROOT] the option of a module is a single value of rootImageType
for i, file_format in enumerate(self.getFileFormat()):
if (file_format == defs.fileFormat.ROOT and len(options[i]) == 1
and len(options[i][0]) == 1):
options[i] = defs.rootImageType(options[i][0][0])
return options
@fformatoptions.setter
def fformatoptions(self, value):
# [ROOT] rootImageType is a single option with a single value
def to_options(v):
if isinstance(v, defs.rootImageType):
return [[int(v)]]
return v
if isinstance(value, dict):
value = {k: to_options(v) for k, v in value.items()}
else:
value = to_options(value)
ut.set_using_dict(self.setFileFormatOptions, value)
@property
@@ -285,6 +285,11 @@
<string>HDF5</string>
</property>
</item>
<item>
<property name="text">
<string>ROOT</string>
</property>
</item>
</widget>
</item>
<item row="0" column="0">
+1
View File
@@ -231,6 +231,7 @@ void qTabDataOutput::GetFileFormat() {
switch (retval) {
case slsDetectorDefs::BINARY:
case slsDetectorDefs::HDF5:
case slsDetectorDefs::ROOT:
comboFileFormat->setCurrentIndex(static_cast<int>(retval));
break;
default:
+1 -1
View File
@@ -1009,7 +1009,7 @@ rx_arping:
input_types: [ bool ]
fformat:
help: "[binary|hdf5]\n\tFile format of data file. For HDF5, package must be compiled with HDF5 flags. Default is binary."
help: "[binary|hdf5|root]\n\tFile format of data file. For HDF5, package must be compiled with HDF5 flags. For ROOT, package must be compiled with ROOT flags. Default is binary."
inherit_actions: INTEGER_COMMAND_VEC_ID
actions:
GET:
+11 -5
View File
@@ -1051,22 +1051,28 @@ class Detector {
* ************************************************/
Result<defs::fileFormat> getFileFormat(Positions pos = {}) const;
/** default binary, Options: BINARY, HDF5 (library must be compiled with
* this option). Changing the file format resets the file format options.
/** default binary, Options: BINARY, HDF5, ROOT (library must be compiled
* with the option for HDF5 or ROOT). Changing the file format resets the
* file format options.
*/
void setFileFormat(defs::fileFormat f, Positions pos = {});
Result<std::vector<std::vector<uint32_t>>>
getFileFormatOptions(Positions pos = {}) const;
/** Options specific to the current file format. Default is no options.
* They are reset when the file format is changed. \n
/** Options specific to the current file format. No options sets the
* default of the file format. They are reset to the default when the
* file format is changed. \n
* [HDF5] List of HDF5 filters (eg. compression) applied in the given
* order to the image dataset. Each filter is a list with the HDF5 filter
* id followed by its parameters (cd_values). eg. {{2}, {1, 6}} for
* shuffle followed by gzip with compression level 6. Filters that are
* not part of the HDF5 library must be available as plugins to the
* receiver (HDF5_PLUGIN_PATH). \n
* receiver (HDF5_PLUGIN_PATH). Default is no filters. \n
* [ROOT] A single option with a single value of defs::rootImageType,
* which is how the images are saved. eg. {{defs::ROOT_TH2I}}.
* ROOT_TTREE (default): every image is an entry in a tree. ROOT_TH2I:
* every image is a histogram. \n
* [Binary] No options available. */
void setFileFormatOptions(const std::vector<std::vector<uint32_t>> &options,
Positions pos = {});
+81 -13
View File
@@ -987,18 +987,49 @@ parseFileFormatOptions(const std::vector<std::string> &args) {
return options;
}
/** true if it is the name of a file format option of root (eg. TTree) */
bool isRootImageType(const std::string &s) {
try {
StringTo<defs::rootImageType>(s);
} catch (const std::exception &) {
return false;
}
return true;
}
/**
* File format options of root as text. The option of a module is a single
* value of rootImageType, which is printed as its name (eg. TTree).
*/
Result<std::string> rootFileFormatOptionsToString(
const Result<std::vector<std::vector<uint32_t>>> &options) {
Result<std::string> names;
for (const auto &it : options) {
if (it.size() == 1 && it[0].size() == 1) {
names.push_back(
ToString(static_cast<defs::rootImageType>(it[0][0])));
} else {
names.push_back(ToString(it));
}
}
return names;
}
} // namespace
std::string Caller::fformatoptions(int action) {
std::ostringstream os;
std::string helpMessage =
std::string("[filter id] [param 1] [param 2]... or [none]\n") +
std::string("[filter id] [param 1] [param 2]... or [TTree|TH2I] or "
"[none]\n") +
"\tOptions specific to the file format (fformat) of the data files "
"written by the receiver. Default is none.\n"
"\tThey are reset to none when the file format is changed.\n\n"
"written by the receiver.\n"
"\tThey are reset to the default of the file format when the file "
"format is changed.\n"
"\tUse 'none' to set the default of the file format.\n\n"
"\t[hdf5] List of HDF5 filters (eg. compression) applied in the "
"given order to the image dataset.\n"
"given order to the image dataset. Default is none (no filters).\n"
"\tA filter is the HDF5 filter id followed by its parameters "
"(cd_values), all unsigned integers.\n"
"\tFor example, to set a single filter (gzip with compression level "
@@ -1019,6 +1050,15 @@ std::string Caller::fformatoptions(int action) {
"HDF5_PLUGIN_PATH.\n"
"\t- Use 'none' to remove all filters.\n\n"
"\t[root] [TTree|TH2I] How the images are saved. Default is TTree.\n"
"\tTTree: Every image is an entry in the tree 'frames'. The image is "
"in the branch 'data' and the header of the image in the other "
"branches.\n"
"\tTH2I: Every image is a histogram called 'frame_[frame number]'. "
"The headers of the images are in the tree 'frames'. The bins are "
"signed 32 bit integers, so larger values are saved as the maximum "
"of a signed 32 bit integer.\n\n"
"\t[binary] No options available.\n";
if (action == defs::HELP_ACTION) {
@@ -1028,25 +1068,53 @@ std::string Caller::fformatoptions(int action) {
WrongNumberOfParameters(0);
}
auto t = det->getFileFormatOptions(std::vector<int>{det_id});
os << OutString(t) << '\n';
// the options depend on the file format
auto format = det->getFileFormat(std::vector<int>{det_id})
.squash(defs::NUM_FILE_FORMATS);
if (format == defs::ROOT) {
os << OutString(rootFileFormatOptionsToString(t)) << '\n';
} else {
os << OutString(t) << '\n';
}
} else if (action == defs::PUT_ACTION) {
if (args.empty()) {
WrongNumberOfParameters(1);
}
// the options depend on the file format
auto format = det->getFileFormat(std::vector<int>{det_id})
.squash(defs::NUM_FILE_FORMATS);
std::vector<std::vector<uint32_t>> t;
// 'none' option already covered as t is empty by default
if (args.size() != 1 || args[0] != "none") {
try {
t = parseFileFormatOptions(args);
} catch (const std::exception &e) {
throw RuntimeError(
"Could not parse file format options: " +
std::string(e.what()) + ". Use sls_detector_help " + cmd +
" to get the right syntax expected.");
if (format == defs::ROOT) {
if (args.size() != 1) {
WrongNumberOfParameters(1);
}
auto type = StringTo<defs::rootImageType>(args[0]);
t.push_back({static_cast<uint32_t>(type)});
} else {
if (args.size() == 1 && isRootImageType(args[0])) {
throw RuntimeError(args[0] + " is only available for " +
ToString(defs::ROOT) + " file format");
}
try {
t = parseFileFormatOptions(args);
} catch (const std::exception &e) {
throw RuntimeError(
"Could not parse file format options: " +
std::string(e.what()) + ". Use sls_detector_help " +
cmd + " to get the right syntax expected.");
}
}
}
det->setFileFormatOptions(t, std::vector<int>{det_id});
os << ToString(t) << '\n';
if (format == defs::ROOT) {
// 'none' sets the default
auto r = det->getFileFormatOptions(std::vector<int>{det_id});
os << OutString(rootFileFormatOptionsToString(r)) << '\n';
} else {
os << ToString(t) << '\n';
}
} else {
throw RuntimeError("Unknown action");
}
+8
View File
@@ -41,6 +41,14 @@ if (SLS_USE_HDF5)
)
endif (SLS_USE_HDF5)
# ROOT file
if (SLS_USE_ROOT)
target_compile_definitions(tests
PUBLIC
-DROOTC
)
endif (SLS_USE_ROOT)
target_compile_options(tests
PRIVATE
# Suppress warnings for deprecated functions in rapidjson
@@ -1035,9 +1035,66 @@ TEST_CASE("fformatoptions", "[.detectorintegration]") {
REQUIRE(oss.str() == "fformatoptions []\n");
}
REQUIRE_THROWS(caller.call("fformatoptions", {"1", "6"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"TTree"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"1"}, -1, GET));
#ifdef ROOTC
// root: how the images are saved (default is TTree)
{
std::ostringstream oss;
caller.call("fformat", {"root"}, -1, PUT, oss);
REQUIRE(oss.str() == "fformat root\n");
}
{
std::ostringstream oss;
caller.call("fformatoptions", {}, -1, GET, oss);
REQUIRE(oss.str() == "fformatoptions TTree\n");
}
{
std::ostringstream oss;
caller.call("fformatoptions", {"TH2I"}, -1, PUT, oss);
REQUIRE(oss.str() == "fformatoptions TH2I\n");
}
{
std::ostringstream oss;
caller.call("fformatoptions", {}, -1, GET, oss);
REQUIRE(oss.str() == "fformatoptions TH2I\n");
}
{
std::ostringstream oss;
caller.call("fformatoptions", {"ttree"}, -1, PUT, oss);
REQUIRE(oss.str() == "fformatoptions TTree\n");
}
REQUIRE_NOTHROW(caller.call("fformatoptions", {"TH2I"}, -1, PUT));
// none sets the default
{
std::ostringstream oss;
caller.call("fformatoptions", {"none"}, -1, PUT, oss);
REQUIRE(oss.str() == "fformatoptions TTree\n");
}
// only the names of the image types
REQUIRE_THROWS(caller.call("fformatoptions", {"TH2F"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"1"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"1", "6"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"TTree", "TH2I"}, -1, PUT));
// only image types as options
REQUIRE_THROWS(det.setFileFormatOptions({{defs::NUM_ROOT_IMAGE_TYPES}}));
REQUIRE_THROWS(det.setFileFormatOptions({{defs::ROOT_TTREE, 1}}));
REQUIRE_THROWS(
det.setFileFormatOptions({{defs::ROOT_TTREE}, {defs::ROOT_TH2I}}));
// changing file format resets the options to the default
REQUIRE_NOTHROW(caller.call("fformatoptions", {"TH2I"}, -1, PUT));
det.setFileFormat(defs::BINARY);
det.setFileFormat(defs::ROOT);
{
std::ostringstream oss;
caller.call("fformatoptions", {}, -1, GET, oss);
REQUIRE(oss.str() == "fformatoptions TTree\n");
}
det.setFileFormat(defs::BINARY);
#endif
#ifdef HDF5C
// hdf5 filters (1: gzip, 2: shuffle, 3: fletcher32 are part of hdf5 lib)
det.setFileFormat(defs::HDF5);
@@ -1084,6 +1141,7 @@ TEST_CASE("fformatoptions", "[.detectorintegration]") {
REQUIRE_THROWS(caller.call("fformatoptions", {"1", "-6"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"1", "6.5"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"gzip"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"TH2I"}, -1, PUT));
// invalid filter id, repeated filter, filter not available
REQUIRE_THROWS(caller.call("fformatoptions", {"0"}, -1, PUT));
REQUIRE_THROWS(caller.call("fformatoptions", {"65536"}, -1, PUT));
+19
View File
@@ -31,6 +31,17 @@ if (SLS_USE_HDF5)
)
endif (SLS_USE_HDF5)
# ROOT file writing
if (SLS_USE_ROOT)
find_package(ROOT REQUIRED COMPONENTS RIO Tree Hist)
add_definitions(
-DROOTC
)
list (APPEND SOURCES
src/RootDataFile.cpp
)
endif (SLS_USE_ROOT)
# Create an object library to avoid building the library twice
# This is only used during the build phase
add_library(slsReceiverObject OBJECT
@@ -67,6 +78,14 @@ if (SLS_USE_HDF5)
endif ()
endif (SLS_USE_HDF5)
# ROOT
if (SLS_USE_ROOT)
target_link_libraries(slsReceiverObject PUBLIC
${ROOT_LIBRARIES})
target_include_directories(slsReceiverObject SYSTEM PUBLIC
${ROOT_INCLUDE_DIRS})
endif (SLS_USE_ROOT)
set(RECEIVER_LIBRARY_TARGETS slsReceiverObject)
#Shared library
+25 -4
View File
@@ -16,6 +16,9 @@
#ifdef HDF5C
#include "HDF5DataFile.h"
#endif
#ifdef ROOTC
#include "RootDataFile.h"
#endif
#include "DataStreamer.h"
#include "sls/container_utils.h"
#include "sls/sls_detector_exceptions.h"
@@ -138,12 +141,18 @@ void DataProcessor::SetupFileWriter(const bool filewriteEnable,
case HDF5:
dataFile = new HDF5DataFile(index, hdf5LibMutex);
break;
#endif
#ifdef ROOTC
case ROOT:
dataFile = new RootDataFile(index);
break;
#endif
case BINARY:
dataFile = new BinaryDataFile(index);
break;
default:
throw RuntimeError("Unknown file format (compile with hdf5 flags");
throw RuntimeError(
"Unknown file format (compile with hdf5 or root flags)");
}
}
}
@@ -163,7 +172,7 @@ void DataProcessor::CreateFirstFiles(
return;
}
#ifdef HDF5C
#if defined(HDF5C) || defined(ROOTC)
int nx = generalData->nPixelsX;
int ny = generalData->nPixelsY;
if (isPartiallyInRoi) {
@@ -182,6 +191,14 @@ void DataProcessor::CreateFirstFiles(
udpPortNumber, generalData->framesPerFile, nTotalFrames, nx, ny,
generalData->dynamicRange, fileFormatOptions);
break;
#endif
#ifdef ROOTC
case ROOT:
dataFile->CreateFirstRootDataFile(
filePath, fileNamePrefix, fileIndex, overWriteEnable, silentMode,
udpPortNumber, generalData->framesPerFile, nx, ny,
generalData->dynamicRange, fileFormatOptions);
break;
#endif
case BINARY:
dataFile->CreateFirstBinaryDataFile(
@@ -189,7 +206,8 @@ void DataProcessor::CreateFirstFiles(
udpPortNumber, generalData->framesPerFile);
break;
default:
throw RuntimeError("Unknown file format (compile with hdf5 flags");
throw RuntimeError(
"Unknown file format (compile with hdf5 or root flags)");
}
}
@@ -262,12 +280,15 @@ std::string DataProcessor::CreateMasterFile(
filePath, fileNamePrefix, fileIndex, overWriteEnable, silentMode,
attr, hdf5LibMutex);
#endif
// root data files have the same master file as binary data files (json)
case ROOT:
case BINARY:
return masterFileUtility::CreateMasterBinaryFile(
filePath, fileNamePrefix, fileIndex, overWriteEnable, silentMode,
attr);
default:
throw RuntimeError("Unknown file format (compile with hdf5 flags");
throw RuntimeError(
"Unknown file format (compile with hdf5 or root flags)");
}
}
+14
View File
@@ -68,6 +68,20 @@ class File : private virtual slsDetectorDefs {
<< "This is a generic function CreateFirstHDF5DataFile that "
"should be overloaded by a derived class";
};
#endif
#ifdef ROOTC
virtual void CreateFirstRootDataFile(
const std::filesystem::path &filePath,
const std::string &fileNamePrefix, const uint64_t fileIndex,
const bool overWriteEnable, const bool silentMode,
const uint16_t udpPortNumber, const uint32_t maxFramesPerFile,
const uint32_t nPixelsX, const uint32_t nPixelsY,
const uint32_t dynamicRange,
const std::vector<std::vector<uint32_t>> &options) {
LOG(logERROR)
<< "This is a generic function CreateFirstRootDataFile that "
"should be overloaded by a derived class";
};
#endif
virtual void CreateFirstBinaryDataFile(
const std::filesystem::path &filePath,
+35 -5
View File
@@ -10,6 +10,9 @@
#endif
#include "Listener.h"
#include "MasterAttributes.h"
#ifdef ROOTC
#include "RootDataFile.h"
#endif
#include "sls/ToString.h"
#include "sls/ZmqSocket.h" //just for the zmq port define
#include "sls/file_utils.h"
@@ -470,6 +473,21 @@ std::vector<slsDetectorDefs::ROI> Implementation::getMultiROIMetadata() const {
* File Parameters *
* *
* ************************************************/
namespace {
/** file format options of a file format when none are set */
std::vector<std::vector<uint32_t>>
GetDefaultFileFormatOptions(const slsDetectorDefs::fileFormat f) {
std::vector<std::vector<uint32_t>> options;
if (f == slsDetectorDefs::ROOT) {
options.push_back(
{static_cast<uint32_t>(slsDetectorDefs::ROOT_TTREE)});
}
return options;
}
} // namespace
slsDetectorDefs::fileFormat Implementation::getFileFormat() const {
return fileFormatType;
}
@@ -481,6 +499,11 @@ void Implementation::setFileFormat(const fileFormat f) {
case HDF5:
fileFormatType = HDF5;
break;
#endif
#ifdef ROOTC
case ROOT:
fileFormatType = ROOT;
break;
#endif
case BINARY:
fileFormatType = BINARY;
@@ -491,10 +514,9 @@ void Implementation::setFileFormat(const fileFormat f) {
for (const auto &it : dataProcessor)
it->SetupFileWriter(fileWriteEnable, fileFormatType, &hdf5LibMutex);
// options belong to the previous file format
if (!fileFormatOptions.empty()) {
fileFormatOptions.clear();
LOG(logINFO) << "File Format Options reset";
}
fileFormatOptions = GetDefaultFileFormatOptions(fileFormatType);
LOG(logINFO) << "File Format Options reset to "
<< ToString(fileFormatOptions);
}
LOG(logINFO) << "File Format: " << ToString(fileFormatType);
@@ -513,6 +535,11 @@ void Implementation::setFileFormatOptions(
std::lock_guard<std::mutex> lock(hdf5LibMutex);
HDF5DataFile::ValidateFilters(options);
} break;
#endif
#ifdef ROOTC
case ROOT:
RootDataFile::ValidateOptions(options);
break;
#endif
default:
if (!options.empty()) {
@@ -521,7 +548,10 @@ void Implementation::setFileFormatOptions(
}
break;
}
fileFormatOptions = options;
// no options sets the default of the file format
fileFormatOptions = options.empty()
? GetDefaultFileFormatOptions(fileFormatType)
: options;
LOG(logINFO) << "File Format Options: " << ToString(fileFormatOptions);
}
+359
View File
@@ -0,0 +1,359 @@
// SPDX-License-Identifier: LGPL-3.0-or-other
// Copyright (C) 2021 Contributors to the SLS Detector Package
// clang-format off
// root headers first, as they must not see the macros of the sls headers
// (eg. RESET and BOLD in sls/ansi.h)
#include "TFile.h"
#include "TH2.h"
#include "TROOT.h"
#include "TTree.h"
// clang-format on
#include "RootDataFile.h"
#include "sls/ToString.h"
#include <algorithm>
#include <cstring>
#include <limits>
#include <mutex>
namespace sls {
namespace {
constexpr char TREE_NAME[] = "frames";
constexpr char IMAGE_BRANCH_NAME[] = "data";
constexpr char HISTOGRAM_NAME_PREFIX[] = "frame_";
/** default size in bytes of the buffer of a branch in root */
constexpr size_t DEFAULT_BASKET_SIZE = 32000;
/**
* By default, root closes the file of a tree after 100 GB and continues in
* a new file with another name. The receiver starts new files itself (max
* frames per file), so this must never happen.
*/
constexpr Long64_t MAX_TREE_SIZE = 1LL << 50;
/** settings of root for the complete process, set only once */
void InitializeRoot() {
static std::once_flag initialized;
std::call_once(initialized, []() {
// every port writes its file in its own thread
::ROOT::EnableThreadSafety();
TTree::SetMaxTreeSize(MAX_TREE_SIZE);
// by default, a histogram adds itself to the current directory of
// the thread it is created in. Here it is written explicitly to the
// file instead.
TH1::AddDirectory(kFALSE);
});
}
/** number of complete pixels in an image as it is sent by the detector */
size_t GetNumPixels(const size_t numBytes, const uint32_t dynamicRange) {
switch (dynamicRange) {
case 4:
return numBytes * 2;
case 12:
// 2 pixels in 3 bytes
return (numBytes / 3) * 2;
case 16:
return numBytes / 2;
case 32:
return numBytes / 4;
default:
return numBytes;
}
}
/** value of a pixel in an image as it is sent by the detector */
uint32_t GetPixel(const uint8_t *data, const size_t pixel,
const uint32_t dynamicRange) {
switch (dynamicRange) {
case 4: {
// 2 pixels in a byte, the first one in the upper 4 bits (as in gui)
const uint8_t byte = data[pixel / 2];
return (pixel % 2 == 0) ? ((byte >> 4) & 0xF) : (byte & 0xF);
}
case 12: {
// 2 pixels in 3 bytes (as in the hdf5 file)
const uint8_t *bytes = data + (pixel / 2) * 3;
if (pixel % 2 == 0) {
return bytes[0] | ((bytes[1] & 0xF) << 8);
}
return ((bytes[1] & 0xF0) >> 4) | (bytes[2] << 4);
}
case 16: {
uint16_t value = 0;
memcpy(&value, data + pixel * sizeof(value), sizeof(value));
return value;
}
case 32: {
uint32_t value = 0;
memcpy(&value, data + pixel * sizeof(value), sizeof(value));
return value;
}
default:
return data[pixel];
}
}
} // namespace
RootDataFile::RootDataFile(const int index) : index(index) { InitializeRoot(); }
RootDataFile::~RootDataFile() { CloseFile(); }
void RootDataFile::ValidateOptions(
const std::vector<std::vector<uint32_t>> &options) {
// default
if (options.empty()) {
return;
}
if (options.size() != 1 || options[0].size() != 1 ||
options[0][0] >= static_cast<uint32_t>(NUM_ROOT_IMAGE_TYPES)) {
throw RuntimeError("Invalid file format options " + ToString(options) +
" for root file format. Options: " +
ToString(ROOT_TTREE) + ", " + ToString(ROOT_TH2I));
}
}
slsDetectorDefs::fileFormat RootDataFile::GetFileFormat() const { return ROOT; }
void RootDataFile::CloseFile() {
if (fd != nullptr) {
// writes the tree (replaces the one saved automatically while
// filling)
fd->Write(nullptr, TObject::kOverwrite);
fd->Close();
delete fd;
fd = nullptr;
}
// deleted by the file
tree = nullptr;
delete histogram;
histogram = nullptr;
}
void RootDataFile::CreateFirstRootDataFile(
const std::filesystem::path &filePath, const std::string &fNamePrefix,
const uint64_t fIndex, const bool ovEnable, const bool sMode,
const uint16_t uPortNumber, const uint32_t mFramesPerFile,
const uint32_t nX, const uint32_t nY, const uint32_t dr,
const std::vector<std::vector<uint32_t>> &options) {
ValidateOptions(options);
imageType = options.empty() ? ROOT_TTREE
: static_cast<rootImageType>(options[0][0]);
if (nX == 0 || nY == 0) {
throw RuntimeError("Could not create root file for an image of " +
std::to_string(nX) + " x " + std::to_string(nY) +
" pixels");
}
subFileIndex = 0;
numFramesInFile = 0;
m_filePath = filePath;
fileNamePrefix = fNamePrefix;
fileIndex = fIndex;
overWriteEnable = ovEnable;
silentMode = sMode;
udpPortNumber = uPortNumber;
maxFramesPerFile = mFramesPerFile;
nPixelsX = nX;
nPixelsY = nY;
dynamicRange = dr;
CreateFile();
}
void RootDataFile::CreateFile() {
numFramesInFile = 0;
std::filesystem::path p =
m_filePath / (fileNamePrefix + "_f" + std::to_string(subFileIndex) +
'_' + std::to_string(fileIndex) + ".root");
fileName = p.string();
// CREATE fails if the file exists
fd = TFile::Open(fileName.c_str(), overWriteEnable ? "RECREATE" : "CREATE");
if (fd == nullptr || fd->IsZombie()) {
delete fd;
fd = nullptr;
if (!overWriteEnable) {
throw RuntimeError("Could not create/overwrite file " + fileName);
}
throw RuntimeError("Could not create file " + fileName);
}
// header of every image
tree = new TTree(TREE_NAME, "one entry per image");
tree->SetDirectory(fd);
tree->Branch("frameNumber", &detHeader.frameNumber, "frameNumber/l");
tree->Branch("expLength", &detHeader.expLength, "expLength/i");
tree->Branch("packetNumber", &detHeader.packetNumber, "packetNumber/i");
tree->Branch("detSpec1", &detHeader.detSpec1, "detSpec1/l");
tree->Branch("timestamp", &detHeader.timestamp, "timestamp/l");
tree->Branch("modId", &detHeader.modId, "modId/s");
tree->Branch("row", &detHeader.row, "row/s");
tree->Branch("column", &detHeader.column, "column/s");
tree->Branch("detSpec2", &detHeader.detSpec2, "detSpec2/s");
tree->Branch("detSpec3", &detHeader.detSpec3, "detSpec3/i");
tree->Branch("detSpec4", &detHeader.detSpec4, "detSpec4/s");
tree->Branch("detType", &detHeader.detType, "detType/b");
tree->Branch("version", &detHeader.version, "version/b");
const std::string maskLeaf =
"packetsMask[" + std::to_string(sizeof(packetsMask)) + "]/b";
tree->Branch("packetsMask", packetsMask, maskLeaf.c_str());
// image
if (imageType == ROOT_TH2I) {
// one bin per pixel
histogram = new TH2I(HISTOGRAM_NAME_PREFIX, HISTOGRAM_NAME_PREFIX,
nPixelsX, 0, nPixelsX, nPixelsY, 0, nPixelsY);
histogram->SetDirectory(nullptr);
} else {
// unsigned integers of the size of the dynamic range
// (b: 8 bit, s: 16 bit, i: 32 bit)
char leafType = 'b';
size_t bytesPerValue = 1;
switch (dynamicRange) {
case 12:
case 16:
leafType = 's';
bytesPerValue = 2;
break;
case 32:
leafType = 'i';
bytesPerValue = 4;
break;
default:
break;
}
// 4 bit is saved as it is sent by the detector (2 pixels in a byte)
const size_t numColumns =
(dynamicRange == 4) ? (nPixelsX + 1) / 2 : nPixelsX;
imageBuffer.assign(nPixelsY * numColumns * bytesPerValue, 0);
const std::string imageLeaf =
std::string(IMAGE_BRANCH_NAME) + '[' + std::to_string(nPixelsY) +
"][" + std::to_string(numColumns) + "]/" + leafType;
// the buffer of the branch holds at least one image
const size_t basketSize =
std::min(std::max(DEFAULT_BASKET_SIZE, imageBuffer.size()),
static_cast<size_t>(std::numeric_limits<Int_t>::max()));
if (tree->Branch(IMAGE_BRANCH_NAME, imageBuffer.data(),
imageLeaf.c_str(),
static_cast<Int_t>(basketSize)) == nullptr) {
CloseFile();
throw RuntimeError("Could not create branch " + imageLeaf +
" in file " + fileName);
}
}
if (!silentMode) {
LOG(logINFO) << "[" << udpPortNumber
<< "]: Root File created: " << fileName;
}
}
void RootDataFile::WriteToFile(char *imageData, sls_receiver_header &header,
const int imageSize,
const uint64_t currentFrameNumber,
const uint32_t numPacketsCaught) {
// check if maxframesperfile = 0 for infinite
if (maxFramesPerFile && (numFramesInFile >= maxFramesPerFile)) {
CloseFile();
++subFileIndex;
CreateFile();
}
if (fd == nullptr || tree == nullptr) {
throw RuntimeError(std::to_string(index) +
" : No root file to write image number " +
std::to_string(currentFrameNumber));
}
++numFramesInFile;
// header of the image (the branches read from these members)
detHeader = header.detHeader;
// contiguous representation of bit mask
memset(packetsMask, 0, sizeof(packetsMask));
for (int i = 0; i < MAX_NUM_PACKETS; ++i)
packetsMask[i >> 3] |= (header.packetsMask[i] << (i & 7));
// image
const size_t numBytes = imageSize > 0 ? static_cast<size_t>(imageSize) : 0;
bool success = true;
if (imageType == ROOT_TH2I) {
FillHistogram(imageData, numBytes);
const std::string name = HISTOGRAM_NAME_PREFIX +
std::to_string(detHeader.frameNumber);
histogram->SetNameTitle(name.c_str(), name.c_str());
success = (fd->WriteTObject(histogram) > 0);
} else {
FillImageBuffer(imageData, numBytes);
}
// one entry in the tree
if (tree->Fill() < 0) {
success = false;
}
if (!success) {
throw RuntimeError(std::to_string(index) +
" : Write to file failed for image number " +
std::to_string(currentFrameNumber));
}
}
void RootDataFile::FillImageBuffer(const char *imageData,
const size_t imageSize) {
// 12 bit is saved as 16 bit, pixels without data are 0
if (dynamicRange == 12) {
const auto *data = reinterpret_cast<const uint8_t *>(imageData);
const size_t numPixels = imageBuffer.size() / sizeof(uint16_t);
const size_t numPixelsInData = GetNumPixels(imageSize, dynamicRange);
for (size_t pixel = 0; pixel != numPixels; ++pixel) {
const uint16_t value =
(pixel < numPixelsInData)
? static_cast<uint16_t>(
GetPixel(data, pixel, dynamicRange))
: 0;
memcpy(imageBuffer.data() + pixel * sizeof(value), &value,
sizeof(value));
}
return;
}
// saved as it is sent by the detector, bytes without data are 0
const size_t numBytes = std::min(imageSize, imageBuffer.size());
memcpy(imageBuffer.data(), imageData, numBytes);
std::fill(imageBuffer.begin() + numBytes, imageBuffer.end(), 0);
}
void RootDataFile::FillHistogram(const char *imageData,
const size_t imageSize) {
const auto *data = reinterpret_cast<const uint8_t *>(imageData);
const size_t numPixelsInData = GetNumPixels(imageSize, dynamicRange);
// bins are signed 32 bit integers
const auto maxValue =
static_cast<uint32_t>(std::numeric_limits<Int_t>::max());
size_t pixel = 0;
for (uint32_t y = 0; y != nPixelsY; ++y) {
for (uint32_t x = 0; x != nPixelsX; ++x) {
// pixels without data are 0
const uint32_t value = (pixel < numPixelsInData)
? GetPixel(data, pixel, dynamicRange)
: 0;
// bin 0 is the underflow bin
histogram->SetBinContent(x + 1, y + 1, std::min(value, maxValue));
++pixel;
}
}
// otherwise the number of calls to SetBinContent of all the images
histogram->SetEntries(static_cast<Double_t>(pixel));
}
} // namespace sls
+97
View File
@@ -0,0 +1,97 @@
// SPDX-License-Identifier: LGPL-3.0-or-other
// Copyright (C) 2021 Contributors to the SLS Detector Package
#pragma once
#include "File.h"
#include <string>
#include <vector>
// root classes (only used as pointers in this header)
class TFile;
class TTree;
class TH2I;
namespace sls {
/**
* Data file in root format (.root).
*
* Every file has a tree called "frames" with one entry per image. Its
* branches are the members of sls_detector_header and the packets mask.
*
* How the image itself is saved depends on the file format option:
* - ROOT_TTREE (default): as the branch "data" of the tree, which is a 2d
* array [rows][columns] of unsigned integers of the size of the dynamic
* range (as in the hdf5 file: 12 bit is saved as 16 bit and 4 bit is saved
* as it is sent by the detector, with 2 pixels in every byte).
* - ROOT_TH2I: as a histogram called "frame_[frame number]" with one bin per
* pixel. The bins are signed 32 bit integers. Larger values are saved as
* the maximum of a signed 32 bit integer.
*/
class RootDataFile : private virtual slsDetectorDefs, public File {
public:
RootDataFile(const int index);
~RootDataFile();
/**
* Throws if the file format options are not valid for a root file.
* Either no options (default) or a single option with a single value of
* rootImageType.
*/
static void
ValidateOptions(const std::vector<std::vector<uint32_t>> &options);
fileFormat GetFileFormat() const override;
void CloseFile() override;
void CreateFirstRootDataFile(
const std::filesystem::path &filePath, const std::string &fNamePrefix,
const uint64_t fIndex, const bool ovEnable, const bool sMode,
const uint16_t uPortNumber, const uint32_t mFramesPerFile,
const uint32_t nX, const uint32_t nY, const uint32_t dr,
const std::vector<std::vector<uint32_t>> &options) override;
void WriteToFile(char *imageData, sls_receiver_header &header,
const int imageSize, const uint64_t currentFrameNumber,
const uint32_t numPacketsCaught) override;
private:
void CreateFile();
/** [ROOT_TTREE] copies the image to the address of the data branch */
void FillImageBuffer(const char *imageData, const size_t imageSize);
/** [ROOT_TH2I] sets the bins of the histogram to the pixels */
void FillHistogram(const char *imageData, const size_t imageSize);
uint32_t index;
rootImageType imageType{ROOT_TTREE};
TFile *fd{nullptr};
/** one entry per image, owned by the file */
TTree *tree{nullptr};
/** [ROOT_TH2I] the same histogram is filled and written for every image */
TH2I *histogram{nullptr};
// addresses of the branches of the tree
sls_detector_header detHeader{};
bitset_storage packetsMask{};
/** [ROOT_TTREE] image as it is saved */
std::vector<char> imageBuffer;
std::string fileName;
std::filesystem::path m_filePath;
uint32_t numFramesInFile{0};
uint32_t subFileIndex{0};
std::string fileNamePrefix;
uint64_t fileIndex{0};
bool overWriteEnable{false};
bool silentMode{false};
uint32_t udpPortNumber{0};
uint32_t maxFramesPerFile{0};
uint32_t nPixelsX{0};
uint32_t nPixelsY{0};
uint32_t dynamicRange{0};
};
} // namespace sls
+2
View File
@@ -32,6 +32,7 @@ std::string ToString(const defs::speedLevel s);
std::string ToString(const defs::timingMode s);
std::string ToString(const defs::frameDiscardPolicy s);
std::string ToString(const defs::fileFormat s);
std::string ToString(const defs::rootImageType s);
std::string ToString(const defs::externalSignalFlag s);
std::string ToString(const defs::readoutMode s);
std::string ToString(const defs::dacIndex s);
@@ -369,6 +370,7 @@ template <> defs::speedLevel StringTo(const std::string &s);
template <> defs::timingMode StringTo(const std::string &s);
template <> defs::frameDiscardPolicy StringTo(const std::string &s);
template <> defs::fileFormat StringTo(const std::string &s);
template <> defs::rootImageType StringTo(const std::string &s);
template <> defs::externalSignalFlag StringTo(const std::string &s);
template <> defs::readoutMode StringTo(const std::string &s);
template <> defs::dacIndex StringTo(const std::string &s);
@@ -235,7 +235,12 @@ class slsDetectorDefs {
NUM_DISCARD_POLICIES
};
enum fileFormat { BINARY, HDF5, NUM_FILE_FORMATS };
enum fileFormat { BINARY, HDF5, ROOT, NUM_FILE_FORMATS };
/** file format option for root files: how the images are saved.
* ROOT_TTREE: one entry per image in a tree, ROOT_TH2I: one histogram
* per image */
enum rootImageType { ROOT_TTREE, ROOT_TH2I, NUM_ROOT_IMAGE_TYPES };
/**
@short structure for a region of interest
+27
View File
@@ -350,6 +350,19 @@ std::string ToString(const defs::fileFormat s) {
return std::string("hdf5");
case defs::BINARY:
return std::string("binary");
case defs::ROOT:
return std::string("root");
default:
return std::string("Unknown");
}
}
std::string ToString(const defs::rootImageType s) {
switch (s) {
case defs::ROOT_TTREE:
return std::string("TTree");
case defs::ROOT_TH2I:
return std::string("TH2I");
default:
return std::string("Unknown");
}
@@ -915,9 +928,23 @@ template <> defs::fileFormat StringTo(const std::string &s) {
return defs::HDF5;
if (s == "binary")
return defs::BINARY;
if (s == "root")
return defs::ROOT;
throw RuntimeError("Unknown file format " + s);
}
template <> defs::rootImageType StringTo(const std::string &s) {
if (s == "TTree")
return defs::ROOT_TTREE;
if (s == "ttree")
return defs::ROOT_TTREE;
if (s == "TH2I")
return defs::ROOT_TH2I;
if (s == "th2i")
return defs::ROOT_TH2I;
throw RuntimeError("Unknown root image type " + s);
}
template <> defs::externalSignalFlag StringTo(const std::string &s) {
if (s == "trigger_in_rising_edge")
return defs::TRIGGER_IN_RISING_EDGE;