leonarski_fandClaude Opus 5
1d16dcd2b9
Build Packages / build:windows:nocuda (push) Successful in 16m8s
Build Packages / build:windows:cuda (push) Successful in 18m58s
Build Packages / build:viewer-tgz:cpu (push) Successful in 20m35s
Build Packages / build:viewer-tgz:cuda (push) Successful in 22m31s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 25m9s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 25m6s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 28m57s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 28m58s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 28m58s
Build Packages / XDS test (durin plugin) (push) Successful in 12m3s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 22m24s
Build Packages / build:rpm (rocky9) (push) Successful in 21m45s
Build Packages / Generate python client (push) Successful in 53s
Build Packages / build:rpm (rocky8) (push) Successful in 26m9s
Build Packages / Create release (push) Skipped
Build Packages / Build documentation (push) Successful in 1m37s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 25m34s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 22m0s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 10m53s
Build Packages / XDS test (neggia plugin) (push) Successful in 9m29s
Build Packages / DIALS test (push) Successful in 23m40s
Build Packages / Unit tests (push) Successful in 1h20m1s
Three costs before and around the image loop. Every image allocated a fresh buffer for its compressed chunk and resized it, which value-initialises, and the read then overwrote every byte. At a few megabytes a chunk the allocation is large enough to be mapped rather than reused, so the zeroing was page-fault bound and cost more than the read it preceded - twenty gigabytes of it over a long sweep. The buffer now uses an allocator that does not construct, and the two HDF5 read paths are templated on the allocator so every existing caller compiles unchanged. The rebind is deliberate: without it the vector base rebinds to the default allocator and the zeroing quietly returns. The bitshuffle decoder allocated a whole uncompressed frame in its constructor - seventy megabytes a worker, five hundred and fifty across the loop - for the route that decodes the shuffled image separately. That route is taken only when a bitshuffle block is too large for the fused kernel, which neither writer this pipeline reads produces, so on a real frame the buffer is allocated, never touched, and freed. It is now allocated where it is used. The comment two lines below already warned against sizing a buffer from the uncompressed size; the line above it had not been given the same treatment. The first call into cuFFT pays the library's one-time initialisation, and it landed in the middle of the first pass with nothing to overlap it. It is now forced on a background thread at startup, alongside the file open and the mapping build, in the manner the shadow finder already uses. Finally, the detector mask was copied into the start message whether or not a file would carry it, which a merging run does not. It is filled where a writer is constructed - both places one is constructed, the second being the fallback that writes a process file when nothing indexed. Faster on eleven of thirty-eight crystals and slower on none; the whole rotation test set falls from four minutes thirty to four minutes seventeen, with each binary repeating itself to within half a per cent. Space groups thirty-five of thirty-eight and no failures throughout, and every column of the comparison table is identical. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EGpGdgmJ8MyY9pCGWjktyi
1333 lines
46 KiB
C++
1333 lines
46 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#include <cstring>
|
|
#include <filesystem>
|
|
#include <iostream>
|
|
|
|
#include <bitshuffle/bshuf_h5filter.h>
|
|
#include "HDF5Objects.h"
|
|
#include "H5FDpoison_sec2.h"
|
|
|
|
std::mutex hdf5_mutex;
|
|
|
|
static void HDF5PoisonCallback(
|
|
const char *filename,
|
|
const char *operation,
|
|
int error_number,
|
|
void *user_data
|
|
) {
|
|
std::cerr << "HDF5 Poison callback triggered: filename=" << filename << ", operation=" << operation << ", error_number=" << error_number << std::endl;
|
|
}
|
|
|
|
hid_t HDF5Id::GetID() const {
|
|
return id;
|
|
}
|
|
|
|
HDF5Id::HDF5Id(HDF5Id &&other) noexcept {
|
|
id = other.id;
|
|
other.id = -1;
|
|
}
|
|
|
|
HDF5Id::HDF5Id(const HDF5Id &other) {
|
|
id = other.id;
|
|
if (H5Iis_valid(id)) {
|
|
if (H5Iinc_ref(id) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot increment HDF5 Counter");
|
|
}
|
|
}
|
|
|
|
HDF5DataSpace::HDF5DataSpace(ScalarTag) : HDF5Id() {
|
|
id = H5Screate(H5S_SCALAR);
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create scalar dataspace");
|
|
|
|
ndims = 0;
|
|
}
|
|
|
|
HDF5DataSpace::HDF5DataSpace(const std::vector<hsize_t> &dims, const std::vector<hsize_t> &max_dims) : HDF5Id() {
|
|
if (dims.empty())
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Dimension vector empty");
|
|
|
|
if (max_dims.empty()) {
|
|
if (dims[0] == 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Value dimension cannot be 0");
|
|
|
|
id = H5Screate_simple(dims.size(), dims.data(), nullptr);
|
|
} else {
|
|
if (max_dims.size() != dims.size())
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Discrepancy in size of dims/max_dims");
|
|
id = H5Screate_simple(dims.size(), dims.data(), max_dims.data());
|
|
}
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create dataspace");
|
|
|
|
ndims = dims.size();
|
|
}
|
|
|
|
HDF5DataSpace::HDF5DataSpace(const HDF5DataSet &data_set) : HDF5Id() {
|
|
id = H5Dget_space(data_set.GetID());
|
|
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open dataspace");
|
|
|
|
int tmp = H5Sget_simple_extent_ndims(id);
|
|
|
|
if (tmp < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot check dimensions of dataspace");
|
|
|
|
ndims = tmp;
|
|
}
|
|
|
|
std::vector<hsize_t> HDF5DataSpace::GetDimensions() const {
|
|
std::vector<hsize_t> dims(ndims);
|
|
int tmp = H5Sget_simple_extent_dims(id, dims.data(), nullptr);
|
|
if ((tmp < 0) || (tmp != ndims))
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot check dimensions of dataspace");
|
|
return dims;
|
|
}
|
|
|
|
void HDF5DataSpace::SelectHyperslab(const std::vector<hsize_t> &start, const std::vector<hsize_t> &size) {
|
|
if ((start.size() != ndims) || (size.size() != ndims))
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Arrays are inconsistent with dataspace dimension number " + std::to_string(ndims));
|
|
|
|
if (H5Sselect_hyperslab(id, H5S_SELECT_SET, start.data(), NULL, size.data(), NULL) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot select hyperslab");
|
|
}
|
|
|
|
void HDF5DataSpace::SelectHyperslabWithStride(const std::vector<hsize_t> &start, const std::vector<hsize_t> &size,
|
|
const std::vector<hsize_t> &stride) {
|
|
if ((start.size() != ndims) || (stride.size() != ndims) || (size.size() != ndims))
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Arrays are inconsistent with dataspace dimension number");
|
|
|
|
if (H5Sselect_hyperslab(id, H5S_SELECT_SET, start.data(), stride.data(), size.data(), NULL) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot select hyperslab");
|
|
}
|
|
|
|
HDF5DataSpace::~HDF5DataSpace() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Sclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
uint8_t HDF5DataSpace::GetNumOfDimensions() const {
|
|
return ndims;
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(CompressedImageMode mode) : HDF5Id() {
|
|
switch (mode) {
|
|
|
|
case CompressedImageMode::Int8:
|
|
id = H5Tcopy(H5T_NATIVE_INT8);
|
|
break;
|
|
case CompressedImageMode::Int16:
|
|
id = H5Tcopy(H5T_NATIVE_INT16);
|
|
break;
|
|
case CompressedImageMode::Int32:
|
|
id = H5Tcopy(H5T_NATIVE_INT32);
|
|
break;
|
|
case CompressedImageMode::RGB:
|
|
case CompressedImageMode::Uint8:
|
|
id = H5Tcopy(H5T_NATIVE_UINT8);
|
|
break;
|
|
case CompressedImageMode::Uint16:
|
|
id = H5Tcopy(H5T_NATIVE_UINT16);
|
|
break;
|
|
case CompressedImageMode::Uint32:
|
|
id = H5Tcopy(H5T_NATIVE_UINT32);
|
|
break;
|
|
case CompressedImageMode::Float16:
|
|
id = H5Tcopy(H5T_IEEE_F16LE);
|
|
break;
|
|
case CompressedImageMode::Float32:
|
|
id = H5Tcopy(H5T_IEEE_F32LE);
|
|
break;
|
|
case CompressedImageMode::Float64:
|
|
id = H5Tcopy(H5T_IEEE_F64LE);
|
|
break;
|
|
}
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(double val) : HDF5Id() {
|
|
id = H5Tcopy(H5T_IEEE_F64LE);
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(float val) : HDF5Id() {
|
|
id = H5Tcopy(H5T_IEEE_F32LE);
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(bool val) : HDF5Id() {
|
|
id = H5Tcopy(H5T_NATIVE_HBOOL);
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(uint8_t val) : HDF5DataType(1, false) {}
|
|
HDF5DataType::HDF5DataType(int8_t val) : HDF5DataType(1, true) {}
|
|
|
|
HDF5DataType::HDF5DataType(uint16_t val) : HDF5DataType(2, false) {}
|
|
HDF5DataType::HDF5DataType(int16_t val) : HDF5DataType(2, true) {}
|
|
|
|
HDF5DataType::HDF5DataType(uint32_t val) : HDF5DataType(4, false) {}
|
|
HDF5DataType::HDF5DataType(int32_t val) : HDF5DataType(4, true) {}
|
|
|
|
HDF5DataType::HDF5DataType(uint64_t val) : HDF5DataType(8, false) {}
|
|
HDF5DataType::HDF5DataType(int64_t val) : HDF5DataType(8, true) {}
|
|
|
|
HDF5DataType::HDF5DataType(uint64_t size_in_bytes, bool is_signed) : HDF5Id() {
|
|
switch (size_in_bytes) {
|
|
case 1:
|
|
if (is_signed)
|
|
id = H5Tcopy(H5T_NATIVE_INT8);
|
|
else
|
|
id = H5Tcopy(H5T_NATIVE_UINT8);
|
|
break;
|
|
case 2:
|
|
if (is_signed)
|
|
id = H5Tcopy(H5T_NATIVE_INT16);
|
|
else
|
|
id = H5Tcopy(H5T_NATIVE_UINT16);
|
|
break;
|
|
case 4:
|
|
if (is_signed)
|
|
id = H5Tcopy(H5T_NATIVE_INT32);
|
|
else
|
|
id = H5Tcopy(H5T_NATIVE_UINT32);
|
|
break;
|
|
case 8:
|
|
if (is_signed)
|
|
id = H5Tcopy(H5T_NATIVE_INT64);
|
|
else
|
|
id = H5Tcopy(H5T_NATIVE_UINT64);
|
|
break;
|
|
default:
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Type outside of standard types");
|
|
}
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(const std::string& str) : HDF5Id() {
|
|
id = H5Tcopy(H5T_C_S1);
|
|
H5Tset_size(id, str.length() + 1);
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(const char * str) : HDF5Id() {
|
|
id = H5Tcopy(H5T_C_S1);
|
|
size_t length = strnlen(str,512);
|
|
H5Tset_size(id, length + 1);
|
|
}
|
|
|
|
HDF5DataType::HDF5DataType(const HDF5DataSet &data_set) :HDF5Id() {
|
|
id = H5Dget_type(data_set.GetID());
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open HDF5 data type");
|
|
}
|
|
|
|
HDF5DataType::~HDF5DataType() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Tclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
size_t HDF5DataType::GetElemSize() const {
|
|
return H5Tget_size(id);
|
|
}
|
|
|
|
bool HDF5DataType::IsSigned() const {
|
|
if (IsFloat())
|
|
return true;
|
|
else if (IsInteger())
|
|
return H5Tget_sign(id) == H5T_SGN_2;
|
|
else
|
|
return false;
|
|
}
|
|
|
|
bool HDF5DataType::IsInteger() const {
|
|
return H5Tget_class(id) == H5T_INTEGER;
|
|
}
|
|
|
|
bool HDF5DataType::IsFloat() const {
|
|
return H5Tget_class(id) == H5T_FLOAT;
|
|
}
|
|
|
|
HDF5Dcpl::HDF5Dcpl() : HDF5Id() {
|
|
id = H5Pcreate(H5P_DATASET_CREATE);
|
|
ndim = 0;
|
|
layout = HDF5DataSetLayout::CONTIGUOUS;
|
|
}
|
|
|
|
HDF5Dcpl::HDF5Dcpl(const HDF5DataSet &data_set) : HDF5Id() {
|
|
id = H5Dget_create_plist(data_set.GetID());
|
|
|
|
// Check if chunking is enabled
|
|
H5D_layout_t h5_layout = H5Pget_layout(id);
|
|
if (h5_layout == H5D_VIRTUAL)
|
|
layout = HDF5DataSetLayout::VIRTUAL;
|
|
else if (h5_layout == H5D_CHUNKED) {
|
|
layout = HDF5DataSetLayout::CHUNKED;
|
|
ndim = H5Pget_chunk(id, 0, nullptr);
|
|
if (ndim <= 0) {
|
|
H5Pclose(id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Error getting number of chunk dimensions");
|
|
}
|
|
} else
|
|
layout = HDF5DataSetLayout::CONTIGUOUS;
|
|
}
|
|
|
|
HDF5Dcpl::~HDF5Dcpl() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Pclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
void HDF5Dcpl::SetChunking(const std::vector<hsize_t> &dims) {
|
|
layout = HDF5DataSetLayout::CHUNKED;
|
|
if ((dims.empty()) || dims[0] == 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Value dimension cannot be 0");
|
|
ndim = dims.size();
|
|
if (H5Pset_chunk(id, dims.size(), dims.data()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot set chunking");
|
|
}
|
|
|
|
std::vector<hsize_t> HDF5Dcpl::GetChunking() {
|
|
if (ndim <= 0)
|
|
return {};
|
|
|
|
std::vector<hsize_t> ret(ndim);
|
|
if (H5Pget_chunk(id, ndim, ret.data()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get chunk dimensions");
|
|
|
|
return ret;
|
|
}
|
|
|
|
uint8_t HDF5Dcpl::GetNumOfDimensions() const {
|
|
return ndim;
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValue32(int32_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_INT32, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValue16(int16_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_INT16, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValue8(int8_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_INT8, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValueU32(uint32_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_UINT32, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValueU16(uint16_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_UINT16, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetFillValueU8(uint8_t val) {
|
|
H5Pset_fill_value(id, H5T_NATIVE_UINT8, &val);
|
|
}
|
|
|
|
void HDF5Dcpl::SetVirtual(const std::string &path, const std::string &dataset, const HDF5DataSpace &src_dataspace, const HDF5DataSpace &dest_dataspace) {
|
|
layout = HDF5DataSetLayout::VIRTUAL;
|
|
std::string filename = ExtractFilename(path);
|
|
if (H5Pset_virtual(id, dest_dataspace.GetID(), filename.c_str(), dataset.c_str(), src_dataspace.GetID()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot set virtual mapping");
|
|
}
|
|
|
|
HDF5DataSetLayout HDF5Dcpl::GetLayout() const {
|
|
return layout;
|
|
}
|
|
|
|
HDF5Fapl::HDF5Fapl() : HDF5Id() {
|
|
id = H5Pcreate(H5P_FILE_ACCESS);
|
|
}
|
|
|
|
HDF5Fapl::~HDF5Fapl() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Pclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
void HDF5Fapl::SetVersionTo1p10orNewer() {
|
|
H5Pset_libver_bounds(id, H5F_LIBVER_V110, H5F_LIBVER_LATEST);
|
|
}
|
|
|
|
void HDF5Fapl::SetCloseStrong() {
|
|
H5Pset_fclose_degree(id, H5F_CLOSE_STRONG);
|
|
}
|
|
|
|
void HDF5Fapl::SetPoisonSec2() {
|
|
if (H5Pset_fapl_poison_sec2(id) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot enable poisoned sec2 HDF5 VFD");
|
|
}
|
|
|
|
template <typename T>
|
|
static HDF5Object& WriteOrCreateScalarAttr(HDF5Object& object, const std::string& name, const T& val) {
|
|
HDF5DataSpace dataspace(HDF5DataSpace::Scalar);
|
|
HDF5DataType datatype(val);
|
|
|
|
hid_t attr_id = -1;
|
|
|
|
if (H5Aexists(object.GetID(), name.c_str()) > 0) {
|
|
attr_id = H5Aopen(object.GetID(), name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
hid_t existing_type = H5Aget_type(attr_id);
|
|
if (existing_type < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get attribute type " + name);
|
|
}
|
|
|
|
const bool recreate =
|
|
(H5Tget_class(existing_type) != H5Tget_class(datatype.GetID())) ||
|
|
(H5Tget_size(existing_type) != H5Tget_size(datatype.GetID()));
|
|
|
|
H5Tclose(existing_type);
|
|
H5Aclose(attr_id);
|
|
|
|
if (recreate) {
|
|
if (H5Adelete(object.GetID(), name.c_str()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot delete attribute " + name);
|
|
|
|
attr_id = H5Acreate2(object.GetID(), name.c_str(), datatype.GetID(), dataspace.GetID(), H5P_DEFAULT, H5P_DEFAULT);
|
|
} else {
|
|
attr_id = H5Aopen(object.GetID(), name.c_str(), H5P_DEFAULT);
|
|
}
|
|
} else {
|
|
attr_id = H5Acreate2(object.GetID(), name.c_str(), datatype.GetID(), dataspace.GetID(), H5P_DEFAULT, H5P_DEFAULT);
|
|
}
|
|
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create/open attribute " + name);
|
|
|
|
herr_t ret = H5Awrite(attr_id, datatype.GetID(), &val);
|
|
H5Aclose(attr_id);
|
|
|
|
if (ret < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute write unsuccessful");
|
|
return object;
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, const std::string &val) {
|
|
HDF5DataSpace dataspace(HDF5DataSpace::Scalar);
|
|
HDF5DataType datatype(val);
|
|
|
|
hid_t attr_id = -1;
|
|
|
|
if (H5Aexists(id, name.c_str()) > 0) {
|
|
attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
hid_t existing_type = H5Aget_type(attr_id);
|
|
if (existing_type < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get attribute type " + name);
|
|
}
|
|
|
|
const bool recreate =
|
|
(H5Tget_class(existing_type) != H5T_STRING) ||
|
|
(H5Tget_size(existing_type) < val.length() + 1);
|
|
|
|
H5Tclose(existing_type);
|
|
H5Aclose(attr_id);
|
|
|
|
if (recreate) {
|
|
if (H5Adelete(id, name.c_str()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot delete attribute " + name);
|
|
|
|
attr_id = H5Acreate2(id, name.c_str(), datatype.GetID(), dataspace.GetID(), H5P_DEFAULT, H5P_DEFAULT);
|
|
} else {
|
|
attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
}
|
|
} else {
|
|
attr_id = H5Acreate2(id, name.c_str(), datatype.GetID(), dataspace.GetID(), H5P_DEFAULT, H5P_DEFAULT);
|
|
}
|
|
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create/open attribute " + name);
|
|
|
|
herr_t ret = H5Awrite(attr_id, datatype.GetID(), val.c_str());
|
|
H5Aclose(attr_id);
|
|
|
|
if (ret < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute write unsuccessful");
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, int32_t val) {
|
|
return WriteOrCreateScalarAttr(*this, name, val);
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, uint32_t val) {
|
|
return WriteOrCreateScalarAttr(*this, name, val);
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, int64_t val) {
|
|
return WriteOrCreateScalarAttr(*this, name, val);
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, uint64_t val) {
|
|
return WriteOrCreateScalarAttr(*this, name, val);
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, double val) {
|
|
return WriteOrCreateScalarAttr(*this, name, val);
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Attr(const std::string &name, const std::vector<double> &val) {
|
|
HDF5DataSpace dataspace({val.size()});
|
|
if (val.empty()) throw JFJochException(JFJochExceptionCategory::HDF5, "Empty array cannot be written");
|
|
HDF5DataType datatype(val[0]);
|
|
|
|
hid_t attr_id = H5Acreate2(id, name.c_str(), datatype.GetID(), dataspace.GetID(), H5P_DEFAULT, H5P_DEFAULT);
|
|
herr_t ret = H5Awrite(attr_id, datatype.GetID(), val.data());
|
|
H5Aclose(attr_id);
|
|
|
|
if (ret < 0) throw JFJochException(JFJochExceptionCategory::HDF5, "Atrribute write unsucessful");
|
|
return *this;
|
|
}
|
|
|
|
std::vector<double> HDF5Object::ReadAttrVec(const std::string &name) {
|
|
hid_t attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
hid_t space_id = H5Aget_space(attr_id);
|
|
if (space_id < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get dataspace for attribute " + name);
|
|
}
|
|
|
|
int ndims = H5Sget_simple_extent_ndims(space_id);
|
|
if (ndims != 1) {
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute " + name + " is not a vector");
|
|
}
|
|
|
|
hsize_t dims[1];
|
|
H5Sget_simple_extent_dims(space_id, dims, NULL);
|
|
std::vector<double> ret(dims[0]);
|
|
|
|
if (H5Aread(attr_id, H5T_NATIVE_DOUBLE, ret.data()) < 0) {
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read attribute " + name);
|
|
}
|
|
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
return ret;
|
|
}
|
|
|
|
bool HDF5Object::AttrExists(const std::string &name) {
|
|
return H5Aexists(id, name.c_str()) > 0;
|
|
}
|
|
|
|
std::string HDF5Object::ReadAttrStr(const std::string &name) {
|
|
hid_t attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
// Get attribute dataspace
|
|
hid_t space_id = H5Aget_space(attr_id);
|
|
if (space_id < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get dataspace for attribute " + name);
|
|
}
|
|
|
|
// Check if attribute is scalar
|
|
H5S_class_t space_type = H5Sget_simple_extent_type(space_id);
|
|
if (space_type != H5S_SCALAR) {
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute " + name + " is not a scalar");
|
|
}
|
|
H5Sclose(space_id);
|
|
|
|
// Get attribute type
|
|
hid_t type_id = H5Aget_type(attr_id);
|
|
if (type_id < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get type for attribute " + name);
|
|
}
|
|
|
|
// Check if attribute is a string type
|
|
H5T_class_t type_class = H5Tget_class(type_id);
|
|
if (type_class != H5T_STRING) {
|
|
H5Tclose(type_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute " + name + " is not a string");
|
|
}
|
|
|
|
// Get string size and read it
|
|
size_t size = H5Tget_size(type_id);
|
|
std::vector<char> buffer(size + 1, '\0'); // +1 for null terminator to be safe
|
|
|
|
if (H5Aread(attr_id, type_id, buffer.data()) < 0) {
|
|
H5Tclose(type_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read attribute " + name);
|
|
}
|
|
|
|
H5Tclose(type_id);
|
|
H5Aclose(attr_id);
|
|
return {buffer.data()};
|
|
}
|
|
|
|
|
|
double HDF5Object::ReadAttrDouble(const std::string &name) {
|
|
hid_t attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
hid_t space_id = H5Aget_space(attr_id);
|
|
if (space_id < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get dataspace for attribute " + name);
|
|
}
|
|
|
|
H5S_class_t space_type = H5Sget_simple_extent_type(space_id);
|
|
if (space_type != H5S_SCALAR) {
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute " + name + " is not a scalar");
|
|
}
|
|
H5Sclose(space_id);
|
|
|
|
double value;
|
|
if (H5Aread(attr_id, H5T_NATIVE_DOUBLE, &value) < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read attribute " + name);
|
|
}
|
|
|
|
H5Aclose(attr_id);
|
|
return value;
|
|
}
|
|
|
|
int64_t HDF5Object::ReadAttrInt(const std::string &name) {
|
|
hid_t attr_id = H5Aopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (attr_id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open attribute " + name);
|
|
|
|
hid_t space_id = H5Aget_space(attr_id);
|
|
if (space_id < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot get dataspace for attribute " + name);
|
|
}
|
|
|
|
H5S_class_t space_type = H5Sget_simple_extent_type(space_id);
|
|
if (space_type != H5S_SCALAR) {
|
|
H5Sclose(space_id);
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Attribute " + name + " is not a scalar");
|
|
}
|
|
H5Sclose(space_id);
|
|
|
|
int64_t value;
|
|
if (H5Aread(attr_id, H5T_NATIVE_INT64, &value) < 0) {
|
|
H5Aclose(attr_id);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read attribute " + name);
|
|
}
|
|
|
|
H5Aclose(attr_id);
|
|
return value;
|
|
}
|
|
|
|
HDF5Object & HDF5Object::Units(const std::string &val) {
|
|
if (!val.empty()) Attr("units", val);
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object & HDF5Object::NXClass(const std::string &val) {
|
|
if (!val.empty()) Attr("NX_class", val);
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object& HDF5Object::Transformation(const std::string& units, const std::string& depends_on,
|
|
const std::string& equipment, const std::string& equipment_component,
|
|
const std::string& transformation_type, const std::vector<double>& vector) {
|
|
if (!units.empty()) Attr("units", units);
|
|
if (!depends_on.empty()) Attr("depends_on", depends_on);
|
|
if (!equipment.empty()) Attr("equipment", equipment);
|
|
if (!equipment_component.empty()) Attr("equipment_component", equipment_component);
|
|
if (!transformation_type.empty()) Attr("transformation_type", transformation_type);
|
|
Attr("vector", vector);
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object& HDF5Object::Transformation(const std::string& units, const std::string& depends_on,
|
|
const std::string& equipment, const std::string& equipment_component,
|
|
const std::string& transformation_type, const std::vector<double>& vector,
|
|
const std::vector<double> &offset, const std::string& offset_units) {
|
|
Transformation(units, depends_on, equipment, equipment_component, transformation_type, vector);
|
|
Attr("offset", offset);
|
|
// An offset is a length, whatever the axis is. Without this attribute a reader falls back to
|
|
// `units`, which on a rotation axis is degrees - NXmx readers then try to convert degrees to
|
|
// millimetres and raise. Harmless while every offset we write is zero, and wrong the moment one
|
|
// is not, so state it always rather than only when a caller remembers to.
|
|
Attr("offset_units", offset_units.empty() ? std::string("m") : offset_units);
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object& HDF5Object::ExternalLink(std::string filename, const std::string &external_dataset, const std::string &local_name) {
|
|
// Linked file is assumed to be in the same directory as current file, so directory information is stripped
|
|
int pos = filename.rfind('/');
|
|
if (pos != std::string::npos) filename = filename.substr(pos+1);
|
|
|
|
if (H5Lcreate_external(filename.c_str(),external_dataset.c_str(), GetID(),
|
|
local_name.c_str(), H5P_DEFAULT, H5P_DEFAULT) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create external link in HDF5 file");
|
|
return *this;
|
|
}
|
|
|
|
HDF5Object & HDF5Object::HardLink(const std::string &source, const std::string &dest) {
|
|
if (H5Lcreate_hard(GetID(), source.c_str(), GetID(), dest.c_str(), H5P_DEFAULT, H5P_DEFAULT) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create link in HDF5 file");
|
|
return *this;
|
|
}
|
|
|
|
std::vector<hsize_t> HDF5Object::GetDimension(const std::string &name) {
|
|
HDF5DataSet dataset(*this, name);
|
|
HDF5DataSpace dscp(dataset);
|
|
return dscp.GetDimensions();
|
|
}
|
|
|
|
std::unique_ptr<HDF5DataSet> HDF5Object::SaveScalar(const std::string &name, const std::string &val) {
|
|
return SaveScalar(name, val.c_str());
|
|
}
|
|
|
|
std::unique_ptr<HDF5DataSet> HDF5Object::SaveScalar(const std::string &name, const char *val) {
|
|
HDF5DataType data_type(val);
|
|
HDF5DataSpace data_space(HDF5DataSpace::Scalar);
|
|
auto dataset = std::make_unique<HDF5DataSet>(*this, name, data_type, data_space);
|
|
dataset->Write(data_type, val);
|
|
return dataset;
|
|
}
|
|
|
|
std::unique_ptr<HDF5DataSet> HDF5Object::SaveVector(const std::string &name, const std::vector<std::string> &val) {
|
|
if (val.empty())
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot write empty vector");
|
|
|
|
int pos = 0;
|
|
size_t len = val[0].length();
|
|
for (int i = 1; i < val.size(); i++) { if (val[i].length() > len) {pos = i; len = val[i].length();}}
|
|
HDF5DataType data_type(val[pos]);
|
|
HDF5Dcpl dcpl;
|
|
HDF5DataSpace data_space({val.size()});
|
|
auto dataset = std::make_unique<HDF5DataSet>(*this, name, data_type, data_space, dcpl);
|
|
|
|
std::vector<char> buffer((len + 1) * val.size(), '\0');
|
|
for (int i = 0; i < val.size(); i++) strncpy(buffer.data() + i * (len+1), val[i].c_str(), len+1);
|
|
dataset->Write(data_type, buffer.data());
|
|
return dataset;
|
|
}
|
|
|
|
HDF5Group::HDF5Group(const HDF5Object& parent, const std::string &name) : HDF5Group(parent, name.c_str()) {
|
|
}
|
|
|
|
HDF5Group::HDF5Group(const HDF5Object& parent, const char *name) : HDF5Object() {
|
|
if (H5Lexists(parent.GetID(), name, H5P_DEFAULT) > 0)
|
|
id = H5Gopen(parent.GetID(), name, H5P_DEFAULT);
|
|
else
|
|
id = H5Gcreate(parent.GetID(), name, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
|
|
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open/create HDF5 group " + std::string(name));
|
|
}
|
|
|
|
HDF5Group::~HDF5Group() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Gclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
HDF5File::HDF5File(const std::string& filename, bool v1_10) : HDF5Object() {
|
|
HDF5Fapl fapl;
|
|
|
|
H5FD_poison_sec2_init();
|
|
H5FD_poison_sec2_set_callback(HDF5PoisonCallback, nullptr);
|
|
if (v1_10)
|
|
fapl.SetVersionTo1p10orNewer();
|
|
fapl.SetCloseStrong();
|
|
fapl.SetPoisonSec2();
|
|
|
|
id = H5Fcreate(filename.c_str(), H5F_ACC_TRUNC, H5P_DEFAULT, fapl.GetID());
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open/create data HDF5 file " + filename);
|
|
}
|
|
|
|
void HDF5File::Close() {
|
|
if (id < 0)
|
|
return;
|
|
|
|
// Invalidate first; if anything below fails we must NOT leave a live id
|
|
// behind for the destructor or later code to touch.
|
|
const hid_t local_id = id;
|
|
id = -1;
|
|
|
|
H5FD_poison_sec2_begin_fail_pass();
|
|
|
|
herr_t close_err = 0;
|
|
H5E_BEGIN_TRY {
|
|
close_err = H5Fclose(local_id);
|
|
} H5E_END_TRY;
|
|
|
|
H5FD_poison_sec2_end_fail_pass();
|
|
|
|
const char *filename = nullptr;
|
|
const char *operation = nullptr;
|
|
int error_number = 0;
|
|
const int poison_error = H5FD_poison_sec2_get_last_error(
|
|
&filename,
|
|
&operation,
|
|
&error_number
|
|
);
|
|
|
|
if (close_err < 0 || poison_error > 0)
|
|
throw JFJochException(
|
|
JFJochExceptionCategory::HDF5,
|
|
"Failed to close HDF5 file, operation=" +
|
|
std::string(operation != nullptr ? operation : "") +
|
|
", filename=" +
|
|
std::string(filename != nullptr ? filename : "") +
|
|
", errno=" +
|
|
std::to_string(error_number)
|
|
);
|
|
}
|
|
|
|
HDF5File::~HDF5File() {
|
|
if (id >= 0) {
|
|
const hid_t tmp = id;
|
|
id = -1;
|
|
H5FD_poison_sec2_begin_fail_pass();
|
|
H5E_BEGIN_TRY {
|
|
H5Fclose(tmp);
|
|
} H5E_END_TRY;
|
|
H5FD_poison_sec2_end_fail_pass();
|
|
}
|
|
}
|
|
|
|
void HDF5File::Delete(const std::string& path) {
|
|
H5Ldelete(id, path.c_str(), H5P_DEFAULT);
|
|
}
|
|
|
|
HDF5ReadOnlyFile::HDF5ReadOnlyFile(const std::string &filename) {
|
|
id = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, H5P_DEFAULT);
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open/create data HDF5 file" + filename);
|
|
}
|
|
|
|
HDF5ReadOnlyFile::~HDF5ReadOnlyFile() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {H5Fclose(id); } H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
HDF5DataSet::HDF5DataSet(const HDF5Object &parent, const std::string &name, const HDF5DataType &data_type,
|
|
const HDF5DataSpace &data_space, const HDF5Dcpl &dcpl) : dataset_name(name) {
|
|
|
|
ndim = data_space.GetNumOfDimensions();
|
|
if ((dcpl.GetNumOfDimensions() != 0) && (dcpl.GetNumOfDimensions() != data_space.GetNumOfDimensions()))
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Inconsistent dimension settings");
|
|
|
|
id = H5Dcreate2(parent.GetID(), name.c_str(), data_type.GetID(), data_space.GetID(), H5P_DEFAULT, dcpl.GetID(), H5P_DEFAULT);
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create HDF5 dataset: " + name);
|
|
}
|
|
|
|
HDF5DataSet::HDF5DataSet(const HDF5Object &parent, const std::string &name, const HDF5DataType &data_type,
|
|
const HDF5DataSpace &data_space) : dataset_name(name) {
|
|
|
|
ndim = data_space.GetNumOfDimensions();
|
|
|
|
id = H5Dcreate2(parent.GetID(), name.c_str(), data_type.GetID(), data_space.GetID(), H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot create HDF5 dataset " + name);
|
|
}
|
|
|
|
HDF5DataSet::HDF5DataSet(const HDF5Object &parent, const std::string &name) : ndim(0), dataset_name(name) {
|
|
id = H5Dopen2(parent.GetID(), name.c_str(), H5P_DEFAULT);
|
|
|
|
if (id < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open HDF5 dataset " + name);
|
|
|
|
ndim = HDF5DataSpace(*this).GetNumOfDimensions();
|
|
}
|
|
|
|
void HDF5DataSet::SetExtent(const std::vector<hsize_t> &dims) {
|
|
if (dims.size() != ndim)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Mismatch in dataset dimensions " + std::to_string(ndim));
|
|
herr_t ret = H5Dset_extent(id, dims.data());
|
|
if (ret < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Error in H5Dset_extent");
|
|
}
|
|
|
|
HDF5DataSet& HDF5DataSet::Write(const HDF5DataType &data_type, const void *val) {
|
|
if (H5Dwrite(id, data_type.GetID(), H5S_ALL, H5S_ALL, H5P_DEFAULT, val) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot write to HDF5 file");
|
|
return *this;
|
|
}
|
|
|
|
HDF5DataSet& HDF5DataSet::Flush() {
|
|
if (H5Dflush(id) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Error flushing HDF5 dataset");
|
|
return *this;
|
|
}
|
|
|
|
HDF5DataSet& HDF5DataSet::WriteDirectChunk(const void *data, hsize_t data_size, const std::vector<hsize_t> &offset) {
|
|
if (offset.size() != ndim)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Inconsistent dimension settings");
|
|
if (H5Dwrite_chunk(id, H5P_DEFAULT, 0, offset.data(), data_size, data) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Error writing dataset to HDF5 file");
|
|
return *this;
|
|
}
|
|
|
|
std::string HDF5DataSet::ReadString() const {
|
|
HDF5DataSpace file_space(*this);
|
|
if (file_space.GetNumOfDimensions() != 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Dataset tries to read string (scalar) from vector dataset");
|
|
|
|
HDF5DataType file_data_type(*this);
|
|
const size_t size = file_data_type.GetElemSize();
|
|
|
|
if (size == 0)
|
|
return "";
|
|
|
|
std::string buffer(size, '\0');
|
|
|
|
if (H5Dread(id, file_data_type.GetID(), H5S_ALL, H5S_ALL, H5P_DEFAULT, buffer.data()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
dataset_name + ": read unsuccessful");
|
|
|
|
const size_t end = buffer.find('\0');
|
|
if (end != std::string::npos)
|
|
buffer.resize(end);
|
|
|
|
return buffer;
|
|
}
|
|
|
|
void HDF5DataSet::Close() {
|
|
if (id < 0)
|
|
return;
|
|
|
|
const hid_t local_id = id;
|
|
id = -1;
|
|
|
|
herr_t err = 0;
|
|
H5E_BEGIN_TRY {
|
|
err = H5Dclose(local_id);
|
|
} H5E_END_TRY;
|
|
|
|
if (err < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot close HDF5 dataset");
|
|
}
|
|
|
|
HDF5DataSet::~HDF5DataSet() {
|
|
if (id >= 0) {
|
|
H5E_BEGIN_TRY {
|
|
H5Dclose(id);
|
|
} H5E_END_TRY;
|
|
id = -1;
|
|
}
|
|
}
|
|
|
|
void RegisterHDF5Filter() {
|
|
if (H5Zregister(bshuf_H5Filter) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot register Bitshuffle filter");
|
|
}
|
|
|
|
void HDF5Dcpl::SetCompression(CompressionAlgorithm c, size_t block_size) {
|
|
constexpr uint32_t cd_nelmts = 2;
|
|
unsigned int params[cd_nelmts];
|
|
switch (c) {
|
|
case CompressionAlgorithm::BSHUF_LZ4:
|
|
params[0] = block_size;
|
|
params[1] = BSHUF_H5_COMPRESS_LZ4;
|
|
if (H5Pset_filter(id, (H5Z_filter_t)BSHUF_H5FILTER, H5Z_FLAG_MANDATORY, cd_nelmts, params) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot set bshuf/lz4 filter");
|
|
break;
|
|
case CompressionAlgorithm::BSHUF_ZSTD:
|
|
case CompressionAlgorithm::BSHUF_ZSTD_RLE:
|
|
case CompressionAlgorithm::BSHUF_ZSTD_RLE_HUFF:
|
|
#ifdef USE_ZSTD
|
|
params[0] = block_size;
|
|
params[1] = BSHUF_H5_COMPRESS_ZSTD;
|
|
if (H5Pset_filter(id, (H5Z_filter_t)BSHUF_H5FILTER, H5Z_FLAG_MANDATORY, cd_nelmts, params) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot set bshuf/zstd filter");
|
|
break;
|
|
#else
|
|
throw SLSException(SLSExceptionCategory::Compression, "ZSTD support not available");
|
|
#endif
|
|
case CompressionAlgorithm::NO_COMPRESSION:
|
|
break;
|
|
default:
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "SetCompression: algorithm not supported");
|
|
}
|
|
|
|
}
|
|
|
|
CompressionAlgorithm HDF5Dcpl::GetCompression() {
|
|
int num_filters = H5Pget_nfilters(id);
|
|
if (num_filters > 1)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Only single filter supported");
|
|
if (num_filters == 0)
|
|
return CompressionAlgorithm::NO_COMPRESSION;
|
|
|
|
uint32_t cd_values[8];
|
|
size_t cd_nelemts = sizeof(cd_values);
|
|
memset(cd_values, 0, sizeof(cd_values));
|
|
|
|
if (H5Pget_filter2(id, 0, nullptr, &cd_nelemts, cd_values, 0, nullptr, nullptr)
|
|
!= BSHUF_H5FILTER)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,"Only Bitshuffle filter supported");
|
|
|
|
if (cd_values[4] == BSHUF_H5_COMPRESS_LZ4)
|
|
return CompressionAlgorithm::BSHUF_LZ4;
|
|
if (cd_values[4] == BSHUF_H5_COMPRESS_ZSTD)
|
|
return CompressionAlgorithm::BSHUF_ZSTD;
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,"Weird value of parameter " + std::to_string(cd_values[1]));
|
|
}
|
|
|
|
std::string ExtractFilename(const std::string& str) {
|
|
std::filesystem::path path(str);
|
|
// if (std::filesystem::is_regular_file(path))
|
|
return path.filename().string();
|
|
}
|
|
|
|
bool HDF5Object::GetBool(const std::string &name) const {
|
|
HDF5DataSet dataset(*this, name);
|
|
return dataset.ReadScalar<bool>();
|
|
}
|
|
|
|
std::optional<bool> HDF5Object::GetOptBool(const std::string &name) const {
|
|
if (Exists(name))
|
|
return GetBool(name);
|
|
return std::nullopt;
|
|
}
|
|
|
|
float HDF5Object::GetFloat(const std::string &name) const {
|
|
HDF5DataSet dataset(*this, name);
|
|
return dataset.ReadScalar<float>();
|
|
}
|
|
|
|
int64_t HDF5Object::GetInt(const std::string &name) const {
|
|
HDF5DataSet dataset(*this, name);
|
|
return dataset.ReadScalar<int64_t>();
|
|
}
|
|
|
|
std::optional<float> HDF5Object::GetOptFloat(const std::string &name) const {
|
|
if (Exists(name))
|
|
return GetFloat(name);
|
|
return std::nullopt;
|
|
}
|
|
|
|
std::optional<int64_t> HDF5Object::GetOptInt(const std::string &name) const {
|
|
if (Exists(name))
|
|
return GetInt(name);
|
|
return std::nullopt;
|
|
}
|
|
|
|
std::string HDF5Object::GetString(const std::string &name, const std::string &def) const {
|
|
if (Exists(name)) {
|
|
HDF5DataSet dataset(*this, name);
|
|
return dataset.ReadString();
|
|
}
|
|
return def;
|
|
}
|
|
|
|
bool HDF5Object::Exists(const std::string &name) const {
|
|
H5E_BEGIN_TRY {
|
|
return H5Lexists(GetID(), name.c_str(), H5P_DEFAULT) > 0;
|
|
} H5E_END_TRY;
|
|
return false;
|
|
}
|
|
|
|
herr_t func(hid_t group, const char *name, const H5L_info2_t *info, void *op_data) {
|
|
auto ret = reinterpret_cast<std::vector<std::string> *>(op_data);
|
|
ret->emplace_back(name);
|
|
return 0;
|
|
}
|
|
|
|
std::vector<std::string> HDF5Object::FindLeafs(const std::string &name) const {
|
|
hid_t gid = H5Gopen(id, name.c_str(), H5P_DEFAULT);
|
|
if (gid < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot open HDF5 group " + name);
|
|
std::vector<std::string> ret;
|
|
hsize_t idx = 0;
|
|
H5Literate(gid, H5_INDEX_NAME, H5_ITER_INC, &idx, func, &ret);
|
|
|
|
H5Gclose(gid);
|
|
return ret;
|
|
}
|
|
|
|
bool HDF5Object::IsExternalLink(const std::string& name) const {
|
|
H5L_info2_t link_info;
|
|
|
|
// Get information about the link
|
|
if (H5Lget_info(id, name.c_str(), &link_info, H5P_DEFAULT) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Failed to retrieve information about the link");
|
|
|
|
return (link_info.type == H5L_TYPE_EXTERNAL);
|
|
}
|
|
|
|
std::string HDF5Object::GetLinkedFileName(const std::string& name) const {
|
|
H5L_info2_t link_info;
|
|
|
|
// Get information about the link
|
|
if (H5Lget_info(id, name.c_str(), &link_info, H5P_DEFAULT) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Failed to retrieve information about the link");
|
|
|
|
if (link_info.type != H5L_TYPE_EXTERNAL)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"This is not an external link");
|
|
|
|
std::vector<char> link_target(link_info.u.val_size + 2, '\0');
|
|
if (H5Lget_val(id,
|
|
name.c_str(),
|
|
link_target.data(),
|
|
link_info.u.val_size + 1, H5P_DEFAULT) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Failed to get link location");
|
|
|
|
const char *target_file_name = nullptr; // To store unpacked file name
|
|
const char *target_object_path = nullptr; // To store unpacked object path
|
|
|
|
// Unpack the external link's value
|
|
if (H5Lunpack_elink_val(link_target.data(),
|
|
link_info.u.val_size,
|
|
nullptr,
|
|
&target_file_name,
|
|
&target_object_path) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Failed to get link location");
|
|
|
|
std::string s(target_file_name);
|
|
|
|
return s;
|
|
}
|
|
|
|
|
|
namespace {
|
|
std::vector<hsize_t> GetDataspaceDimensions(hid_t dataspace_id) {
|
|
const int ndims = H5Sget_simple_extent_ndims(dataspace_id);
|
|
if (ndims < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read dataspace dimensions");
|
|
|
|
std::vector<hsize_t> dims(ndims);
|
|
if (ndims > 0 && H5Sget_simple_extent_dims(dataspace_id, dims.data(), nullptr) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5, "Cannot read dataspace dimensions");
|
|
|
|
return dims;
|
|
}
|
|
|
|
void GetRegularSelection(hid_t dataspace_id,
|
|
std::vector<hsize_t> &start,
|
|
std::vector<hsize_t> &stride,
|
|
std::vector<hsize_t> &count,
|
|
std::vector<hsize_t> &block) {
|
|
const auto dims = GetDataspaceDimensions(dataspace_id);
|
|
|
|
const H5S_sel_type selection_type = H5Sget_select_type(dataspace_id);
|
|
if (selection_type == H5S_SEL_ALL) {
|
|
start.clear();
|
|
stride.clear();
|
|
count.clear();
|
|
block.clear();
|
|
return;
|
|
}
|
|
|
|
start.assign(dims.size(), 0);
|
|
stride.assign(dims.size(), 1);
|
|
count.assign(dims.size(), 1);
|
|
block = dims;
|
|
|
|
if (selection_type != H5S_SEL_HYPERSLABS)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Only regular hyperslab VDS selections are supported");
|
|
|
|
if (H5Sis_regular_hyperslab(dataspace_id) <= 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Only regular hyperslab VDS selections are supported");
|
|
|
|
if (H5Sget_regular_hyperslab(dataspace_id,
|
|
start.data(),
|
|
stride.data(),
|
|
count.data(),
|
|
block.data()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot decode VDS hyperslab selection");
|
|
}
|
|
|
|
bool ContainsInRegularHyperslab(hsize_t value,
|
|
hsize_t start,
|
|
hsize_t stride,
|
|
hsize_t count,
|
|
hsize_t block) {
|
|
for (hsize_t i = 0; i < count; i++) {
|
|
const hsize_t block_start = start + i * stride;
|
|
if ((value >= block_start) && (value < block_start + block))
|
|
return true;
|
|
}
|
|
return false;
|
|
}
|
|
|
|
hsize_t IndexInRegularHyperslab(hsize_t value,
|
|
hsize_t start,
|
|
hsize_t stride,
|
|
hsize_t count,
|
|
hsize_t block) {
|
|
for (hsize_t i = 0; i < count; i++) {
|
|
const hsize_t block_start = start + i * stride;
|
|
if ((value >= block_start) && (value < block_start + block))
|
|
return i * block + (value - block_start);
|
|
}
|
|
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Image is not part of VDS hyperslab");
|
|
}
|
|
|
|
hsize_t ValueFromRegularHyperslabIndex(hsize_t index,
|
|
hsize_t start,
|
|
hsize_t stride,
|
|
hsize_t count,
|
|
hsize_t block) {
|
|
if (index >= count * block)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Source image is outside of VDS source hyperslab");
|
|
|
|
const hsize_t block_number = index / block;
|
|
const hsize_t in_block = index % block;
|
|
return start + block_number * stride + in_block;
|
|
}
|
|
}
|
|
|
|
bool HDF5VirtualDatasetMapping::ContainsVirtualImage(hsize_t image_number) const {
|
|
if (virtual_start.empty() || virtual_stride.empty() || virtual_count.empty() || virtual_block.empty())
|
|
return false;
|
|
|
|
return ContainsInRegularHyperslab(image_number,
|
|
virtual_start[0],
|
|
virtual_stride[0],
|
|
virtual_count[0],
|
|
virtual_block[0]);
|
|
}
|
|
|
|
hsize_t HDF5VirtualDatasetMapping::SourceImage(hsize_t image_number) const {
|
|
if (!ContainsVirtualImage(image_number))
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Image is outside of VDS mapping");
|
|
|
|
const hsize_t source_index = IndexInRegularHyperslab(image_number,
|
|
virtual_start[0],
|
|
virtual_stride[0],
|
|
virtual_count[0],
|
|
virtual_block[0]);
|
|
|
|
if (source_start.empty() || source_stride.empty() || source_count.empty() || source_block.empty())
|
|
return source_index;
|
|
|
|
return ValueFromRegularHyperslabIndex(source_index,
|
|
source_start[0],
|
|
source_stride[0],
|
|
source_count[0],
|
|
source_block[0]);
|
|
}
|
|
|
|
|
|
std::vector<HDF5VirtualDatasetMapping> HDF5Dcpl::GetVirtualMappings() const {
|
|
size_t mapping_count = 0;
|
|
|
|
if (H5Pget_virtual_count(id, &mapping_count) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get number of VDS mappings");
|
|
if (mapping_count < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get number of VDS mappings");
|
|
|
|
std::vector<HDF5VirtualDatasetMapping> ret;
|
|
ret.reserve(static_cast<size_t>(mapping_count));
|
|
|
|
for (size_t i = 0; i < static_cast<size_t>(mapping_count); i++) {
|
|
HDF5VirtualDatasetMapping mapping;
|
|
|
|
const ssize_t filename_size = H5Pget_virtual_filename(id, i, nullptr, 0);
|
|
if (filename_size < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS source filename size");
|
|
|
|
std::vector<char> filename(filename_size + 1, '\0');
|
|
if (H5Pget_virtual_filename(id, i, filename.data(), filename.size()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS source filename");
|
|
mapping.filename = filename.data();
|
|
|
|
const ssize_t dataset_size = H5Pget_virtual_dsetname(id, i, nullptr, 0);
|
|
if (dataset_size < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS source dataset size");
|
|
|
|
std::vector<char> dataset(dataset_size + 1, '\0');
|
|
if (H5Pget_virtual_dsetname(id, i, dataset.data(), dataset.size()) < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS source dataset");
|
|
mapping.dataset = dataset.data();
|
|
|
|
const hid_t virtual_space = H5Pget_virtual_vspace(id, i);
|
|
if (virtual_space < 0)
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS virtual dataspace");
|
|
|
|
const hid_t source_space = H5Pget_virtual_srcspace(id, i);
|
|
if (source_space < 0) {
|
|
H5Sclose(virtual_space);
|
|
throw JFJochException(JFJochExceptionCategory::HDF5,
|
|
"Cannot get VDS source dataspace");
|
|
}
|
|
|
|
try {
|
|
GetRegularSelection(virtual_space,
|
|
mapping.virtual_start,
|
|
mapping.virtual_stride,
|
|
mapping.virtual_count,
|
|
mapping.virtual_block);
|
|
|
|
GetRegularSelection(source_space,
|
|
mapping.source_start,
|
|
mapping.source_stride,
|
|
mapping.source_count,
|
|
mapping.source_block);
|
|
} catch (...) {
|
|
H5Sclose(source_space);
|
|
H5Sclose(virtual_space);
|
|
throw;
|
|
}
|
|
|
|
H5Sclose(source_space);
|
|
H5Sclose(virtual_space);
|
|
|
|
ret.emplace_back(std::move(mapping));
|
|
}
|
|
|
|
return ret;
|
|
} |