Files
Jungfraujoch/frame_serialize/CBORStream2Deserializer.cpp
T
leonarski_fandClaude Opus 5 61a7c91b90 Ice: detect it on two channels, and only handle it when it is there
The per-image ice score was read off the PLAIN azimuthal profile. That profile is a
per-ring mean, so a few strong Bragg reflections landing in a ring's q bin lift it
exactly as ice would. Measured over 37 rotation crystals, that did not merely add
noise - it INVERTED the metric: the two highest-scoring crystals had no ice at all
(4.23 and 4.06), while a clean control read 1.57. A decoy null - the identical
statistic evaluated at q positions where hexagonal ice cannot be - reaches 1.51 at its
99th percentile and 2.70 at its maximum, so that metric cannot support any absolute
threshold whatsoever.

The adaptive spot finder already computes the right input for its own threshold: a
sigma-clipped per-resolution-ring background, in the same bins. A powder ring is
azimuthally smooth and survives the clip; Bragg peaks do not. On the clipped profile
the clean population tightens to 1.00-1.22 and the crystals with confirmed ice sit at
2.08-2.37, against a decoy null that never exceeds 1.29.

That channel is blind to one thing: ice in large crystallites diffracts as DISCRETE
spots and leaves the radial profile flat. So a second channel counts found spots on the
rings against the same q width of ice-free flanks beside them. The two barely overlap -
the smooth-ice crystals read 2.1-2.4 / ~1.0 and the textured ones ~1.1 / 3.8-17.6,
while a clean crystal reads 1.04 on both.

Both are then used as a GATE (--ice-min-score 1.5, --ice-min-spot-ratio 2.0, both
calibrated on the battery, 0 disables): the eleven fixed hexagonal bands cover 16-26 %
of the unique reflections at typical resolutions whether or not the crystal has ice, so
flagging, the exclusion from the scale fit and the merge-time CC1/2 ring mask are now
all skipped when neither channel sees any. The gate is applied in the full pipeline and
in --scale, which reads the stored per-image values back out of the _process.h5.

Also fixes the merge-time mask's control: the shoulder now excludes reflections that
are themselves on an ice ring. The rings are not evenly spaced - 1.947/1.916/1.882 A
sit 0.05-0.06 apart in q - so for those three the [w,3w) shoulder landed squarely on
the neighbours and the test compared ice against ice. Measured, that is the only thing
this changes: it removes firings on those three rings and leaves every other firing's
CC pair identical to three decimals.

And the online ice half-width, which was 0.02 in the API against 0.03 offline, so the
same data got a narrower band online than the measured ~0.06 ring FWHM justifies.

Battery (37 rotation crystals, against the previous behaviour): space groups 34/37 in
both and NO crystal's space group changes; 6 crystals gain unique reflections, 1 loses.
Best of them gains 7082 unique reflections with R_meas 16.0 -> 14.3, CC1/2 95.9 -> 97.3
and ISa 13.7 -> 19.0; another goes R_meas 54.9 -> 42.9, CC1/2 84.0 -> 90.4, ISa
3.9 -> 5.5; a third reaches CC1/2 99.4 from 95.7 at an unchanged reflection count. The
one crystal that loses reflections improves on both R_meas and CC1/2.

Not done here: the ScanResult/API/plot-type/frontend/viewer layers for the new
spot_count_ice_control (they need the OpenAPI regeneration). Message, CBOR, HDF5
write/read and the receiver plots are.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 16:17:23 +02:00

1547 lines
66 KiB
C++

// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "CBORStream2Deserializer.h"
#include "CborErr.h"
#include "CborUtil.h"
#include "../compression/JFJochDecompress.h"
#include <nlohmann/json.hpp>
namespace {
std::string GetCBORString(CborValue &value) {
if (!cbor_value_is_text_string(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "String expected");
std::vector<char> s(16384);
size_t len = s.size() - 1;
cborErr(cbor_value_copy_text_string(&value, s.data(), &len, nullptr));
s[len] = 0;
cborErr(cbor_value_advance(&value));
return {s.data()};
}
int64_t GetCBORInt(CborValue &value) {
if (!cbor_value_is_integer(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Integer expected");
int64_t tmp;
cborErr(cbor_value_get_int64(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
}
uint64_t GetCBORUInt(CborValue &value) {
if (!cbor_value_is_unsigned_integer(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Unsigned integer expected");
uint64_t tmp;
cborErr(cbor_value_get_uint64(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
}
float GetCBORFloat(CborValue &value) {
if (cbor_value_is_double(&value)) {
double tmp;
cborErr(cbor_value_get_double(&value, &tmp));
cborErr(cbor_value_advance(&value));
return static_cast<float>(tmp);
} else if (cbor_value_is_float(&value)) {
float tmp;
cborErr(cbor_value_get_float(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
} else if (cbor_value_is_half_float(&value)) {
float tmp;
cborErr(cbor_value_get_half_float_as_float(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
}
throw JFJochException(JFJochExceptionCategory::CBORError, "Float expected");
}
bool GetCBORBool(CborValue &value) {
if (!cbor_value_is_boolean(&value)) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Bool expected " + std::to_string(cbor_value_get_type(&value)));
}
bool tmp;
cborErr(cbor_value_get_boolean(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
}
CborTag GetCBORTag(CborValue &value) {
if (!cbor_value_is_tag(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Tag expected");
CborTag tmp;
cborErr(cbor_value_get_tag(&value, &tmp));
cborErr(cbor_value_advance(&value));
return tmp;
}
std::string GetCBORDateTime(CborValue &value) {
auto tag = GetCBORTag(value);
if (tag == CborDateTimeStringTag)
return GetCBORString(value);
else if (tag == CborUnixTime_tTag) {
time_t t = GetCBORUInt(value);
struct tm tm_buf{};
#ifdef _WIN32
gmtime_s(&tm_buf, &t);
#else
gmtime_r(&t, &tm_buf);
#endif
char buf1[255];
strftime(buf1, sizeof(buf1), "%FT%T", &tm_buf);
return std::string(buf1) + "Z";
} else
throw JFJochException(JFJochExceptionCategory::CBORError, "Time/date tag error");
}
uint64_t GetCBORArrayLen(CborValue &value) {
if (!cbor_value_is_array(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Array expected");
size_t array_len;
cborErr(cbor_value_get_array_length(&value, &array_len));
return array_len;
}
uint64_t GetCBORMapLen(CborValue &value) {
if (!cbor_value_is_map(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Map expected");
size_t map_len;
cborErr(cbor_value_get_map_length(&value, &map_len));
return map_len;
}
Coord GetCoord(CborValue &value) {
CborValue array_value;
Coord c;
if (GetCBORArrayLen(value) != 3)
throw JFJochException(JFJochExceptionCategory::CBORError, "Vector of 3 elements expected");
cborErr(cbor_value_enter_container(&value, &array_value));
c.x = GetCBORFloat(array_value);
c.y = GetCBORFloat(array_value);
c.z = GetCBORFloat(array_value);
cborErr(cbor_value_leave_container(&value, &array_value));
return c;
}
std::pair<uint64_t, uint64_t> GetRational(CborValue &value) {
std::pair<uint64_t, uint64_t> ret;
if (GetCBORArrayLen(value) != 2)
throw JFJochException(JFJochExceptionCategory::CBORError,
"Rational number expects array of 2 elements");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
ret.first = GetCBORUInt(array_value);
ret.second = GetCBORUInt(array_value);
cborErr(cbor_value_leave_container(&value, &array_value));
return ret;
}
std::map<std::string, float> ProcessThresholdEnergy(CborValue &value) {
if (!cbor_value_is_map(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Map expected");
std::map<std::string, float> ret;
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
auto val = GetCBORFloat(map_value);
ret[key] = val;
}
cborErr(cbor_value_leave_container(&value, &map_value));
return ret;
}
UnitCell ProcessUnitCellElement(CborValue &value) {
UnitCell unit_cell{};
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "a")
unit_cell.a = GetCBORFloat(map_value);
else if (key == "b")
unit_cell.b = GetCBORFloat(map_value);
else if (key == "c")
unit_cell.c = GetCBORFloat(map_value);
else if (key == "alpha")
unit_cell.alpha = GetCBORFloat(map_value);
else if (key == "beta")
unit_cell.beta = GetCBORFloat(map_value);
else if (key == "gamma")
unit_cell.gamma = GetCBORFloat(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
return unit_cell;
}
std::vector<uint64_t> GetCBORDimensionArray(CborValue &value) {
std::vector<uint64_t> ret;
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
while (!cbor_value_at_end(&array_value))
ret.push_back(GetCBORUInt(array_value));
cborErr(cbor_value_leave_container(&value, &array_value));
return ret;
}
std::pair<const uint8_t *, size_t> GetCBORByteString(CborValue &value) {
if (!cbor_value_is_byte_string(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Bool expected");
size_t len;
cborErr(cbor_value_get_string_length(&value, &len));
// Code below from: https://github.com/dectris/documentation/blob/main/stream_v2/examples/stream2.c
const uint8_t *ptr = cbor_value_get_next_byte(&value);
uint8_t val = *ptr;
if (val < 0x40)
throw JFJochException(JFJochExceptionCategory::CBORError, "Something wrong with byte string tag");
val -= 0x40;
if (val < 24)
ptr += 1;
else if (val == 24)
ptr += 1 + 1;
else if (val == 25)
ptr += 1 + 2;
else if (val == 26)
ptr += 1 + 4;
else if (val == 27)
ptr += 1 + 8;
else
throw JFJochException(JFJochExceptionCategory::CBORError, "Something wrong with byte string tag");
cborErr(cbor_value_advance(&value));
return {ptr, len};
}
void GetCBORUInt8Array(CborValue &value, std::vector<uint8_t> &v) {
if (GetCBORTag(value) != TagUnsignedInt8Bit)
throw JFJochException(JFJochExceptionCategory::CBORError, "Incorrect array type tag");
auto [ptr, len] = GetCBORByteString(value);
v.resize(len);
memcpy(v.data(), ptr, len);
}
void GetCBORInt32Array(CborValue &value, std::vector<int32_t> &v) {
if (GetCBORTag(value) != TagSignedInt32BitLE)
throw JFJochException(JFJochExceptionCategory::CBORError, "Incorrect array type tag");
auto [ptr, len] = GetCBORByteString(value);
if (len % sizeof(int32_t))
throw JFJochException(JFJochExceptionCategory::CBORError, "Size mismatch");
v.resize(len / sizeof(int32_t));
memcpy(v.data(), ptr, len);
}
void GetCBORInt64Array(CborValue &value, std::vector<int64_t> &v) {
if (GetCBORTag(value) != TagSignedInt64BitLE)
throw JFJochException(JFJochExceptionCategory::CBORError, "Incorrect array type tag");
auto [ptr, len] = GetCBORByteString(value);
if (len % sizeof(int64_t))
throw JFJochException(JFJochExceptionCategory::CBORError, "Size mismatch");
v.resize(len / sizeof(int64_t));
memcpy(v.data(), ptr, len);
}
void GetCBORFloatArray(CborValue &value, std::vector<float> &v) {
if (GetCBORTag(value) != TagFloatLE)
throw JFJochException(JFJochExceptionCategory::CBORError, "Incorrect array type tag");
auto [ptr, len] = GetCBORByteString(value);
if (len % sizeof(float))
throw JFJochException(JFJochExceptionCategory::CBORError, "Size mismatch");
v.resize(len / sizeof(float));
memcpy(v.data(), ptr, len);
}
void GetCBORStringArray(CborValue &value, std::vector<std::string> &v) {
if (!cbor_value_is_array(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Array expected");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
while (!cbor_value_at_end(&array_value)) {
v.emplace_back(GetCBORString(array_value));
}
cborErr(cbor_value_leave_container(&value, &array_value));
}
void GetCBORUInt64Array(CborValue &value, std::vector<uint64_t> &v) {
if (GetCBORTag(value) != TagUnsignedInt64BitLE)
throw JFJochException(JFJochExceptionCategory::CBORError, "Incorrect array type tag");
auto [ptr, len] = GetCBORByteString(value);
if (len % sizeof(uint64_t))
throw JFJochException(JFJochExceptionCategory::CBORError, "Size mismatch");
v.resize(len / sizeof(uint64_t));
memcpy(v.data(), ptr, len);
}
CompressedImageMode GetImageMode(CborTag input) {
switch (input) {
case TagHalfLE:
return CompressedImageMode::Float16;
case TagFloatLE:
return CompressedImageMode::Float32;
case TagDoubleLE:
return CompressedImageMode::Float64;
case TagSignedInt8Bit:
return CompressedImageMode::Int8;
case TagUnsignedInt8Bit:
return CompressedImageMode::Uint8;
case TagSignedInt16BitLE:
return CompressedImageMode::Int16;
case TagUnsignedInt16BitLE:
return CompressedImageMode::Uint16;
case TagSignedInt32BitLE:
return CompressedImageMode::Int32;
case TagUnsignedInt32BitLE:
return CompressedImageMode::Uint32;
default:
throw JFJochException(JFJochExceptionCategory::CBORError,
"Only int/uint arrays of 1, 2, 4 bytes per element allowed");
}
}
CompressedImage GetCBORTypedArray(CborValue &value, const std::vector<uint64_t> &dim) {
CompressedImageMode mode = GetImageMode(GetCBORTag(value));
CompressionAlgorithm algorithm = CompressionAlgorithm::NO_COMPRESSION;
size_t xpixel, ypixel;
size_t size;
const uint8_t *ptr = nullptr;
if (cbor_value_is_tag(&value)) {
if (GetCBORTag(value) != TagDECTRISCompression)
throw JFJochException(JFJochExceptionCategory::CBORError, "Unsupported tag");
if (GetCBORArrayLen(value) != 3)
throw JFJochException(JFJochExceptionCategory::CBORError, "Expected 3 element array");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
auto algorithm_text = GetCBORString(array_value);
if (algorithm_text == "bslz4")
algorithm = CompressionAlgorithm::BSHUF_LZ4;
else if (algorithm_text == "bszstd")
algorithm = CompressionAlgorithm::BSHUF_ZSTD;
else
throw JFJochException(JFJochExceptionCategory::CBORError, "Unsupported compression algorithm");
auto pixel_depth_bytes = GetCBORUInt(array_value);
auto ret = GetCBORByteString(array_value);
ptr = ret.first;
size = ret.second;
cborErr(cbor_value_leave_container(&value, &array_value));
} else if (cbor_value_is_byte_string(&value)) {
algorithm = CompressionAlgorithm::NO_COMPRESSION;
auto ret = GetCBORByteString(value);
ptr = ret.first;
size = ret.second;
} else
throw JFJochException(JFJochExceptionCategory::CBORError, "Byte string or compressed array expected");
if (dim.size() == 2) {
xpixel = dim[1];
ypixel = dim[0];
} else if (dim.size() == 3 && dim[0] == 3 && mode == CompressedImageMode::Uint8) {
mode = CompressedImageMode::RGB;
xpixel = dim[2];
ypixel = dim[1];
} else
throw JFJochException(JFJochExceptionCategory::CBORError, "Image dimension not supported");
return CompressedImage(ptr, size, xpixel, ypixel, mode, algorithm);
}
CompressedImage GetCBORMultidimTypedArray(CborValue &value) {
if (GetCBORTag(value) != TagMultiDimArray)
throw JFJochException(JFJochExceptionCategory::CBORError, "Multidim array expected");
if (GetCBORArrayLen(value) != 2)
throw JFJochException(JFJochExceptionCategory::CBORError, "2 element array expected");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
auto dim = GetCBORDimensionArray(array_value);
CompressedImage image = GetCBORTypedArray(array_value, dim);
cborErr(cbor_value_leave_container(&value, &array_value));
return image;
}
void CheckMagicNumber(CborValue &v) {
if (GetCBORUInt(v) != user_data_magic_number)
throw JFJochException(JFJochExceptionCategory::CBORError,
"Inconsistency between Jungfraujoch server and writer");
}
CompressedImage ProcessImageData(CborValue &value) {
if (!cbor_value_is_map(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Map expected");
if (GetCBORMapLen(value) != 1)
throw JFJochException(JFJochExceptionCategory::CBORError,
"Single channel images only supported at the moment");
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
std::string channel = GetCBORString(map_value);
CompressedImage image = GetCBORMultidimTypedArray(map_value);
image.Channel(channel);
cborErr(cbor_value_leave_container(&value, &map_value));
return image;
}
// Data message
SpotToSave GetSpot(CborValue &value) {
SpotToSave s{};
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "x")
s.x = GetCBORFloat(map_value);
else if (key == "y")
s.y = GetCBORFloat(map_value);
else if (key == "I")
s.intensity = GetCBORFloat(map_value);
else if (key == "maxc")
s.maxc = GetCBORInt(map_value);
else if (key == "ice_ring")
s.ice_ring = GetCBORBool(map_value);
else if (key == "indexed")
s.indexed = GetCBORBool(map_value);
else if (key == "latt")
s.lattice = GetCBORInt(map_value);
else if (key == "h")
s.h = GetCBORInt(map_value);
else if (key == "k")
s.k = GetCBORInt(map_value);
else if (key == "l")
s.l = GetCBORInt(map_value);
else if (key == "dist_ewald")
s.dist_ewald_sphere = GetCBORFloat(map_value);
else if (key == "image")
s.image = GetCBORInt(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
return s;
}
void GetCBORSpots(DataMessage &message, CborValue &value) {
size_t array_len = GetCBORArrayLen(value);
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
for (int i = 0; i < array_len; i++)
message.spots.push_back(GetSpot(array_value));
cborErr(cbor_value_leave_container(&value, &array_value));
}
Reflection GetReflection(CborValue &value) {
Reflection r{};
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "h")
r.h = GetCBORInt(map_value);
else if (key == "k")
r.k = GetCBORInt(map_value);
else if (key == "l")
r.l = GetCBORInt(map_value);
else if (key == "phi")
r.delta_phi_deg = GetCBORFloat(map_value);
else if (key == "x")
r.predicted_x = GetCBORFloat(map_value);
else if (key == "y")
r.predicted_y = GetCBORFloat(map_value);
else if (key == "obs_x")
r.observed_x = GetCBORFloat(map_value);
else if (key == "obs_y")
r.observed_y = GetCBORFloat(map_value);
else if (key == "d")
r.d = GetCBORFloat(map_value);
else if (key == "I")
r.I = GetCBORFloat(map_value);
else if (key == "bkg")
r.bkg = GetCBORFloat(map_value);
else if (key == "sigma")
r.sigma = GetCBORFloat(map_value);
else if (key == "image")
r.image_number = GetCBORFloat(map_value);
else if (key == "rlp")
r.rlp = GetCBORFloat(map_value);
else if (key == "rp")
r.dist_ewald = GetCBORFloat(map_value);
else if (key == "partiality")
r.partiality = GetCBORFloat(map_value);
else if (key == "zeta")
r.zeta = GetCBORFloat(map_value);
else if (key == "image_scale_corr")
r.image_scale_corr = GetCBORFloat(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
return r;
}
void GetCBORReflections(DataMessage &message, CborValue &value) {
size_t array_len = GetCBORArrayLen(value);
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
for (int i = 0; i < array_len; i++)
message.reflections.push_back(GetReflection(array_value));
cborErr(cbor_value_leave_container(&value, &array_value));
}
LatticeMessage GetCBORLatticeMessage(CborValue &value) {
LatticeMessage lm{};
lm.centering = 'P';
lm.niggli_class = 0;
lm.crystal_system = gemmi::CrystalSystem::Triclinic;
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "centering") {
auto s = GetCBORString(map_value);
char c = s.empty() ? 'P' : s[0];
switch (c) {
case 'P':
case 'A':
case 'B':
case 'C':
case 'I':
case 'F':
case 'R':
lm.centering = c;
break;
default:
break;
}
} else if (key == "niggli_class") {
lm.niggli_class = GetCBORInt(map_value);
} else if (key == "system") {
auto sys = GetCBORString(map_value);
// map known strings to enum
if (sys == "triclinic") lm.crystal_system = gemmi::CrystalSystem::Triclinic;
else if (sys == "monoclinic") lm.crystal_system = gemmi::CrystalSystem::Monoclinic;
else if (sys == "orthorhombic") lm.crystal_system = gemmi::CrystalSystem::Orthorhombic;
else if (sys == "tetragonal") lm.crystal_system = gemmi::CrystalSystem::Tetragonal;
else if (sys == "trigonal") lm.crystal_system = gemmi::CrystalSystem::Trigonal;
else if (sys == "hexagonal") lm.crystal_system = gemmi::CrystalSystem::Hexagonal;
else if (sys == "cubic") lm.crystal_system = gemmi::CrystalSystem::Cubic;
else {
// unknown string, keep default
}
} else {
cbor_value_advance(&map_value);
}
}
cborErr(cbor_value_leave_container(&value, &map_value));
return lm;
}
XrayFluorescenceSpectrum GetCBORFluorescenceSpectrum(CborValue &value) {
std::vector<float> data, energy;
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "energy")
GetCBORFloatArray(map_value, energy);
else if (key == "data")
GetCBORFloatArray(map_value, data);
}
cborErr(cbor_value_leave_container(&value, &map_value));
if (data.size() == energy.size() && data.size() > 0)
return {energy, data};
else
return {};
}
CBORImageType DecodeType(CborValue &value) {
if (cbor_value_at_end(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Message empty");
if (GetCBORString(value) != "type")
throw JFJochException(JFJochExceptionCategory::CBORError, "First CBOR entry must by type");
if (cbor_value_at_end(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Message empty");
auto type_str = GetCBORString(value);
if (type_str == "start")
return CBORImageType::START;
if (type_str == "end")
return CBORImageType::END;
if (type_str == "image")
return CBORImageType::IMAGE;
if (type_str == "calibration")
return CBORImageType::CALIBRATION;
if (type_str == "metadata")
return CBORImageType::METADATA;
throw JFJochException(JFJochExceptionCategory::CBORError, "Unknown message type");
}
// Data message
ROIMessage ProcessROIElement(CborValue &value) {
ROIMessage msg{};
// value
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "sum")
msg.sum = GetCBORInt(map_value);
else if (key == "sum_square")
msg.sum_square = GetCBORUInt(map_value);
else if (key == "max_count")
msg.max_count = GetCBORInt(map_value);
else if (key == "pixels")
msg.pixels = GetCBORUInt(map_value);
else if (key == "x_weighted_sum")
msg.x_weighted = GetCBORInt(map_value);
else if (key == "y_weighted_sum")
msg.y_weighted = GetCBORInt(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
return msg;
}
void ProcessROIElementMap(DataMessage &message, CborValue &value) {
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
// key
const std::string roi_name = GetCBORString(map_value);
// value
message.roi[roi_name] = ProcessROIElement(map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
}
bool ProcessDataMessageElement(DataMessage &message, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
const auto key = GetCBORString(value);
try {
if (key == "image_id")
message.number = GetCBORInt(value);
else if (key == "original_image_id")
message.original_number = GetCBORInt(value);
else if (key == "data")
message.image = ProcessImageData(value);
else if (key == "series_unique_id")
message.run_name = GetCBORString(value);
else if (key == "series_id")
message.run_number = GetCBORUInt(value);
else if (key == "real_time") {
auto r = GetRational(value);
message.exptime = r.first;
message.exptime_base = r.second;
} else if (key == "start_time") {
auto r = GetRational(value);
message.timestamp = r.first;
message.timestamp_base = r.second;
} else if (key == "user_data") {
const std::string s = GetCBORString(value);
try {
message.user_data = nlohmann::json::parse(s);
} catch (...) {
message.user_data = s;
}
} else if (key == "spots")
GetCBORSpots(message, value);
else if (key == "reflections")
GetCBORReflections(message, value);
else if (key == "spot_count")
message.spot_count = GetCBORUInt(value);
else if (key == "spot_count_low_res")
message.spot_count_low_res = GetCBORUInt(value);
else if (key == "spot_count_indexed")
message.spot_count_indexed = GetCBORUInt(value);
else if (key == "spot_count_ice_rings")
message.spot_count_ice_rings = GetCBORUInt(value);
else if (key == "spot_count_ice_control")
message.spot_count_ice_control = GetCBORFloat(value);
else if (key == "az_int_profile")
GetCBORFloatArray(value, message.az_int_profile);
else if (key == "az_int_profile_std")
GetCBORFloatArray(value, message.az_int_profile_std);
else if (key == "az_int_profile_count")
GetCBORUInt64Array(value, message.az_int_profile_count);
else if (key == "indexing_result")
message.indexing_result = GetCBORBool(value);
else if (key == "indexing_lattice_count")
message.indexing_lattice_count = GetCBORInt(value);
else if (key == "indexing_lattice") {
std::vector<float> tmp;
GetCBORFloatArray(value, tmp);
message.indexing_lattice = CrystalLattice(tmp);
} else if (key == "indexing_extra_lattices") {
size_t array_len = GetCBORArrayLen(value);
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
for (size_t i = 0; i < array_len; i++) {
std::vector<float> tmp;
GetCBORFloatArray(array_value, tmp);
if (tmp.size() == 9)
message.indexing_extra_lattices.emplace_back(tmp);
}
cborErr(cbor_value_leave_container(&value, &array_value));
} else if (key == "indexing_time")
message.indexing_time_s = GetCBORFloat(value);
else if (key == "spot_finding_time")
message.spot_finding_time_s = GetCBORFloat(value);
else if (key == "integration_time")
message.integration_time_s = GetCBORFloat(value);
else if (key == "bragg_prediction_time")
message.bragg_prediction_time_s = GetCBORFloat(value);
else if (key == "refinement_time")
message.refinement_time_s = GetCBORFloat(value);
else if (key == "index_analysis_time")
message.index_analysis_time_s = GetCBORFloat(value);
else if (key == "processing_time")
message.processing_time_s = GetCBORFloat(value);
else if (key == "preprocessing_time")
message.preprocessing_time_s = GetCBORFloat(value);
else if (key == "compression_time")
message.compression_time_s = GetCBORFloat(value);
else if (key == "image_scale_time")
message.image_scale_time_s = GetCBORFloat(value);
else if (key == "azint_time")
message.azint_time_s = GetCBORFloat(value);
else if (key == "profile_radius")
message.profile_radius = GetCBORFloat(value);
else if (key == "mosaicity")
message.mosaicity_deg = GetCBORFloat(value);
else if (key == "b_factor")
message.b_factor = GetCBORFloat(value);
else if (key == "integrated_reflections")
message.integrated_reflections = GetCBORInt(value);
else if (key == "indexing_unit_cell")
message.indexing_unit_cell = ProcessUnitCellElement(value);
else if (key == "lattice_type")
message.lattice_type = GetCBORLatticeMessage(value);
else if (key == "jf_info")
message.jf_info = GetCBORUInt(value) & UINT32_MAX;
else if (key == "receiver_aq_dev_delay")
message.receiver_aq_dev_delay = GetCBORInt(value);
else if (key == "receiver_free_send_buf")
message.receiver_buf_available = GetCBORInt(value);
else if (key == "receiver_buf_in_sending")
message.receiver_buf_in_sending = GetCBORInt(value);
else if (key == "receiver_buf_in_preparation")
message.receiver_buf_in_preparation = GetCBORInt(value);
else if (key == "storage_cell")
message.storage_cell = GetCBORUInt(value) & UINT32_MAX;
else if (key == "xfel_pulse_id")
message.xfel_pulse_id = GetCBORUInt(value);
else if (key == "xfel_event_code")
message.xfel_event_code = GetCBORUInt(value);
else if (key == "saturated_pixel_count")
message.saturated_pixel_count = GetCBORUInt(value);
else if (key == "error_pixel_count")
message.error_pixel_count = GetCBORUInt(value);
else if (key == "pixel_sum")
message.pixel_sum = GetCBORInt(value);
else if (key == "strong_pixel_count")
message.strong_pixel_count = GetCBORUInt(value);
else if (key == "min_viable_pixel_value")
message.min_viable_pixel_value = GetCBORInt(value);
else if (key == "max_viable_pixel_value")
message.max_viable_pixel_value = GetCBORInt(value);
else if (key == "resolution_estimate")
message.resolution_estimate = GetCBORFloat(value);
else if (key == "data_collection_efficiency")
message.image_collection_efficiency = GetCBORFloat(value);
else if (key == "packets_expected")
message.packets_expected = GetCBORUInt(value);
else if (key == "packets_received")
message.packets_received = GetCBORUInt(value);
else if (key == "bkg_estimate")
message.bkg_estimate = GetCBORFloat(value);
else if (key == "ice_ring_score")
message.ice_ring_score = GetCBORFloat(value);
else if (key == "adu_histogram")
GetCBORUInt64Array(value, message.adu_histogram);
else if (key == "beam_corr_x")
message.beam_corr_x = GetCBORFloat(value);
else if (key == "beam_corr_y")
message.beam_corr_y = GetCBORFloat(value);
else if (key == "image_scale_factor")
message.image_scale_factor = GetCBORFloat(value);
else if (key == "image_scale_cc")
message.image_scale_cc = GetCBORFloat(value);
else if (key == "image_scale_mosaicity")
message.image_scale_mosaicity = GetCBORFloat(value);
else if (key == "roi_integrals")
ProcessROIElementMap(message, value);
else {
if (cbor_value_is_tag(&value))
cbor_value_advance(&value);
cbor_value_advance(&value);
}
return true;
} catch (const JFJochException &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Error processing image message, key " + key + ":" + e.what());
}
}
bool ProcessMetadataImagesElement(MetadataMessage &metadata, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
if (!cbor_value_is_map(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "images@metadata must be map");
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
DataMessage message{};
while (ProcessDataMessageElement(message, map_value)) {}
metadata.images.push_back(std::move(message));
cborErr(cbor_value_leave_container(&value, &map_value));
return true;
}
bool ProcessMetadataMessageElement(MetadataMessage &metadata, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
const auto key = GetCBORString(value);
try {
if (key == "series_unique_id")
metadata.run_name = GetCBORString(value);
else if (key == "series_id")
metadata.run_number = GetCBORUInt(value);
else if (key == "images") {
if (!cbor_value_is_array(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "images@metadata must be array");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
while (ProcessMetadataImagesElement(metadata, array_value)) {}
cborErr(cbor_value_leave_container(&value, &array_value));
} else {
if (cbor_value_is_tag(&value))
cbor_value_advance(&value);
cbor_value_advance(&value);
}
return true;
} catch (const JFJochException &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Error processing image message, key " + key + ":" + e.what());
}
}
// Start message
void ProcessAxis(CborValue &value, float v[3]) {
if (GetCBORArrayLen(value) != 3)
throw JFJochException(JFJochExceptionCategory::CBORError, "Array with 3 floats expected");
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
for (int i = 0; i < 3; i++)
v[i] = GetCBORFloat(array_value);
cborErr(cbor_value_leave_container(&value, &array_value));
}
std::optional<GridScanSettings> ProcessGridScan(CborValue &value) {
int64_t n_fast = 0;
int64_t n_slow = 0;
float step_x_um = 0;
float step_y_um = 0;
bool snake = false;
bool vertical = false;
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "n_fast")
n_fast = GetCBORInt(map_value);
else if (key == "n_slow")
n_slow = GetCBORInt(map_value);
else if (key == "step_x_axis")
step_x_um = GetCBORFloat(map_value) * 1e6f;
else if (key == "step_y_axis")
step_y_um = GetCBORFloat(map_value) * 1e6f;
else if (key == "snake_scan")
snake = GetCBORBool(map_value);
else if (key == "vertical_scan")
vertical = GetCBORBool(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
GridScanSettings grid(n_fast, step_x_um, step_y_um,
snake, vertical);
grid.ImageNum(n_fast * n_slow);
return grid;
}
std::optional<GoniometerAxis> ProcessGoniometer(std::string &name, CborValue &value) {
float start = 0;
float increment = 1;
Coord axis = {1,0,0};
std::optional<Coord> helical;
std::optional<float> screening_wedge;
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
if (key == "increment")
increment = GetCBORFloat(map_value);
else if (key == "start")
start = GetCBORFloat(map_value);
else if (key == "axis")
axis = GetCoord(map_value);
else if (key == "helical_step")
helical = GetCoord(map_value);
else if (key == "screening_wedge")
screening_wedge = GetCBORFloat(map_value);
else
cbor_value_advance(&map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
if (increment == 0)
return {};
else {
auto g = GoniometerAxis(name, start, increment, axis, helical);
g.ScreeningWedge(screening_wedge);
return g;
}
}
void ProcessROIConfig(StartMessage &message, const nlohmann::json &j) {
if (!j.is_array())
return;
for (const auto &jr: j) {
ROIConfig cfg;
try {
cfg.name = jr["name"];
if (jr["type"] == "box") {
cfg.type = ROIConfig::ROIType::Box;
cfg.box.xmin = jr["xmin"];
cfg.box.xmax = jr["xmax"];
cfg.box.ymin = jr["ymin"];
cfg.box.ymax = jr["ymax"];
message.rois.emplace_back(std::move(cfg));
} else if (jr["type"] == "circle") {
cfg.type = ROIConfig::ROIType::Circle;
cfg.circle.r = jr["r"];
cfg.circle.x = jr["x"];
cfg.circle.y = jr["y"];
message.rois.emplace_back(std::move(cfg));
} else if (jr["type"] == "azim") {
cfg.type = ROIConfig::ROIType::Azim;
cfg.azim.qmin = jr["qmin"];
cfg.azim.qmax = jr["qmax"];
// Absent phi (older streams) means a full ring.
cfg.azim.phi_min = jr.value("phi_min", 0.0f);
cfg.azim.phi_max = jr.value("phi_max", 0.0f);
message.rois.emplace_back(std::move(cfg));
} else
continue;
} catch (...) {
// For now ignore things that are incorrect
}
}
}
void ProcessGoniometerMap(StartMessage &message, CborValue &value) {
CborValue map_value;
if (GetCBORMapLen(value) > 1)
throw JFJochException(JFJochExceptionCategory::CBORError, "Max one rotation angle allowed");
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
message.goniometer = ProcessGoniometer(key, map_value);
}
cborErr(cbor_value_leave_container(&value, &map_value));
}
void ProcessPixelMaskElement(StartMessage &message, CborValue &value) {
// For calibration and mask - image might be accessed later on - so it is saved in internal storage of the object
// It allows the original start message to be lost
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
CompressedImage image = GetCBORMultidimTypedArray(map_value);
image.Channel(key);
if (image.GetMode() != CompressedImageMode::Uint32)
continue;
std::vector<uint32_t> mask_uncompressed;
JFJochDecompress(mask_uncompressed, image.GetCompressionAlgorithm(),
image.GetCompressed(), image.GetCompressedSize(),
image.GetWidth() * image.GetHeight());
message.pixel_mask[key] = mask_uncompressed;
}
cborErr(cbor_value_leave_container(&value, &map_value));
}
void ProcessAzintMapElement(StartMessage &message, CborValue &value) {
CompressedImage image = GetCBORMultidimTypedArray(value);
if (image.GetMode() == CompressedImageMode::Uint16)
JFJochDecompress(message.az_int_map, image.GetCompressionAlgorithm(),
image.GetCompressed(), image.GetCompressedSize(),
image.GetWidth() * image.GetHeight());
}
void ProcessROIMapElement(StartMessage &message, CborValue &value) {
CompressedImage image = GetCBORMultidimTypedArray(value);
if (image.GetMode() == CompressedImageMode::Uint16)
JFJochDecompress(message.roi_map, image.GetCompressionAlgorithm(),
image.GetCompressed(), image.GetCompressedSize(),
image.GetWidth() * image.GetHeight());
}
std::optional<FileWriterFormat> ProcessHDF5Format(int input) {
auto tmp = static_cast<FileWriterFormat>(input);
switch (tmp) {
case FileWriterFormat::DataOnly:
case FileWriterFormat::NXmxLegacy:
case FileWriterFormat::NXmxIntegrated:
case FileWriterFormat::NXmxVDS:
case FileWriterFormat::NoFile:
return tmp;
default:
// don't throw exception, just do default
return {};
}
}
void ProcessStartUserData(StartMessage &message, CborValue &value) {
try {
const std::string s = GetCBORString(value);
auto j = nlohmann::json::parse(s);
if (!j.is_object())
return;
if (j.contains("file_prefix"))
message.file_prefix = j["file_prefix"];
if (j.contains("images_per_file"))
message.images_per_file = j["images_per_file"];
if (j.contains("user"))
message.user_data = j["user"];
if (j.contains("sample_name"))
message.sample_name = j["sample_name"];
if (j.contains("source_name"))
message.source_name = j["source_name"];
if (j.contains("source_type"))
message.source_type = j["source_type"];
if (j.contains("instrument_name"))
message.instrument_name = j["instrument_name"];
if (j.contains("total_flux"))
message.total_flux = j["total_flux"];
if (j.contains("attenuator_transmission"))
message.attenuator_transmission = j["attenuator_transmission"];
if (j.contains("space_group_number"))
message.space_group_number = j["space_group_number"];
if (j.contains("roi"))
ProcessROIConfig(message, j["roi"]);
if (j.contains("gain_file_names"))
message.gain_file_names = j["gain_file_names"];
if (j.contains("write_images"))
message.write_images = j["write_images"];
if (j.contains("write_master_file"))
message.write_master_file = j["write_master_file"];
if (j.contains("data_reduction_factor_serialmx"))
message.data_reduction_factor_serialmx = j["data_reduction_factor_serialmx"];
if (j.contains("experiment_group"))
message.experiment_group = j["experiment_group"];
if (j.contains("jfjoch_release"))
message.jfjoch_release = j["jfjoch_release"];
if (j.contains("socket_number"))
message.socket_number = j["socket_number"];
if (j.contains("writer_notification_zmq_addr"))
message.writer_notification_zmq_addr = j["writer_notification_zmq_addr"];
if (j.contains("bit_depth_readout"))
message.bit_depth_readout = j["bit_depth_readout"];
if (j.contains("summation_mode"))
message.summation_mode = j["summation_mode"];
if (j.contains("overwrite"))
message.overwrite = j["overwrite"];
if (j.contains("xfel_pulse_id"))
message.xfel_pulse_id = j["xfel_pulse_id"];
if (j.contains("file_format"))
message.file_format = ProcessHDF5Format(j["file_format"]);
if (j.contains("poni_rot1"))
message.poni_rot1 = j["poni_rot1"];
if (j.contains("poni_rot2"))
message.poni_rot2 = j["poni_rot2"];
if (j.contains("poni_rot3"))
message.poni_rot3 = j["poni_rot3"];
if (j.contains("detect_ice_rings"))
message.detect_ice_rings = j["detect_ice_rings"];
if (j.contains("images_per_trigger"))
message.images_per_trigger = j["images_per_trigger"];
if (j.contains("indexing_algorithm")) {
if (j["indexing_algorithm"] == "fft")
message.indexing_algorithm = IndexingAlgorithmEnum::FFT;
else if (j["indexing_algorithm"] == "fftw")
message.indexing_algorithm = IndexingAlgorithmEnum::FFTW;
else if (j["indexing_algorithm"] == "ffbidx")
message.indexing_algorithm = IndexingAlgorithmEnum::FFBIDX;
else
message.indexing_algorithm = IndexingAlgorithmEnum::None;
}
if (j.contains("geom_refinement_algorithm")) {
if (j["geom_refinement_algorithm"] == "beam_center")
message.geom_refinement_algorithm = GeomRefinementAlgorithmEnum::BeamCenter;
else
message.geom_refinement_algorithm = GeomRefinementAlgorithmEnum::None;
}
if (j.contains("smargon")
&& j["smargon"].is_object()
&& j["smargon"].contains("chi_deg")
&& j["smargon"].contains("phi_deg")
&& j["smargon"]["chi_deg"].is_number()
&& j["smargon"]["phi_deg"].is_number()) {
SmargonPosition smargon;
smargon.phi_deg = j["smargon"]["phi_deg"].get<float>();
smargon.chi_deg = j["smargon"]["chi_deg"].get<float>();
if (j["smargon"].contains("chi_axis") && j["smargon"]["chi_axis"].is_array()
&& j["smargon"]["chi_axis"].size() == 3)
smargon.chi_axis = {j["smargon"]["chi_axis"][0].get<float>(),
j["smargon"]["chi_axis"][1].get<float>(),
j["smargon"]["chi_axis"][2].get<float>()};
if (j["smargon"].contains("phi_axis") && j["smargon"]["phi_axis"].is_array()
&& j["smargon"]["phi_axis"].size() == 3)
smargon.phi_axis = {j["smargon"]["phi_axis"][0].get<float>(),
j["smargon"]["phi_axis"][1].get<float>(),
j["smargon"]["phi_axis"][2].get<float>()};
message.smargon_position = smargon;
}
if (j.contains("ring_current_mA"))
message.ring_current_mA = j["ring_current_mA"];
if (j.contains("sample_temperature_K"))
message.sample_temperature_K = j["sample_temperature_K"];
} catch (const std::exception &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Cannot parse user_data as valid JSON " + std::string(e.what()));
}
}
bool ProcessStartMessageElement(StartMessage &message, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
const auto key = GetCBORString(value);
try {
if (key == "magic_number")
CheckMagicNumber(value);
else if (key == "beam_center_x")
message.beam_center_x = GetCBORFloat(value);
else if (key == "beam_center_y")
message.beam_center_y = GetCBORFloat(value);
else if (key == "detector_distance")
message.detector_distance = GetCBORFloat(value);
else if (key == "number_of_images")
message.number_of_images = GetCBORUInt(value);
else if (key == "countrate_correction_enabled")
message.countrate_correction_enabled = GetCBORBool(value);
else if (key == "flatfield_enabled")
message.flatfield_enabled = GetCBORBool(value);
else if (key == "image_size_x")
message.image_size_x = GetCBORUInt(value);
else if (key == "image_size_y")
message.image_size_y = GetCBORUInt(value);
else if (key == "incident_energy")
message.incident_energy = GetCBORFloat(value);
else if (key == "incident_wavelength")
message.incident_wavelength = GetCBORFloat(value);
else if (key == "incident_wavelength_spread")
message.incident_wavelength_spread = GetCBORFloat(value);
else if (key == "frame_time")
message.frame_time = GetCBORFloat(value);
else if (key == "count_time")
message.count_time = GetCBORFloat(value);
else if (key == "saturation_value")
message.saturation_value = GetCBORInt(value);
else if (key == "error_value")
message.error_value = GetCBORInt(value);
else if (key == "pixel_size_x")
message.pixel_size_x = GetCBORFloat(value);
else if (key == "pixel_size_y")
message.pixel_size_y = GetCBORFloat(value);
else if (key == "sensor_thickness")
message.sensor_thickness = GetCBORFloat(value);
else if (key == "sensor_material")
message.sensor_material = GetCBORString(value);
else if (key == "detector_description")
message.detector_description = GetCBORString(value);
else if (key == "detector_serial_number")
message.detector_serial_number = GetCBORString(value);
else if (key == "series_unique_id")
message.run_name = GetCBORString(value);
else if (key == "series_id")
message.run_number = GetCBORUInt(value);
else if (key == "pixel_mask")
ProcessPixelMaskElement(message, value);
else if (key == "az_int_map")
ProcessAzintMapElement(message, value);
else if (key == "roi_map")
ProcessROIMapElement(message, value);
else if (key == "channels")
GetCBORStringArray(value, message.channels);
else if (key == "detector_translation")
ProcessAxis(value, message.detector_translation);
else if (key == "goniometer")
ProcessGoniometerMap(message, value);
else if (key == "grid_scan")
message.grid_scan = ProcessGridScan(value);
else if (key == "pixel_mask_enabled")
message.pixel_mask_enabled = GetCBORBool(value);
else if (key == "jungfrau_conversion_enabled")
message.jungfrau_conversion_enabled = GetCBORBool(value);
else if (key == "geometry_transformation_enabled")
message.geometry_transformation_enabled = GetCBORBool(value);
else if (key == "jungfrau_conversion_factor")
message.jungfrau_conversion_factor = GetCBORFloat(value);
else if (key == "arm_date")
message.arm_date = GetCBORDateTime(value);
else if (key == "fluorescence")
message.fluorescence_spectrum = GetCBORFluorescenceSpectrum(value);
else if (key == "user_data")
ProcessStartUserData(message, value);
else if (key == "unit_cell")
message.unit_cell = ProcessUnitCellElement(value);
else if (key == "max_spot_count")
message.max_spot_count = GetCBORUInt(value);
else if (key == "max_extra_lattices")
message.max_extra_lattices = GetCBORUInt(value);
else if (key == "threshold_energy")
message.threshold_energy = ProcessThresholdEnergy(value);
else if (key == "image_dtype") {
auto val = GetCBORString(value);
if (val == "uint8") {
message.pixel_signed = false;
message.bit_depth_image = 8;
} else if (val == "int8") {
message.pixel_signed = true;
message.bit_depth_image = 8;
} else if (val == "uint16") {
message.pixel_signed = false;
message.bit_depth_image = 16;
} else if (val == "int16") {
message.pixel_signed = true;
message.bit_depth_image = 16;
} else if (val == "uint32") {
message.pixel_signed = false;
message.bit_depth_image = 32;
} else if (val == "int32") {
message.pixel_signed = true;
message.bit_depth_image = 32;
}
} else if (key == "az_int_q_bin_count")
message.az_int_q_bin_count = GetCBORInt(value);
else if (key == "az_int_phi_bin_count")
message.az_int_phi_bin_count = GetCBORInt(value);
else if (key == "az_int_bin_to_q")
GetCBORFloatArray(value, message.az_int_bin_to_q);
else if (key == "az_int_bin_to_two_theta")
GetCBORFloatArray(value, message.az_int_bin_to_two_theta);
else if (key == "az_int_bin_to_phi")
GetCBORFloatArray(value, message.az_int_bin_to_phi);
else if (key == "summation")
message.summation = GetCBORUInt(value);
else if (key == "storage_cell_number")
message.storage_cell_number = GetCBORUInt(value);
else if (key == "storage_cell_delay")
message.storage_cell_delay_ns = GetRational(value).first;
else {
if (cbor_value_is_tag(&value))
cbor_value_advance(&value);
cbor_value_advance(&value);
}
return true;
} catch (const JFJochException &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Error processing start message, key " + key + ":" + e.what());
}
}
// Calibration
bool ProcessCalibration(CompressedImage &message, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
else {
auto key = GetCBORString(value);
try {
if (key == "data") {
message = ProcessImageData(value);
} else {
if (cbor_value_is_tag(&value))
cbor_value_advance(&value);
cbor_value_advance(&value);
}
return true;
} catch (const JFJochException &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Error processing calibration message, key " + key + ":" + e.what());
}
}
}
// EndMessage
void ProcessRadIntResultElement(EndMessage &message, CborValue &value) {
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
std::vector<float> val;
GetCBORFloatArray(map_value, val);
message.az_int_result[key] = val;
}
cborErr(cbor_value_leave_container(&value, &map_value));
}
void ProcessADUHistogramElement(EndMessage &message, CborValue &value) {
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
while (!cbor_value_at_end(&map_value)) {
auto key = GetCBORString(map_value);
std::vector<uint64_t> val;
GetCBORUInt64Array(map_value, val);
message.adu_histogram[key] = val;
}
cborErr(cbor_value_leave_container(&value, &map_value));
}
bool ProcessEndMessageElement(EndMessage &message, CborValue &value) {
if (cbor_value_at_end(&value))
return false;
auto key = GetCBORString(value);
try {
if (key == "end_date")
message.end_date = GetCBORString(value);
else if (key == "series_unique_id")
message.run_name = GetCBORString(value);
else if (key == "series_id")
message.run_number = GetCBORUInt(value);
else if (key == "max_image_number")
message.max_image_number = GetCBORUInt(value);
else if (key == "images_collected")
message.images_collected_count = GetCBORUInt(value);
else if (key == "images_sent_to_write")
message.images_sent_to_write_count = GetCBORUInt(value);
else if (key == "max_receiver_delay")
message.max_receiver_delay = GetCBORUInt(value);
else if (key == "data_collection_efficiency")
message.efficiency = GetCBORFloat(value);
else if (key == "az_int_result")
ProcessRadIntResultElement(message, value);
else if (key == "adu_histogram")
ProcessADUHistogramElement(message, value);
else if (key == "adu_histogram_bin_width")
message.adu_histogram_bin_width = GetCBORUInt(value);
else if (key == "bkg_estimate")
message.bkg_estimate = GetCBORFloat(value);
else if (key == "indexing_rate")
message.indexing_rate = GetCBORFloat(value);
else if (key == "indexed_lattice_count")
GetCBORInt32Array(value, message.indexed_lattice_count);
else if (key == "data_collection_efficiency_image")
GetCBORFloatArray(value, message.data_collection_efficiency);
else if (key == "spot_count")
GetCBORInt32Array(value, message.spot_count);
else if (key == "spot_count_ice_ring")
GetCBORInt32Array(value, message.spot_count_ice_ring);
else if (key == "spot_count_low_res")
GetCBORInt32Array(value, message.spot_count_low_res);
else if (key == "spot_count_indexed")
GetCBORInt32Array(value, message.spot_count_indexed);
else if (key == "image_indexed")
GetCBORUInt8Array(value, message.image_indexed);
else if (key == "v_bkg_estimate")
GetCBORFloatArray(value, message.v_bkg_estimate);
else if (key == "ice_ring_score")
GetCBORFloatArray(value, message.ice_ring_score);
else if (key == "spot_count_ice_control")
GetCBORFloatArray(value, message.spot_count_ice_control);
else if (key == "ice_ring_score_mean")
message.ice_ring_score_mean = GetCBORFloat(value);
else if (key == "profile_radius")
GetCBORFloatArray(value, message.profile_radius);
else if (key == "mosaicity")
GetCBORFloatArray(value, message.mosaicity);
else if (key == "bFactor")
GetCBORFloatArray(value, message.bFactor);
else if (key == "resolution_estimate")
GetCBORFloatArray(value, message.resolution_estimate);
else if (key == "min_viable_pixel_value")
GetCBORInt64Array(value, message.min_viable_pixel_value);
else if (key == "max_viable_pixel_value")
GetCBORInt64Array(value, message.max_viable_pixel_value);
else if (key == "saturated_pixel_count")
GetCBORInt32Array(value, message.saturated_pixel_count);
else if (key == "error_pixel_count")
GetCBORInt32Array(value, message.error_pixel_count);
else if (key == "image_scale_factor")
GetCBORFloatArray(value, message.image_scale_factor);
else if (key == "image_scale_cc")
GetCBORFloatArray(value, message.image_scale_cc);
else if (key == "image_scale_mosaicity")
GetCBORFloatArray(value, message.image_scale_mosaicity);
else if (key == "integrated_reflections")
GetCBORInt32Array(value, message.integrated_reflections);
else if (key == "niggli_class")
GetCBORUInt8Array(value, message.niggli_class);
else if (key == "unit_cell")
message.unit_cell = ProcessUnitCellElement(value);
else if (key == "pixel_sum")
GetCBORInt64Array(value, message.pixel_sum);
else if (key == "rotation_lattice") {
std::vector<float> tmp;
GetCBORFloatArray(value, tmp);
message.rotation_lattice = CrystalLattice(tmp);
} else if (key == "rotation_lattice_type")
message.rotation_lattice_type = GetCBORLatticeMessage(value);
else if (key == "rotation_extra_lattices") {
size_t array_len = GetCBORArrayLen(value);
CborValue array_value;
cborErr(cbor_value_enter_container(&value, &array_value));
for (size_t i = 0; i < array_len; i++) {
std::vector<float> tmp;
GetCBORFloatArray(array_value, tmp);
if (tmp.size() == 9)
message.rotation_extra_lattices.emplace_back(tmp);
}
cborErr(cbor_value_leave_container(&value, &array_value));
} else
cbor_value_advance(&value);
return true;
} catch (const JFJochException &e) {
throw JFJochException(JFJochExceptionCategory::CBORError,
"Error processing end message, key " + key + ":" + e.what());
}
}
}
std::shared_ptr<CBORStream2DeserializerOutput> CBORStream2Deserialize(const uint8_t *msg, size_t msg_size) {
CborParser parser;
CborValue value;
cborErr(cbor_parser_init(msg, msg_size, 0, &parser, &value));
if (GetCBORTag(value) != CborSignatureTag)
throw JFJochException(JFJochExceptionCategory::CBORError, "CBOR must start with dedicated tag");
if (!cbor_value_is_map(&value))
throw JFJochException(JFJochExceptionCategory::CBORError, "Serialized frame must be map in top level");
auto ret = std::make_shared<CBORStream2DeserializerOutput>();
CborValue map_value;
cborErr(cbor_value_enter_container(&value, &map_value));
ret->msg_type = DecodeType(map_value);
switch (ret->msg_type) {
case CBORImageType::IMAGE:
ret->data_message = DataMessage();
while (ProcessDataMessageElement(*ret->data_message, map_value)) {}
break;
case CBORImageType::CALIBRATION: {
ret->calibration = CompressedImage();
while (ProcessCalibration(*ret->calibration, map_value)) {}
break;
}
case CBORImageType::START: {
ret->start_message = StartMessage();
while (ProcessStartMessageElement(*ret->start_message, map_value)) {}
break;
}
case CBORImageType::END:
ret->end_message = EndMessage();
while (ProcessEndMessageElement(*ret->end_message, map_value)) {}
break;
case CBORImageType::METADATA:
ret->metadata = MetadataMessage();
while (ProcessMetadataMessageElement(*ret->metadata, map_value)) {}
break;
case CBORImageType::NONE:
break;
}
cborErr(cbor_value_leave_container(&value, &map_value));
return ret;
}
std::shared_ptr<CBORStream2DeserializerOutput> CBORStream2Deserialize(const std::vector<uint8_t>& msg) {
return CBORStream2Deserialize(msg.data(), msg.size());
}
std::shared_ptr<CBORStream2DeserializerOutput> CBORStream2Deserialize(const std::string& msg) {
return CBORStream2Deserialize(reinterpret_cast<const uint8_t*>(msg.data()), msg.size());
}