leonarski_fandClaude Opus 5
259b43154e
Two ways the written files could misdescribe themselves without anyone noticing. The pixel format is stated twice and the two were never compared: each image carries its own type as a CBOR tag, which is what the data files are written with, while the master is typed from the start message. A stream whose header contradicts its images produced data files of one type under a master declaring another, and with NXmxVDS, HDF5 then converts silently on every read. HDF5DataFile::CreateFile now checks the two agree and refuses the run otherwise - the point where the values first meet, so it covers every path into the writer. Two test fixtures were relying on exactly that inconsistency. The HDF5 writer tests wrote uint16 buffers under a JUNGFRAU experiment, which converts to photon counts by default and so declares int16; they never read the pixels back, so it went unnoticed. The receiver-lite tests feed frames from compression_benchmark.h5, which really are signed int16, through a DECTRIS experiment, which declares unsigned by default - the same class of bug the pixel_signed propagation fixed on the live path. Both now declare what they send. Second: a virtual dataset whose source file is absent reads as the fill value, and HDF5 defaults that to zero, so a data file that was not copied alongside the master is indistinguishable from frames of genuine zero counts. Measured with DIALS on a four-file set with one file removed: 25 frames of pure zeros, no error and no warning. The image VDS is now filled with the error marker instead, which sits outside underload_value..saturation_value, so a reader masks those frames. Same measurement after the change: -32768 throughout, which DIALS excludes. Only the images ask for a fill value; the per-image metadata datasets keep the default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
66 lines
1.8 KiB
C++
66 lines
1.8 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <queue>
|
|
#include <string>
|
|
#include <chrono>
|
|
|
|
#include "HDF5Objects.h"
|
|
#include "../common/SpotToSave.h"
|
|
#include "../common/JFJochMessages.h"
|
|
|
|
#include "HDF5DataFilePlugin.h"
|
|
|
|
struct HDF5DataFileStatistics {
|
|
std::string filename;
|
|
uint64_t file_number; // counting from 1!
|
|
size_t max_image_number;
|
|
uint64_t total_images;
|
|
};
|
|
|
|
class HDF5DataFile {
|
|
const std::string filename;
|
|
const uint64_t file_number;
|
|
const bool write_images;
|
|
|
|
// Pixel format the START message declared. The data files are typed from the images themselves
|
|
// and the master from the start message, so these two have to agree - see CreateFile.
|
|
const uint64_t declared_bit_depth;
|
|
const bool declared_pixel_signed;
|
|
|
|
std::string tmp_filename;
|
|
|
|
std::shared_ptr<HDF5File> data_file = nullptr;
|
|
std::unique_ptr<HDF5DataSet> data_set = nullptr;
|
|
std::vector<std::unique_ptr<HDF5DataFilePlugin>> plugins;
|
|
size_t images_per_file;
|
|
size_t xpixel;
|
|
size_t ypixel;
|
|
size_t max_image_number;
|
|
size_t nimages;
|
|
|
|
std::string image_units;
|
|
|
|
std::vector<uint64_t> timestamp;
|
|
std::vector<uint32_t> exptime;
|
|
std::vector<uint64_t> number;
|
|
|
|
int32_t image_low;
|
|
|
|
bool closed = false;
|
|
|
|
bool overwrite = false;
|
|
bool new_file = true;
|
|
bool manage_file = false;
|
|
public:
|
|
HDF5DataFile(const StartMessage &msg, uint64_t file_number, const std::string &filename);
|
|
~HDF5DataFile();
|
|
std::optional<HDF5DataFileStatistics> Close();
|
|
void Write(const DataMessage& msg, uint64_t image_number);
|
|
size_t GetNumImages() const;
|
|
|
|
void CreateFile(const DataMessage& msg, std::shared_ptr<HDF5File> data_file);
|
|
};
|