leonarski_fandClaude Fable 5
ad73b4af99
Previously an existing output file was only detected at the final rename, after the whole collection had run. Add an up-front check in the FileWriter constructor that fails before acquisition arms. Only the master file is checked, and only by the single writer that owns it (write_master_file). In a multi-writer TCP/ZMQ setup the per-image data files are staggered across writers by file number, and a writer does not even know the total writer count, so it cannot enumerate its own subset; checking all data files in every writer would make each writer stat files it never writes and race the writers already creating them. Data-file conflicts stay caught by their owning writer at the final rename. Gate the check on whether the transport can report the failure to the broker: the direct HDF5 pusher throws in-process and the TCP writer returns a START-failure ACK, so both abort the collection cleanly (verified: an asymmetric rejection cancels the already-started sibling). The ZeroMQ path is fire-and-forget with no back-channel, so StreamWriter opts out via image_puller.SupportsAck() and keeps writing .tmp files, failing only at the final rename (leaving the images on disk). Documented in docs/JFJOCH_WRITER.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
51 lines
2.0 KiB
C++
51 lines
2.0 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <numeric>
|
|
|
|
#include "HDF5DataFile.h"
|
|
#include "../common/JFJochMessages.h"
|
|
#include "../common/ZMQWrappers.h"
|
|
#include "HDF5NXmx.h"
|
|
#include "CBFWriter.h"
|
|
#include <unordered_set>
|
|
|
|
class FileWriter {
|
|
FileWriterFormat format = FileWriterFormat::NXmxLegacy;
|
|
|
|
StartMessage start_message;
|
|
std::unique_ptr<NXmx> master_file;
|
|
std::vector<std::unique_ptr<HDF5DataFile> > files;
|
|
std::vector<HDF5DataFileStatistics> stats;
|
|
std::unique_ptr<ZMQSocket> finalized_file_socket;
|
|
std::unique_ptr<CBFWriter> cbf_writer;
|
|
|
|
std::unordered_set<uint64_t> closed_files;
|
|
constexpr static uint64_t close_file_lag_images = 1000;
|
|
constexpr static uint64_t default_images_per_file = 1000;
|
|
void CreateHDF5MasterFile(const StartMessage& msg);
|
|
void CheckOutputFilesAvailable() const;
|
|
void AddStats(const std::optional<HDF5DataFileStatistics>& s);
|
|
void CloseFile(uint64_t file_number);
|
|
void CloseOldFiles(uint64_t current_image_number);
|
|
|
|
public:
|
|
// check_overwrite_at_start: fail the constructor if an output file already
|
|
// exists and overwrite is off. Only safe for transports with a back-channel
|
|
// (direct HDF5 pusher, TCP) that can report the failure to the broker before
|
|
// acquisition arms. The ZeroMQ pusher is fire-and-forget with no back-channel,
|
|
// so it must pass false: the writer then falls back to writing .tmp files and
|
|
// failing at the final rename (see docs/JFJOCH_WRITER.md).
|
|
explicit FileWriter(const StartMessage &request, bool check_overwrite_at_start = true);
|
|
void Write(const DataMessage& msg);
|
|
void WriteTIFF(const DataMessage& msg);
|
|
void WriteHDF5(const DataMessage& msg);
|
|
void WriteHDF5(const CompressedImage& msg);
|
|
void WriteHDF5(const EndMessage& msg);
|
|
std::vector<HDF5DataFileStatistics> Finalize();
|
|
void SetupFinalizedFileSocket(const std::string &addr);
|
|
std::optional<std::string> GetZMQAddr();
|
|
};
|