writer: detect overwrite conflict at start for back-channel transports
Previously an existing output file was only detected at the final rename, after the whole collection had run. Add an up-front check in the FileWriter constructor that fails before acquisition arms. Only the master file is checked, and only by the single writer that owns it (write_master_file). In a multi-writer TCP/ZMQ setup the per-image data files are staggered across writers by file number, and a writer does not even know the total writer count, so it cannot enumerate its own subset; checking all data files in every writer would make each writer stat files it never writes and race the writers already creating them. Data-file conflicts stay caught by their owning writer at the final rename. Gate the check on whether the transport can report the failure to the broker: the direct HDF5 pusher throws in-process and the TCP writer returns a START-failure ACK, so both abort the collection cleanly (verified: an asymmetric rejection cancels the already-started sibling). The ZeroMQ path is fire-and-forget with no back-channel, so StreamWriter opts out via image_puller.SupportsAck() and keeps writing .tmp files, failing only at the final rename (leaving the images on disk). Documented in docs/JFJOCH_WRITER.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -66,7 +66,11 @@ void StreamWriter::ProcessStartMessage() {
|
||||
writer_notification_zmq_addr = image_puller_output.cbor->start_message->writer_notification_zmq_addr;
|
||||
|
||||
try {
|
||||
file_writer = std::make_unique<FileWriter>(*image_puller_output.cbor->start_message);
|
||||
// Fail fast on an overwrite conflict only when the transport can report it
|
||||
// back to the broker (TCP ACK). The ZeroMQ path has no back-channel, so it
|
||||
// keeps writing .tmp files and fails at the final rename instead.
|
||||
file_writer = std::make_unique<FileWriter>(*image_puller_output.cbor->start_message,
|
||||
image_puller.SupportsAck());
|
||||
if (!file_done_address.empty())
|
||||
file_writer->SetupFinalizedFileSocket(file_done_address);
|
||||
logger.Info("Starting writing for dataset {} of {} images",
|
||||
|
||||
Reference in New Issue
Block a user