writer: detect overwrite conflict at start for back-channel transports

Previously an existing output file was only detected at the final rename,
after the whole collection had run. Add an up-front check in the FileWriter
constructor that fails before acquisition arms.

Only the master file is checked, and only by the single writer that owns it
(write_master_file). In a multi-writer TCP/ZMQ setup the per-image data files
are staggered across writers by file number, and a writer does not even know
the total writer count, so it cannot enumerate its own subset; checking all
data files in every writer would make each writer stat files it never writes
and race the writers already creating them. Data-file conflicts stay caught by
their owning writer at the final rename.

Gate the check on whether the transport can report the failure to the broker:
the direct HDF5 pusher throws in-process and the TCP writer returns a
START-failure ACK, so both abort the collection cleanly (verified: an asymmetric
rejection cancels the already-started sibling). The ZeroMQ path is fire-and-forget
with no back-channel, so StreamWriter opts out via image_puller.SupportsAck() and
keeps writing .tmp files, failing only at the final rename (leaving the images on
disk). Documented in docs/JFJOCH_WRITER.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-09 15:34:37 +02:00
co-authored by Claude Fable 5
parent 36be60774f
commit ad73b4af99
6 changed files with 199 additions and 18 deletions
+5 -1
View File
@@ -66,7 +66,11 @@ void StreamWriter::ProcessStartMessage() {
writer_notification_zmq_addr = image_puller_output.cbor->start_message->writer_notification_zmq_addr;
try {
file_writer = std::make_unique<FileWriter>(*image_puller_output.cbor->start_message);
// Fail fast on an overwrite conflict only when the transport can report it
// back to the broker (TCP ACK). The ZeroMQ path has no back-channel, so it
// keeps writing .tmp files and fails at the final rename instead.
file_writer = std::make_unique<FileWriter>(*image_puller_output.cbor->start_message,
image_puller.SupportsAck());
if (!file_done_address.empty())
file_writer->SetupFinalizedFileSocket(file_done_address);
logger.Info("Starting writing for dataset {} of {} images",