TCPStreamPusher::SendImage is called from the receiver's frame-transformation
threads. It used a 128-deep per-connection queue and waited when it was full.
With images routed per file ((n / images_per_file) % writers), every writer
receives a whole file as a burst at the full frame rate (1000 images/file at
2 kHz fills 128 in ~64 ms), so the queue filled on every burst and all
transformation threads ended up waiting on one writer. They then stopped
returning FPGA frame buffers, and the FPGA dropped packets: partial images,
images without a background estimate, data loss on every writer. The ZeroMQ
pusher has an unbounded queue and never blocks, which is why it coped.
- The per-connection queue is unbounded: every element owns a distinct
image-buffer slot, so the bound is the image buffer's slot count. When slots
run out the receiver drops only the write (GetImageSlot() == nullptr), after
analysis. Absorbing one file burst therefore needs an image buffer of about
images_per_file slots (~20 GB for a 9M at 1000 images/file); below that,
images are still dropped, now as writer-queue-full instead of packet loss.
- SendImage never blocks. A dead or wedged writer is still caught in SendAll
(15 s silence / 60 s without progress), after which WriterThread returns the
queued slots unsent.
- The 200 ms fallback in StopDataCollectionThreads is gone: at END it closed
the connection of any writer whose queue stayed full for 200 ms - a normal
state for a slow but healthy writer - discarding up to 128 queued images and
never sending END. END now follows the whole backlog.
- A per-connection enqueue mutex makes "check active, enqueue" atomic against
"clear active, enqueue the end sentinel": an image could land behind the
sentinel when a connection was torn down mid-run, leaking its slot
(CheckIfBufferReturned then threw) or being sent as stale data in the next
run. New stress test SendImageRacingEnd_NoSlotLeak (verified to fail with
the race widened and the lock removed).
- TearDownConnection marks the connection broken first, so the destructor
returns a backlog instead of sending it.
Writer side:
- jfjoch_writer -b/--rcv_buffer sets the TCP receive socket buffer (was
hardcoded to the kernel default).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
leonarski_fandClaude Opus 5.5
d7521bf1ea
