TCPStreamPusher: hard backpressure cap so a wedged writer can't hang the run

The peer-liveness timeout only catches a *silent* writer. A misbehaving writer
that keeps sending BUSY heartbeats while never draining (e.g. a permanently
wedged filesystem) would otherwise block SendAll -- and, through it, the queued
SendImage path and the end-of-run frame_transformation_futures.get() -- forever.

Add a progress-based cap in SendAll: if no bytes leave the socket for
max_backpressure_timeout (default 60s, tunable via SetMaxBackpressureTimeout)
the connection is declared dead regardless of heartbeats. It is one global cap,
enforced everywhere SendAll runs, so it bounds both mid-run stalls and
finalization. Generous relative to the 15s liveness window, since a heartbeating
peer is given more grace than a silent one -- but finite.

Add TCPImageCommTest_WedgedWriter_DroppedByBackpressureCap: a writer that ACKs
START then stalls forever while heartbeating (cap 1.5s, liveness 5s) must have
its connection dropped, and neither the producers nor EndDataCollection may
hang. Verified to hang (timeout) with the cap disabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-25 15:55:49 +02:00
co-authored by Claude Opus 4.8
parent 2a9fd084ab
commit 3cd96b8607
3 changed files with 90 additions and 2 deletions
+59
View File
@@ -930,3 +930,62 @@ TEST_CASE("TCPImageCommTest_StalledWriter_SurvivesViaHeartbeat", "[TCP]") {
image_buffer.Finalize(std::chrono::seconds(5));
}
TEST_CASE("TCPImageCommTest_WedgedWriter_DroppedByBackpressureCap", "[TCP]") {
// A writer that keeps heartbeating but never drains (e.g. a permanently wedged
// filesystem) must not block the run or its finalization forever. The hard
// backpressure cap tears the connection down even though BUSY keeps arriving, and
// well before the (longer) peer-liveness timeout that those heartbeats keep at bay.
constexpr int64_t N = 1000;
constexpr auto liveness = std::chrono::milliseconds(5000); // kept fresh by heartbeats
constexpr auto max_backpressure = std::chrono::milliseconds(1500);
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1, 16 * 1024);
pusher.SetPeerLivenessTimeout(liveness);
pusher.SetMaxBackpressureTimeout(max_backpressure);
StallableWriterDouble writer(pusher.GetAddress()[0], 16 * 1024); // never released
REQUIRE(writer.Connected());
for (int attempt = 0; attempt < 200 && pusher.GetConnectedWriters() < 1; ++attempt)
std::this_thread::sleep_for(std::chrono::milliseconds(25));
REQUIRE(pusher.GetConnectedWriters() == 1);
ImageBuffer image_buffer(16 * 1024 * 1024);
image_buffer.StartMeasurement(static_cast<size_t>(4096));
StartMessage start{.images_per_file = 1000, .write_master_file = true};
pusher.StartDataCollection(start); // writer ACKs START, then stalls forever
auto sender = std::async(std::launch::async, [&] {
for (int64_t i = 0; i < N; i++) {
ZeroCopyReturnValue *slot = nullptr;
while ((slot = image_buffer.GetImageSlot()) == nullptr)
std::this_thread::sleep_for(std::chrono::milliseconds(1));
std::memset(slot->GetImage(), 0, 256);
slot->SetImageNumber(i);
slot->SetImageSize(256);
slot->ReadyToSend();
pusher.SendImage(*slot);
}
});
// The cap must fire and drop the connection despite continuous heartbeats.
bool dropped = false;
for (int attempt = 0; attempt < 400 && !dropped; ++attempt) {
if (pusher.GetConnectedWriters() == 0)
dropped = true;
else
std::this_thread::sleep_for(std::chrono::milliseconds(25));
}
CHECK(dropped);
// Neither the producers nor finalization may hang once the writer is wedged.
REQUIRE(sender.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
sender.get();
EndMessage end{};
CHECK(pusher.EndDataCollection(end) == false); // bounded, and reports failure
image_buffer.Finalize(std::chrono::seconds(5));
}