TCPStreamPusher: hard backpressure cap so a wedged writer can't hang the run
The peer-liveness timeout only catches a *silent* writer. A misbehaving writer that keeps sending BUSY heartbeats while never draining (e.g. a permanently wedged filesystem) would otherwise block SendAll -- and, through it, the queued SendImage path and the end-of-run frame_transformation_futures.get() -- forever. Add a progress-based cap in SendAll: if no bytes leave the socket for max_backpressure_timeout (default 60s, tunable via SetMaxBackpressureTimeout) the connection is declared dead regardless of heartbeats. It is one global cap, enforced everywhere SendAll runs, so it bounds both mid-run stalls and finalization. Generous relative to the 15s liveness window, since a heartbeating peer is given more grace than a silent one -- but finite. Add TCPImageCommTest_WedgedWriter_DroppedByBackpressureCap: a writer that ACKs START then stalls forever while heartbeating (cap 1.5s, liveness 5s) must have its connection dropped, and neither the producers nor EndDataCollection may hang. Verified to hang (timeout) with the cap disabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -930,3 +930,62 @@ TEST_CASE("TCPImageCommTest_StalledWriter_SurvivesViaHeartbeat", "[TCP]") {
|
||||
|
||||
image_buffer.Finalize(std::chrono::seconds(5));
|
||||
}
|
||||
|
||||
TEST_CASE("TCPImageCommTest_WedgedWriter_DroppedByBackpressureCap", "[TCP]") {
|
||||
// A writer that keeps heartbeating but never drains (e.g. a permanently wedged
|
||||
// filesystem) must not block the run or its finalization forever. The hard
|
||||
// backpressure cap tears the connection down even though BUSY keeps arriving, and
|
||||
// well before the (longer) peer-liveness timeout that those heartbeats keep at bay.
|
||||
constexpr int64_t N = 1000;
|
||||
constexpr auto liveness = std::chrono::milliseconds(5000); // kept fresh by heartbeats
|
||||
constexpr auto max_backpressure = std::chrono::milliseconds(1500);
|
||||
|
||||
TCPStreamPusher pusher("tcp://127.0.0.1:*", 1, 16 * 1024);
|
||||
pusher.SetPeerLivenessTimeout(liveness);
|
||||
pusher.SetMaxBackpressureTimeout(max_backpressure);
|
||||
|
||||
StallableWriterDouble writer(pusher.GetAddress()[0], 16 * 1024); // never released
|
||||
REQUIRE(writer.Connected());
|
||||
|
||||
for (int attempt = 0; attempt < 200 && pusher.GetConnectedWriters() < 1; ++attempt)
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(25));
|
||||
REQUIRE(pusher.GetConnectedWriters() == 1);
|
||||
|
||||
ImageBuffer image_buffer(16 * 1024 * 1024);
|
||||
image_buffer.StartMeasurement(static_cast<size_t>(4096));
|
||||
|
||||
StartMessage start{.images_per_file = 1000, .write_master_file = true};
|
||||
pusher.StartDataCollection(start); // writer ACKs START, then stalls forever
|
||||
|
||||
auto sender = std::async(std::launch::async, [&] {
|
||||
for (int64_t i = 0; i < N; i++) {
|
||||
ZeroCopyReturnValue *slot = nullptr;
|
||||
while ((slot = image_buffer.GetImageSlot()) == nullptr)
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(1));
|
||||
std::memset(slot->GetImage(), 0, 256);
|
||||
slot->SetImageNumber(i);
|
||||
slot->SetImageSize(256);
|
||||
slot->ReadyToSend();
|
||||
pusher.SendImage(*slot);
|
||||
}
|
||||
});
|
||||
|
||||
// The cap must fire and drop the connection despite continuous heartbeats.
|
||||
bool dropped = false;
|
||||
for (int attempt = 0; attempt < 400 && !dropped; ++attempt) {
|
||||
if (pusher.GetConnectedWriters() == 0)
|
||||
dropped = true;
|
||||
else
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(25));
|
||||
}
|
||||
CHECK(dropped);
|
||||
|
||||
// Neither the producers nor finalization may hang once the writer is wedged.
|
||||
REQUIRE(sender.wait_for(std::chrono::seconds(10)) == std::future_status::ready);
|
||||
sender.get();
|
||||
|
||||
EndMessage end{};
|
||||
CHECK(pusher.EndDataCollection(end) == false); // bounded, and reports failure
|
||||
|
||||
image_buffer.Finalize(std::chrono::seconds(5));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user