v1.0.0-rc.154 #64

Merged

leonarski_f merged 7 commits from 2606-tcp into main 2026-06-25 18:12:00 +02:00

Conversation 0 Commits 7 Files Changed 135

+639 -413

leonarski_f commented 2026-06-25 16:03:49 +02:00

Owner

Copy Link

Copy Source

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.

• jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability)

This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use. * jfjoch_broker: Fix to TCP file pusher (remove kernel zero copy to improve reliability)

leonarski_f added 6 commits 2026-06-25 16:03:50 +02:00

TCPStreamPusher: remove MSG_ZEROCOPY machinery, use plain blocking send ... f859f8108f

The MSG_ZEROCOPY path was the common factor behind the occasional mid-run
writer disconnects and added substantial failure surface for no benefit at
this throughput (tens of MB/s): the socket error queue raises POLLERR as a
normal event (entangled with liveness detection), and the per-connection
completion-id counter was reset every run while the kernel's sk_zckey is
monotonic for the life of the socket, so on a persistent connection the
bookkeeping diverged from run 2 onward.

Replace it with a straightforward synchronous send():
- SendAll/SendFrame lose all zerocopy params; DATA payloads are sent with a
  plain ::send(MSG_NOSIGNAL), so the image-buffer slot is owned by the kernel
  once send() returns and WriterThread releases it immediately.
- Drop ZeroCopyCompletionThread and the zc_pending/zc_mutex/zc_cv/zc_*_id
  state, SO_ZEROCOPY setup, and the errqueue include.
- StopDataCollectionThreads now drains and releases any queued-but-unsent
  slots on the stalled-writer path (active==false makes WriterThread exit
  without draining) instead of Clear()-ing them, avoiding a slot leak.

The BUSY-heartbeat peer-liveness timeout (backpressure tolerance) is kept;
it is independent of zerocopy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TCPStreamPusher: post-zerocopy cleanup + fix queue-path backpressure drop ... 2a9fd084ab

Follow-up simplifications after removing the zerocopy machinery, plus a real
backpressure bug the cleanup surfaced:

- SendImage(ZeroCopyReturnValue&) imposed a hard 2s deadline on enqueueing and
  then marked the connection broken. At high frame rate the 128-deep queue
  fills in tens of ms, so any filesystem stall longer than ~2s dropped the run
  even though the writer was alive and heartbeating -- defeating the whole
  BUSY-heartbeat backpressure design. Block instead while the peer is alive
  (!broken && active); the real liveness decision already lives in SendAll's
  peer-liveness timeout, which the writer's BUSY heartbeats keep fresh. This
  makes the queue path consistent with the send path: both wait out arbitrarily
  long stalls and only give up when the peer goes genuinely silent.
- Drop the dead per-connection data_sent counter (written, never read) and the
  redundant ImagePusherQueueElement.image_data set on the TCP path (only the
  HDF5 pusher reads that field).
- Add SetPeerLivenessTimeout() so the liveness window is tunable (and testable).

Add TCPImageCommTest_StalledWriter_SurvivesViaHeartbeat: a controllable raw
writer double connects, ACKs START, then stops draining for 4s while still
sending BUSY heartbeats (peer-liveness window set to 2s). The run must ride out
the stall on the zero-copy queue path and deliver all 1000 images. Verified to
fail (115/1000 delivered, connection dropped) against the old 2s-deadline
behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TCPStreamPusher: hard backpressure cap so a wedged writer can't hang the run ... 3cd96b8607

The peer-liveness timeout only catches a *silent* writer. A misbehaving writer
that keeps sending BUSY heartbeats while never draining (e.g. a permanently
wedged filesystem) would otherwise block SendAll -- and, through it, the queued
SendImage path and the end-of-run frame_transformation_futures.get() -- forever.

Add a progress-based cap in SendAll: if no bytes leave the socket for
max_backpressure_timeout (default 60s, tunable via SetMaxBackpressureTimeout)
the connection is declared dead regardless of heartbeats. It is one global cap,
enforced everywhere SendAll runs, so it bounds both mid-run stalls and
finalization. Generous relative to the 15s liveness window, since a heartbeating
peer is given more grace than a silent one -- but finite.

Add TCPImageCommTest_WedgedWriter_DroppedByBackpressureCap: a writer that ACKs
START then stalls forever while heartbeating (cap 1.5s, liveness 5s) must have
its connection dropped, and neither the producers nor EndDataCollection may
hang. Verified to hang (timeout) with the cap disabled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

broker: optional TCP liveness/backpressure timeouts in config ... a02fd19af4

Expose the two new TCPStreamPusher knobs through the broker config so they can be
tuned per deployment, like send_buffer_size already is:

- peer_liveness_timeout_ms  -> SetPeerLivenessTimeout()
- max_backpressure_timeout_ms -> SetMaxBackpressureTimeout()

Both are optional fields on tcp_settings; when unset the parser leaves the
pusher's built-in defaults (15s / 60s) in place, so existing config files keep
working unchanged. The parser also ignores non-positive values.

The generated broker/gen/model/Tcp_settings.{h,cpp} are hand-patched to mirror
exactly what openapi-generator emits for these fields (verified against the
send_buffer_size pattern and by a from_json/to_json round-trip), so re-running
update_version.sh reproduces them with no diff.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Regenerate OpenAPI 51bfb57782

VERSION: 1.0.0-rc.154 26554c86e3

leonarski_f added 1 commit 2026-06-25 16:52:08 +02:00

TCPStreamPusher: flush queued images before END so writer can't drop tail ... cdbd0b7bed

Under load on the CI machine, the JFJochIntegrationTest_TCP_* tests
intermittently failed with the writer reporting fewer images than sent
(e.g. 4 of 5) and the DATA-ACK count short by one.

EndDataCollection sent the END frame as soon as it could acquire
send_mutex, without waiting for the per-connection WriterThread to drain
its queue. SendImage only enqueues; the WriterThread actually transmits
DATA frames. On a fast machine the queue is already empty when END is
sent, but on a loaded machine the WriterThread falls behind, END
overtakes the still-queued trailing image(s), the remote writer sees END
and finalizes, and the late DATA frame is silently dropped by
ProcessDataImage's Finalized case (no ACK emitted).

Fix: flush before END. WriterThread now drains the queue until the end
sentinel instead of bailing the moment c->active is cleared, and
EndDataCollection stops (and joins) the WriterThread before sending END.
Joining the writer is a race-free barrier: every image is on the wire
before END goes out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

leonarski_f merged commit 6136f858af into main 2026-06-25 18:12:00 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

appleb_m (appleb_m) buntschu_d (buntschu_d) duan_j (duan_j) glettig_w (glettig_w) gotthard_g (gotthard_g) leonarski_f (leonarski_f) mx-gitea perl_d (perl_d) takaba_k (takaba_k)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mx/Jungfraujoch#64