Files
Jungfraujoch/image_puller/ReceiveRingBuffer.cpp
T
leonarski_fandClaude Fable 5.1 1be6093429 Writer: received images live in a byte-bounded ring buffer until written and republished
Every stage between the socket and the HDF5 write was bounded by a message count and never by
bytes: ZeroMQ receive watermark 100, cbor_fifo 200, outside_fifo 1000, repub_fifo 200 - up to
some 1500 images in flight per writer process, each a whole compressed image, so 6-12 GB for a
JUNGFRAU 4M and 30-60 GB for a 9M, and the writer's memory was a quantity nobody had chosen.

ReceiveRingBuffer is a fixed ring of bytes owned by the ImagePuller base class, so the ZeroMQ
and TCP pullers share it. A message is received straight into it (TCP reads the payload into
the ring; ZeroMQ copies the message in) and travels as a shared_ptr region in ImagePullerOutput;
the deserialised image points into that region, republish sends it as is, and the bytes come
back when the last holder lets go. Space is reclaimed in arrival order: a region released out
of turn - republish finishes with messages in its own order - is marked and waits for the ones
in front of it. Because of that, what republish may hold is capped at a quarter of the ring,
and an image over the cap is not republished, exactly as one that finds the republish queue
full; otherwise a stalled republish consumer would pin the ring and slow the writer to its
rate. A message larger than the whole ring (a START or END of a very long run) goes on the heap,
so it cannot wedge the receiver.

A full ring stalls the receive thread, and what that does depends on the stream. On TCP the
pusher waits behind the writer's BUSY heartbeats and gives up only after a minute in which no
image at all was written. On ZeroMQ the pusher's send blocks for one second, then drops the
image and sends without waiting for the rest of the run - so on ZeroMQ a ring that is smaller
than the writer's stalls costs images, where the old 1500-message queues absorbed them. That
is the trade: the budget is now a number, and it has to be chosen. jfjoch_writer sets it with
-m/--rcv_ring_mb, default 2048 MB per writer process; the ZeroMQ receive watermark default goes
from 100 to 10 messages, since it now only counts what sits beyond the ring. The ring's peak
use is in the end-of-run log line, and docs/JFJOCH_WRITER.md has a Memory section with the
sizing rule.

The puller's own default is no ring at all, which keeps every other user as it was: the
broker's DECTRIS stream2 puller (JFJochServices, JFJochReceiverService) stays zero-copy with
the ZeroMQ message as the buffer, and the broker's direct HDF5 writer (HDF5FilePusher ->
FileWriter) never goes through a puller. The puller threads now also catch std::exception, as
the writer does since the previous commit: deserialising the END message of a million-image
run allocates a vector per per-image array, and a bad_alloc leaving the thread ended the process.

Tests: ReceiveRingBufferTest (reuse, wrap-around, full ring, out-of-order release, heap
fallback, zero size, stop, a region outliving its ring); ZMQImageCommTest_SmallRingBuffer (64
one-megabyte images through a three-image ring, nothing lost, peak within the bound); one of
the two writers in TCPImageCommTest_2Writers_WithAck runs on a three-image ring.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-13 12:51:34 +02:00

147 lines
4.6 KiB
C++

// SPDX-FileCopyrightText: 2026 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
// SPDX-License-Identifier: GPL-3.0-only
#include "ReceiveRingBuffer.h"
#include <algorithm>
#include <optional>
struct ReceiveRingBuffer::State {
// Uninitialised on purpose: a ring that is never filled should not cost its capacity in
// resident memory.
std::unique_ptr<uint8_t[]> memory;
const size_t capacity;
std::mutex m;
std::condition_variable cv;
// Live regions in arrival order. Only the oldest can give back space, so a region released out of
// turn is marked and waits.
struct Slot {
size_t offset;
size_t size;
bool released = false;
};
std::deque<Slot> slots;
// Next byte to hand out. The ring is empty when slots is empty; the oldest live byte is then
// slots.front().offset.
size_t tail = 0;
size_t utilization = 0;
size_t max_utilization = 0;
bool stopped = false;
explicit State(size_t in_capacity)
: memory(in_capacity > 0 ? std::make_unique_for_overwrite<uint8_t[]>(in_capacity) : nullptr),
capacity(in_capacity) {}
// Where a region of `size` bytes would start, or nothing if it does not fit right now. Called
// under the mutex.
std::optional<size_t> FindSpace(size_t size) const {
if (slots.empty())
return 0;
const size_t head = slots.front().offset;
// tail has come round to head: treated as full. (After a wrap the piece between the old tail
// and the end is free, but it is behind the live data and not worth tracking.)
if (tail == head)
return std::nullopt;
if (tail > head) {
// Live data is [head, tail): free space is behind it to the end, or before it at the start.
if (capacity - tail >= size)
return tail;
if (head >= size)
return 0;
return std::nullopt;
}
// Wrapped: live data is [head, capacity) + [0, tail), free space is [tail, head).
if (head - tail >= size)
return tail;
return std::nullopt;
}
void Release(size_t offset) {
std::unique_lock ul(m);
for (auto &s: slots) {
if (s.offset == offset) {
s.released = true;
break;
}
}
while (!slots.empty() && slots.front().released) {
utilization -= slots.front().size;
slots.pop_front();
}
if (slots.empty())
tail = 0;
cv.notify_all();
}
void ReleaseHeap(size_t size) {
std::unique_lock ul(m);
utilization -= size;
}
};
ReceiveRingBuffer::ReceiveRingBuffer(size_t capacity) : state(std::make_shared<State>(capacity)) {}
std::shared_ptr<ReceiveBufferRegion> ReceiveRingBuffer::Allocate(size_t size) {
// Nothing to hold, and a zero-length slot would share its offset with the next real one.
if (size == 0)
return std::make_shared<ReceiveBufferRegion>();
std::unique_lock ul(state->m);
if (size > state->capacity) {
// Heap fallback: counted, not bounded.
state->utilization += size;
state->max_utilization = std::max(state->max_utilization, state->utilization);
auto *region = new ReceiveBufferRegion{new uint8_t[size], size};
return std::shared_ptr<ReceiveBufferRegion>(region, [s = state](ReceiveBufferRegion *r) {
delete[] r->data;
s->ReleaseHeap(r->size);
delete r;
});
}
std::optional<size_t> offset;
state->cv.wait(ul, [&] {
if (state->stopped)
return true;
offset = state->FindSpace(size);
return offset.has_value();
});
if (!offset)
return nullptr;
state->slots.push_back({*offset, size});
state->tail = *offset + size;
state->utilization += size;
state->max_utilization = std::max(state->max_utilization, state->utilization);
auto *region = new ReceiveBufferRegion{state->memory.get() + *offset, size};
return std::shared_ptr<ReceiveBufferRegion>(region, [s = state, off = *offset](ReceiveBufferRegion *r) {
s->Release(off);
delete r;
});
}
void ReceiveRingBuffer::Stop() {
std::unique_lock ul(state->m);
state->stopped = true;
state->cv.notify_all();
}
size_t ReceiveRingBuffer::GetCapacity() const {
return state->capacity;
}
size_t ReceiveRingBuffer::GetUtilization() const {
std::unique_lock ul(state->m);
return state->utilization;
}
size_t ReceiveRingBuffer::GetMaxUtilization() const {
std::unique_lock ul(state->m);
return state->max_utilization;
}