mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 66 Wiki Activity

viewer: fix HTTP live-follow OOM via datasetLoaded backpressure + shared pixel mask

Browse Source 
The viewer could grow to ~100 GB RAM when live-following an HTTP broker. The
rc.153 images_in_flight backpressure only throttled imageLoaded; the heavy
per-frame payload rides datasetLoaded, fanned out over ~10 queued cross-thread
connections with no cap. In HTTPSyncDataset follow mode (entered when an operator
clicks an image while following live) RefreshDatasetOnly_i emits a fresh full
dataset every autoload tick with no imageLoaded, so the gate never engaged and
the queued events - each pinning a full JFJochReaderDataset (full-detector
PixelMask + per-image plots) - accumulated without bound.

Backpressure datasetLoaded the same way as imageLoaded: a datasets_in_flight
counter (cap 2), all emits routed through EmitDatasetLoaded_i, and
AutoLoadTimerExpired gated on it (covers HTTPSyncDataset). The window routes the
worker's datasetLoaded through a single OnDatasetReady sink that fans out
synchronously via datasetReady and acks with datasetConsumed. Under load stale
datasets are dropped; the next tick sends the latest.

Share the pixel mask instead of deep-copying it: JFJochReaderDataset::pixel_mask
is now shared_ptr<const PixelMask>, so per-frame dataset copies share the ~72 MB
mask. UpdateUserMask does copy-on-write; JFJochHttpReader caches the mask by
arm_date so a live refresh reuses one shared mask per acquisition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-07-09 10:58:10 +02:00
co-authored by Claude Fable 5
parent b4a29ebb32
commit dc8a80f3e0
16 changed files with 120 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
reader/JFJochHttpReader.cpp
+13 -2
View File
@@ -19,6 +19,8 @@ void JFJochHttpReader::Close() {
SetStartMessage({});
last_image_buffer_counter = {};
last_op_http_sync = false;
cached_pixel_mask.reset();
cached_pixel_mask_arm_date.clear();
}
ImageBufferStatus JFJochHttpReader::GetImageBufferStatus() const {
@@ -162,8 +164,17 @@ std::shared_ptr<JFJochReaderDataset> JFJochHttpReader::UpdateDataset_i() {
std::chrono::microseconds(std::lround(msg->start_message->count_time * 1e6))
);
if (!msg->start_message->pixel_mask.empty())
dataset->pixel_mask = PixelMask(msg->start_message->pixel_mask.begin()->second);
if (!msg->start_message->pixel_mask.empty()) {
// The pixel mask is constant for an acquisition; build the full-detector (~tens of MB)
// PixelMask once per arm and share it across the per-refresh dataset snapshots, so a live
// dataset refresh does not reconstruct and copy it on every tick.
if (!cached_pixel_mask || cached_pixel_mask_arm_date != dataset->arm_date) {
cached_pixel_mask =
std::make_shared<const PixelMask>(msg->start_message->pixel_mask.begin()->second);
cached_pixel_mask_arm_date = dataset->arm_date;
}
dataset->pixel_mask = cached_pixel_mask;
}
dataset->experiment.NumTriggers(1);
dataset->experiment.ImagesPerTrigger(msg->start_message->number_of_images);