mx/Jungfraujoch
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 66 Wiki Activity

viewer: fix HTTP live-follow OOM via datasetLoaded backpressure + shared pixel mask

Browse Source 
The viewer could grow to ~100 GB RAM when live-following an HTTP broker. The
rc.153 images_in_flight backpressure only throttled imageLoaded; the heavy
per-frame payload rides datasetLoaded, fanned out over ~10 queued cross-thread
connections with no cap. In HTTPSyncDataset follow mode (entered when an operator
clicks an image while following live) RefreshDatasetOnly_i emits a fresh full
dataset every autoload tick with no imageLoaded, so the gate never engaged and
the queued events - each pinning a full JFJochReaderDataset (full-detector
PixelMask + per-image plots) - accumulated without bound.

Backpressure datasetLoaded the same way as imageLoaded: a datasets_in_flight
counter (cap 2), all emits routed through EmitDatasetLoaded_i, and
AutoLoadTimerExpired gated on it (covers HTTPSyncDataset). The window routes the
worker's datasetLoaded through a single OnDatasetReady sink that fans out
synchronously via datasetReady and acks with datasetConsumed. Under load stale
datasets are dropped; the next tick sends the latest.

Share the pixel mask instead of deep-copying it: JFJochReaderDataset::pixel_mask
is now shared_ptr<const PixelMask>, so per-frame dataset copies share the ~72 MB
mask. UpdateUserMask does copy-on-write; JFJochHttpReader caches the mask by
arm_date so a live refresh reuses one shared mask per acquisition.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
leonarski_f
2026-07-09 10:58:10 +02:00
co-authored by Claude Fable 5
parent b4a29ebb32
commit dc8a80f3e0
16 changed files with 120 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
viewer/JFJochViewerWindow.cpp
+25 -12
View File
@@ -192,6 +192,14 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
connect(this, &JFJochViewerWindow::imageConsumed,
reading_worker, &JFJochImageReadingWorker::ImageConsumed);
// Dataset stream routed the same way, so it gets the same backpressure: the worker's datasetLoaded
// crosses the thread boundary once into OnDatasetReady, which re-emits datasetReady to every GUI
// consumer synchronously and then acks the worker (datasetConsumed).
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
this, &JFJochViewerWindow::OnDatasetReady);
connect(this, &JFJochViewerWindow::datasetConsumed,
reading_worker, &JFJochImageReadingWorker::DatasetConsumed);
connect(this, &JFJochViewerWindow::imageReady,
viewer, &JFJochDiffractionImage::loadImage);
@@ -257,13 +265,13 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
connect(menuBar, &JFJochViewerMenu::uploadUserMaskSelected,
reading_worker, &JFJochImageReadingWorker::UploadUserMask);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
tableWindow, &JFJochViewerImageListWindow::datasetLoaded);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
spotWindow, &JFJochViewerSpotListWindow::datasetLoaded);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
metadataWindow, &JFJochViewerMetadataWindow::datasetLoaded);
connect(this, &JFJochViewerWindow::imageReady,
@@ -297,7 +305,7 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
reading_worker, &JFJochImageReadingWorker::FindCenter);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
reflectionWindow, &JFJochViewerReflectionListWindow::datasetLoaded);
connect(this, &JFJochViewerWindow::imageReady,
@@ -360,12 +368,12 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
connect(calibrationWindow, &JFJochCalibrationWindow::loadCalibration,
reading_worker, &JFJochImageReadingWorker::LoadCalibration);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
calibrationWindow, &JFJochCalibrationWindow::datasetLoaded);
connect(reading_worker, &JFJochImageReadingWorker::simpleImageLoaded,
calibrationWindow, &JFJochCalibrationWindow::calibrationLoaded);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
reciprocalWindow, &JFJochViewerReciprocalSpaceWindow::datasetLoaded);
connect(this, &JFJochViewerWindow::imageReady,
reciprocalWindow, &JFJochViewerReciprocalSpaceWindow::imageLoaded);
@@ -395,9 +403,8 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
// Ensure worker is deleted in its own thread when the thread stops
connect(reading_thread, &QThread::finished, reading_worker, &QObject::deleteLater);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded, this,
[this](std::shared_ptr<const JFJochReaderDataset> ds) { lastDataset = std::move(ds); });
// lastImage is captured in OnImageReady, which sees every frame before fanning it out.
// lastDataset and lastImage are captured in OnDatasetReady / OnImageReady, which see every
// dataset / frame before fanning them out.
connect(this, &JFJochViewerWindow::imageReady,
toolBarDisplay, &JFJochViewerToolbarDisplay::imageLoaded);
@@ -488,7 +495,7 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
reading_worker, &JFJochImageReadingWorker::UpdateDataset);
connect(settingsPanel, &JFJochViewerSettingsDock::findBeamCenter,
reading_worker, &JFJochImageReadingWorker::FindCenter);
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
settingsPanel, &JFJochViewerSettingsDock::datasetLoaded);
connect(this, &JFJochViewerWindow::imageReady,
settingsPanel, &JFJochViewerSettingsDock::loadImage);
@@ -534,7 +541,7 @@ JFJochViewerWindow::JFJochViewerWindow(QWidget *parent, bool dbus, const QString
resizeDocks({lastDatasetInfoDock, imageStripDock}, {260, 140}, Qt::Vertical);
}
menuBar->AddDockEntry(imageStripDock, "Image strip");
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
imageStrip, &JFJochViewerImageStrip::datasetLoaded);
connect(reading_worker, &JFJochImageReadingWorker::fileOpened,
imageStrip, &JFJochViewerImageStrip::resetForNewFile);
@@ -616,6 +623,12 @@ void JFJochViewerWindow::OnImageReady(std::shared_ptr<const JFJochReaderImage> i
emit imageConsumed(); // ack the worker so it may fetch/produce the next frame
}
void JFJochViewerWindow::OnDatasetReady(std::shared_ptr<const JFJochReaderDataset> dataset) {
lastDataset = dataset; // newest dataset, for dataset-info docks opened later
emit datasetReady(dataset); // fan out to every GUI consumer synchronously (same thread)
emit datasetConsumed(); // ack the worker so it may produce the next dataset refresh
}
void JFJochViewerWindow::NewDatasetInfo() {
auto info = new JFJochViewerDatasetInfo(this);
info->datasetLoaded(lastDataset);
@@ -637,7 +650,7 @@ void JFJochViewerWindow::NewDatasetInfo() {
lastDatasetInfoDock = dock;
// Wire signals like the initial dataset_info
connect(reading_worker, &JFJochImageReadingWorker::datasetLoaded,
connect(this, &JFJochViewerWindow::datasetReady,
info, &JFJochViewerDatasetInfo::datasetLoaded);
connect(this, &JFJochViewerWindow::imageReady,
info, &JFJochViewerDatasetInfo::imageLoaded);