mx/Jungfraujoch
3
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 81 Wiki Activity

rugnux pre-scan: a frame past the kept-frame budget is read into the worker's buffer

Browse Source 
The budget is checked before the read, so a frame that will not be kept is read into the worker's
reused buffer instead of a fresh one paged in and let go (a 16 Mpx CBF sample kept 29 of 60 frames
and paged in all 60), and a kept frame borrows the worker's reader scratch for the read instead of
allocating one of its own. The budget can now be passed by at most a frame per worker.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EizBKhTqrAYago9KJG3dA3
This commit is contained in:
leonarski_f
2026-10-12 09:56:39 +02:00
co-authored by Claude Opus 5.5
parent 70e11d9ab8
commit be5e6feb15
1 changed files with 15 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
rugnux/RugnuxPreScan.cpp
+15 -14
View File
@@ -154,8 +154,8 @@ namespace {
// measurement and the defective pixels used to read it again each, and a CBF, SMV or marCCD frame
// is decoded on the host every time it is read. Frames are kept as the reader gives them,
// compressed where the file is: a few MB a frame on a 16 Mpx EIGER, 4 bytes a pixel where the
// reader decodes. A frame that would take the kept frames past this is let go and read again
// where it is wanted, so a sample of large decoded frames costs at most this much memory.
// reader decodes. Frames are kept while the kept ones take less than this - so it is passed by at
// most a frame per worker - and the rest are read again where they are wanted.
constexpr size_t PRESCAN_KEPT_BYTES = size_t{2} << 30;
// fn(t) for t < n on threads of their own, thread t on card t % gpus as an image-loop worker is: an
@@ -428,15 +428,23 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
for (size_t i = next.fetch_add(1); i < ordinals.size(); i = next.fetch_add(1)) {
const int ordinal = ordinals[i];
const int image_idx = start_image + ordinal * config_.stride;
JFJochReaderRawImage &raw_image = keep_frames ? kept[i] : scratch;
// Into a slot of its own while there is room to keep it, otherwise into the worker's
// buffer, which is reused frame after frame rather than paged in afresh for each. The
// reader's own scratch (JFJochReaderRawImage::read_buffer) is not part of the image, so
// a kept frame borrows the worker's for the read and gives it back.
const bool keep = keep_frames && kept_bytes.load() < PRESCAN_KEPT_BYTES;
JFJochReaderRawImage &raw_image = keep ? kept[i] : scratch;
if (keep)
raw_image.read_buffer = std::move(scratch.read_buffer);
bool read = false;
try {
read = reader_.ReadRawImage(image_idx, raw_image);
} catch (const std::exception &e) {
if (IsFatalResourceError(e)) throw;
logger.Warning("Pre-scan: failed to load image {}: {}", image_idx, e.what());
continue;
}
if (keep)
scratch.read_buffer = std::move(raw_image.read_buffer);
if (!read) continue;
frames_read.fetch_add(1, std::memory_order_relaxed);
// Gated on find_shadow, not on the frame set alone: AddImage projects the whole
@@ -474,16 +482,9 @@ void Rugnux::PreScan(int start_image, int images_to_process, int frame_count, Ru
img.GetHeight(), pixel_mask_.GetMask(),
experiment_.GetSaturationLimit());
}
if (keep_frames) {
// The reader's own scratch is not part of the image (JFJochReaderRawImage).
raw_image.read_buffer = {};
const size_t bytes = raw_image.image_buffer.capacity();
if (kept_bytes.fetch_add(bytes) + bytes <= PRESCAN_KEPT_BYTES)
is_kept[i] = 1;
else {
kept_bytes.fetch_sub(bytes);
raw_image = {};
}
if (keep) {
kept_bytes.fetch_add(raw_image.image_buffer.capacity());
is_kept[i] = 1;
}
}
});