jfjoch_fpga_stills_test: one collection, as fast as it goes; docs say no host stage discards

Removed -r (host pacing and the per-rate loop with its table), -d (the
dropping run), -p (the queue size) and -U with the dummy-module padding; the
blocked fraction, measured from the removed queue-wait time, is gone - the
card's own wait for host buffers (pipeline_stalls_host), printed per card,
says how long the chain held the source. Options: -i -s -N -B -C -S -X -t
-k -m -K -v. docs/TOOLS.md rewritten to match, and docs/JFJOCH_BROKER.md
no longer says a frame finding the stills hold full is dropped: it waits,
and frames are lost only at the card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-11-18 19:53:43 +01:00
1 parent 472759ee0b
commit 0f2b89cb80
3 files changed
+120 -219

No files matched your search

+4 -3
View File
@@ -383,9 +383,10 @@ Frames arriving during the pre-pass, or faster than the analysis, wait in at mos
of its slots hold a frame each (it can be seen through the image routes while it waits), the rest of that half
holds frames as their compressed bytes, several to a slot; the other half is for the analysed frames, which go to
the writer from a slot each, with one slot kept per analysis thread. The receiver's image memory is the image
buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. A frame that finds the waiting
half full is dropped and counted as for a full image buffer, so analysis slower than the frame rate costs frames,
not memory. Where the writer lags behind, the analysis
buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. No stage on the host discards a
frame: a frame that finds the waiting half full waits for room, and the wait goes back to the source - from FPGA
cards, `FPGAImagePuller` holds the cards' buffers and the cards wait for the host, so frames are lost only at the
card. Analysis slower than the frame rate thus costs frames at the card, not memory. Where the writer lags behind, the analysis
waits for slots to come back. A pre-pass that fails leaves the stated geometry and
mask in use. Where the start message cannot be sent after the pre-pass (no writer listening) or the analysis
cannot be set up, the collection ends with that error, reported by `/status` and `/wait_till_done`, and the
+12 -25
View File
@@ -74,7 +74,7 @@ synthesis C model on the CPU, so no FPGA device is required.
Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector:
```
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-X <indexer>] [-r <Hz>,...] [-p <slots>] [-k <F>] [-m <frames>] [-d] [-v] [-K[<MiB>]] <master.h5>
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-S <group>] [-X <indexer>] [-t <threads>] [-k <F>] [-m <frames>] [-v] [-K[<MiB>]] <master.h5>
```
- The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement
@@ -88,7 +88,7 @@ jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C
- The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU
conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon
counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live
on (`-d`: not blocking, as the broker runs; `-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
on (`-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
F images on average, the rugnux live setting `stills_downsample`; `-m <N>`: N frames summed into each image, by
the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit
images). Beam centre, distance, energy and the cell
@@ -101,30 +101,17 @@ jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C
apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images
(default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and
indexing rate are meaningful, the merge and its statistics are not.
- The run blocks at the source, no image is dropped: `FPGAImagePuller` waits for a free slot instead of
dropping an image (and gives no module up for being behind the others), the stills receiver waits for
room in the image buffer, so the card keeps its frames, its host writer waits for host buffers and the
stall reaches the internal generator through the card's AXI-stream back-pressure (the generator sits
before the network stack; a real detector's UDP packets would be lost at that point instead). The
firmware cannot pace the generator (it has no frame period): `-r` paces it from the host, the puller
taking image *i* from the cards no earlier than *i / rate* after the start, so the card waits for the
host either way. One collection is run per rate of `-r` (default `0`, as fast as it goes). `-r` is refused
with `-d`: holding a frame on the host makes the card wait, which is blocking again, and dropping the frames
that come early would keep a different subset on each card, as their generators run apart.
- One collection per invocation, as fast as it goes. Nothing on the host discards an image: `FPGAImagePuller`
waits for room in its queue to the receiver and the stills receiver for room in the image buffer, so the card
keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through
the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP
packets would be lost at that point instead, the only place frames are lost). The card's own count of the
cycles it waited for host buffers is printed per card: how long the chain behind it held the source.
- The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) drift apart - unlike the modules of a detector, which a
common trigger keeps together - and with `-d` the images started by the cards ahead hold the puller's slots
until the frames of the card behind arrive. Where the modules do not divide evenly, every card is therefore
given as many modules as the largest, the rest dummy modules (zeros) the generator sends and the puller does
not read (`FPGAImagePuller::CardExperiment`; not part of the image, nor of its efficiency); `-U` leaves them
out. The start lines give each card's modules and dummies; the distinct images follow the padded count. With
`-U` (or cards that still drift, `-v` shows each card's frames) give `-p` at least the frames they drift apart.
- For each rate it reports the rate images were delivered (first to last image), the fraction of that time
the source was blocked by the chain behind it (the time the puller's module threads waited for a free
slot, mean over the modules), the sustained rate (images / wall time, start to the end of the analysis)
and the analysis rate after the pre-pass, and at the end a table of them: the highest rate with ~0%
blocked is the rate the live stills chain sustains. The card's own count of the cycles it waited for host
buffers is printed too; with `-r` it includes the pacing.
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector,
which a common trigger keeps together; the run then goes at the pace of the slowest card.
- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time,
start to the end of the analysis) and the analysis rate after the pre-pass.
- At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing
rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and
distance it decided), the latency from the last image to the end of the analysis and rugnux's steps.
+104 -191
View File
@@ -11,7 +11,6 @@
#include <atomic>
#include <getopt.h>
#include <iostream>
#include <sstream>
#include <thread>
#include <sys/resource.h>
@@ -40,16 +39,9 @@ void print_usage(Logger &logger) {
logger.Info(" -S<group> Space group of the collection (a space group stored in the file is ignored)");
logger.Info(" -X<txt> Indexing (auto|fft|fftw|ffbidx), auto is default");
logger.Info(" -t<num> Indexing thread pool size (default: 8, as offline rugnux on stills; the broker's default is 4)");
logger.Info(" -r<Hz,...> Rates the source is paced at, one collection each (default: 0 = as fast as it goes); the");
logger.Info(" host takes frame i no earlier than i/rate, the card waits meanwhile: blocking runs only");
logger.Info(" -p<num> Images FPGAImagePuller keeps compressed for the receiver (default 128)");
logger.Info(" -U No dummy modules: where the cards carry different numbers of modules, they run apart");
logger.Info(" -k<num> Analyse 1 of every <num> images on average (rugnux live stills_downsample, default 1)");
logger.Info(" -m<num> Frames summed into an image (default 1): the card sums the largest divisor up to 256, the");
logger.Info(" host the rest (1000 = 250 on the card x 4 on the host)");
logger.Info(" -d Not blocking, as the broker runs: images the host has no room for are dropped. The cards'");
logger.Info(" generators run free, each at a speed that goes with its modules: where the modules do not");
logger.Info(" divide evenly, each card is given dummy modules up to the largest count (-U: not)");
logger.Info(" -K{<MiB>} Mock of the card instead of /dev/jfjoch* (optional: its generator memory, default 64 MiB =");
logger.Info(" 128 module frames)");
}
@@ -94,9 +86,8 @@ int64_t GeneratorImages(AcquisitionDeviceGroup &aq_devices, const DiffractionExp
// A stored dataset through the card and rugnux live stills; returns false where the run failed
bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::string &file,
std::optional<size_t> nimages, uint16_t nstreams, uint16_t nthreads, size_t send_buffer_size_MiB,
const IndexingSettings &indexing, const std::optional<UnitCell> &cell, size_t puller_queue,
const std::vector<double> &rates, bool dropping, bool pad, bool verbose, int64_t downsample,
const std::optional<gemmi::SpaceGroup> &space_group, int64_t summation) {
const IndexingSettings &indexing, const std::optional<UnitCell> &cell, bool verbose,
int64_t downsample, const std::optional<gemmi::SpaceGroup> &space_group, int64_t summation) {
RegisterHDF5Filter();
JFJochHDF5Reader reader;
reader.ReadFile(file);
@@ -133,18 +124,12 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
file_x.GetGemmiSpaceGroup()->hm);
x.ImportIndexingSettings(indexing);
// The generator's speed goes with the modules of its card: with the modules not dividing evenly, every card is
// given as many as the largest, the rest dummies the puller does not read (FPGAImagePuller::CardExperiment)
const int64_t modules_per_card = (pad && (x.GetModulesNum() % nstreams != 0)) ? x.GetModulesNum(0) : 0;
DiffractionExperiment card_x = FPGAImagePuller::CardExperiment(x, modules_per_card);
const int64_t generator_images = GeneratorImages(aq_devices, card_x, reader.GetNumberOfImages());
const int64_t generator_images = GeneratorImages(aq_devices, x, reader.GetNumberOfImages());
if (generator_images < 1) {
logger.Error("The card memory cannot hold one image of {} modules", card_x.GetModulesNum(0));
logger.Error("The card memory cannot hold one image of {} modules", x.GetModulesNum(0));
return false;
}
x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES);
card_x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES);
PixelMask mask(x);
if (dataset->pixel_mask->GetMask().size() == x.GetPixelsNum())
@@ -153,17 +138,9 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
logger.Info("Dataset {}: {} images of {} modules, {} distinct ones in the generator, {} in the run",
file, reader.GetNumberOfImages(), x.GetModulesNum(), generator_images, x.GetImageNum());
for (int i = 0; i < x.GetDataStreamsNum(); i++)
logger.Info(" card {}: modules {}-{} and {} dummy, generator memory {} module frames, {} used", i,
logger.Info(" card {}: modules {}-{}, generator memory {} module frames, {} used", i,
x.GetFirstModuleOfDataStream(i), x.GetFirstModuleOfDataStream(i) + x.GetModulesNum(i) - 1,
card_x.GetModulesNum(i) - x.GetModulesNum(i), aq_devices[i].GetInternalGeneratorModuleFrames(),
card_x.GetModulesNum(i) * generator_images);
if (dropping && !pad) {
for (int i = 1; i < x.GetDataStreamsNum(); i++)
if (x.GetModulesNum(i) != x.GetModulesNum(0)) {
logger.Warning("The cards carry different numbers of modules: their generators run apart (-U)");
break;
}
}
aq_devices[i].GetInternalGeneratorModuleFrames(), x.GetModulesNum(i) * generator_images);
for (int m = 0; m < x.GetModulesNum(); m++)
logger.Debug(" module {}: pixel 0 at ({}, {}), fast step {}, slow step {}", m,
x.GetPixel0OfModuleConv(m) % x.GetXPixelsNum(), x.GetPixel0OfModuleConv(m) / x.GetXPixelsNum(),
@@ -175,146 +152,110 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
service.Indexing(x.GetIndexingSettings());
service.RugnuxLive(true, nullptr, 4, 60, false, false, downsample);
// Error, gap and beam-stop pixels of the reader become error pixels of the module frame. Card by card: each
// card's modules of the image, then its dummies (zeros)
// Error, gap and beam-stop pixels of the reader become error pixels of the module frame
std::vector<int16_t> frame(x.GetModulesNum() * RAW_MODULE_SIZE);
std::vector<uint16_t> card_frame(card_x.GetModulesNum() * RAW_MODULE_SIZE);
std::vector<uint16_t> raw_frame(x.GetModulesNum() * RAW_MODULE_SIZE);
for (int64_t i = 0; i < generator_images; i++) {
const auto image = reader.LoadImage(i);
SqueezeToRawGeometry<int16_t, int32_t>(x, frame.data(), image->Image().data(), BEAM_STOP_PXL_VALUE,
SATURATED_PXL_VALUE, INT16_MIN, INT16_MAX);
for (int d = 0; d < x.GetDataStreamsNum(); d++)
memcpy(card_frame.data() + card_x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE,
frame.data() + x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE,
x.GetModulesNum(d) * RAW_MODULE_SIZE * sizeof(int16_t));
service.LoadInternalGeneratorImage(card_x, card_frame, i);
}
struct Row {
double rate_hz, delivered_hz, blocked, sustained_hz, after_prepass_hz, indexing_rate;
uint64_t images, dropped;
};
std::vector<Row> rows;
bool ok = true;
for (const double rate: rates) {
// Blocking at source: the puller waits for a free slot and the receiver for room in the image buffer,
// the cards hold their frames and their generators stall
service.FPGAImagePullerSettings(puller_queue, !dropping, rate, false, modules_per_card);
logger.Info("");
logger.Info("=== FPGAImagePuller (queue {}, {}), {} ===", puller_queue, dropping ? "dropping" : "blocking",
(rate > 0) ? fmt::format("source paced at {:.1f} Hz", rate) : std::string("source as fast as it goes"));
rusage usage_start{};
getrusage(RUSAGE_SELF, &usage_start);
const auto wall_start = std::chrono::steady_clock::now();
service.Start(x, mask, nullptr);
// -v: each card's progress, the frames of its fastest and slowest module written to the host
std::atomic<bool> running = true;
std::thread progress([&] {
while (running) {
std::this_thread::sleep_for(std::chrono::milliseconds(500));
std::string line;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(),
aq_devices[i].Counters().GetFastestFrameNumber(),
aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : "");
// Where a run stands, so that one that stops shows which part waits
if (const auto st = service.GetStatus())
line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected,
st->images_sent, st->images_dropped);
const auto live = service.GetRugnuxStatus();
if (live.stills)
line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held,
live.stills->images_analysed, static_cast<int>(live.stills->prepass));
if (verbose)
logger.Info("{:.1f} s{}", std::chrono::duration<double>(std::chrono::steady_clock::now()
- wall_start).count(), line);
}
});
const auto output = service.Stop();
running = false;
progress.join();
const auto live = service.GetRugnuxStatus();
const double wall_time = std::chrono::duration<double>(std::chrono::steady_clock::now() - wall_start).count();
rusage usage_end{};
getrusage(RUSAGE_SELF, &usage_end);
auto seconds = [](const timeval &t) { return static_cast<double>(t.tv_sec) + static_cast<double>(t.tv_usec) / 1e6; };
const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime)
+ seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime);
int64_t spots = 0, spot_images = 0;
for (const auto &e: output.scan_result.images)
if (e.spot_count) {
spots += e.spot_count.value();
spot_images++;
}
// From the first image to the last one into the receiver; the source was blocked for the time the module
// threads waited for a free slot, as a fraction of that span (mean over the modules)
const double delivery_s = live.last_image_s.value_or(0);
const double delivered_hz = (delivery_s > 0) ? static_cast<double>(output.status.images_collected - 1) / delivery_s : 0;
const double blocked = (delivery_s > 0) ? service.GetFPGAImagePullerQueueWaitTime_s() / delivery_s : 0;
logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}",
x.GetImageNum(), output.status.images_collected, output.status.images_sent,
output.status.images_dropped, output.efficiency);
logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s), source blocked {:.1f}% of it",
delivered_hz, delivery_s, 100.0 * blocked);
logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores"
" max resident memory {:.1f} GiB",
static_cast<double>(output.status.images_collected) / wall_time, wall_time, cpu_time,
cpu_time / wall_time, static_cast<double>(usage_end.ru_maxrss) / (1024.0 * 1024.0));
// The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host
for (int i = 0; i < x.GetDataStreamsNum(); i++) {
const auto status = aq_devices[i].GetDataCollectionStatus();
logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles",
i, status.pipeline_stalls_host / 200e6,
(delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0,
status.pipeline_stalls_hbm);
}
logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0),
static_cast<double>(spots) / std::max<int64_t>(1, spot_images));
logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} "
"refinement {:.0f} integration {:.0f} total {:.0f}",
output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6,
output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6,
output.processing_time.integration * 1e6, output.processing_time.processing * 1e6);
double after_prepass_hz = 0;
if (live.stills) {
const auto &st = *live.stills;
after_prepass_hz = st.analysis_rate_hz.value_or(0);
logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})",
st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN),
st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm);
logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz",
st.images_analysed, static_cast<double>(st.images_analysed) / x.GetImageNum(),
st.images_indexed, after_prepass_hz);
}
logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message,
live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN));
for (const auto &step: live.steps)
logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title);
rows.push_back(Row{.rate_hz = rate, .delivered_hz = delivered_hz, .blocked = blocked,
.sustained_hz = static_cast<double>(output.status.images_collected) / wall_time,
.after_prepass_hz = after_prepass_hz,
.indexing_rate = output.status.indexing_rate.value_or(0),
.images = output.status.images_collected, .dropped = output.status.images_dropped});
ok = ok && (output.efficiency > 0) && (live.state == RugnuxLiveState::Done);
memcpy(raw_frame.data(), frame.data(), frame.size() * sizeof(int16_t));
service.LoadInternalGeneratorImage(x, raw_frame, i);
}
logger.Info("");
logger.Info("=== Rates ({} images, {} distinct: images repeat, the merge and its statistics are not a measure of the data) ===",
x.GetImageNum(), generator_images);
logger.Info(" source Hz delivered Hz blocked sustained Hz after pre-pass Hz indexing images dropped");
for (const auto &r: rows)
logger.Info(" {:>9} {:>12.1f} {:>6.1f}% {:>12.1f} {:>17.1f} {:>8.3f} {:>6} {:>7}",
(r.rate_hz > 0) ? fmt::format("{:.1f}", r.rate_hz) : std::string("max"), r.delivered_hz,
100.0 * r.blocked, r.sustained_hz, r.after_prepass_hz, r.indexing_rate, r.images, r.dropped);
return ok;
logger.Info("=== FPGAImagePuller, {} images ===", x.GetImageNum());
rusage usage_start{};
getrusage(RUSAGE_SELF, &usage_start);
const auto wall_start = std::chrono::steady_clock::now();
service.Start(x, mask, nullptr);
// -v: each card's progress, the frames of its fastest and slowest module written to the host
std::atomic<bool> running = true;
std::thread progress([&] {
while (running) {
std::this_thread::sleep_for(std::chrono::milliseconds(500));
std::string line;
for (int i = 0; i < x.GetDataStreamsNum(); i++)
line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(),
aq_devices[i].Counters().GetFastestFrameNumber(),
aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : "");
// Where a run stands, so that one that stops shows which part waits
if (const auto st = service.GetStatus())
line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected,
st->images_sent, st->images_dropped);
const auto live = service.GetRugnuxStatus();
if (live.stills)
line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held,
live.stills->images_analysed, static_cast<int>(live.stills->prepass));
if (verbose)
logger.Info("{:.1f} s{}", std::chrono::duration<double>(std::chrono::steady_clock::now()
- wall_start).count(), line);
}
});
const auto output = service.Stop();
running = false;
progress.join();
const auto live = service.GetRugnuxStatus();
const double wall_time = std::chrono::duration<double>(std::chrono::steady_clock::now() - wall_start).count();
rusage usage_end{};
getrusage(RUSAGE_SELF, &usage_end);
auto seconds = [](const timeval &t) { return static_cast<double>(t.tv_sec) + static_cast<double>(t.tv_usec) / 1e6; };
const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime)
+ seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime);
int64_t spots = 0, spot_images = 0;
for (const auto &e: output.scan_result.images)
if (e.spot_count) {
spots += e.spot_count.value();
spot_images++;
}
// From the first image to the last one into the receiver
const double delivery_s = live.last_image_s.value_or(0);
const double delivered_hz = (delivery_s > 0) ? static_cast<double>(output.status.images_collected - 1) / delivery_s : 0;
logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}",
x.GetImageNum(), output.status.images_collected, output.status.images_sent,
output.status.images_dropped, output.efficiency);
logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s)", delivered_hz, delivery_s);
logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores"
" max resident memory {:.1f} GiB",
static_cast<double>(output.status.images_collected) / wall_time, wall_time, cpu_time,
cpu_time / wall_time, static_cast<double>(usage_end.ru_maxrss) / (1024.0 * 1024.0));
// The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host: the
// source blocked
for (int i = 0; i < x.GetDataStreamsNum(); i++) {
const auto status = aq_devices[i].GetDataCollectionStatus();
logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles",
i, status.pipeline_stalls_host / 200e6,
(delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0,
status.pipeline_stalls_hbm);
}
logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0),
static_cast<double>(spots) / std::max<int64_t>(1, spot_images));
logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} "
"refinement {:.0f} integration {:.0f} total {:.0f}",
output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6,
output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6,
output.processing_time.integration * 1e6, output.processing_time.processing * 1e6);
if (live.stills) {
const auto &st = *live.stills;
logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})",
st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN),
st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm);
logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz",
st.images_analysed, static_cast<double>(st.images_analysed) / x.GetImageNum(),
st.images_indexed, st.analysis_rate_hz.value_or(0));
}
logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message,
live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN));
for (const auto &step: live.steps)
logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title);
return (output.efficiency > 0) && (live.state == RugnuxLiveState::Done);
}
int main(int argc, char **argv) {
@@ -330,18 +271,14 @@ int main(int argc, char **argv) {
std::optional<UnitCell> cell;
IndexingAlgorithmEnum indexing = IndexingAlgorithmEnum::Auto;
uint16_t indexing_threads = 8; // as offline rugnux's stills (Rugnux::StillsIndexerThreads)
size_t puller_queue = FPGAImagePuller::DefaultQueue;
std::optional<uint32_t> mock_card_MiB;
bool verbose = false;
std::vector<double> rates = {0};
bool dropping = false;
bool pad = true;
int64_t downsample = 1;
int64_t summation = 1;
std::optional<gemmi::SpaceGroup> space_group;
int opt;
while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:r:p:dUk:m:K::")) != -1) {
while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:k:m:K::")) != -1) {
switch (opt) {
case 'v':
verbose = true;
@@ -380,23 +317,6 @@ int main(int argc, char **argv) {
case 't':
indexing_threads = atol(optarg);
break;
case 'r': {
rates.clear();
std::stringstream list(optarg);
std::string rate;
while (std::getline(list, rate, ','))
rates.push_back(atof(rate.c_str()));
break;
}
case 'p':
puller_queue = atol(optarg);
break;
case 'd':
dropping = true;
break;
case 'U':
pad = false;
break;
case 'k':
downsample = atol(optarg);
break;
@@ -425,13 +345,6 @@ int main(int argc, char **argv) {
print_usage(logger);
exit(EXIT_FAILURE);
}
// The generator has no frame period: -r holds frames on the host, which makes the card wait - blocking again.
// Dropping the frames that come early instead would keep a different subset on each card, as they run apart.
if (dropping && std::any_of(rates.begin(), rates.end(), [](double r) { return r > 0; })) {
logger.Error("-r paces the source through the card waiting for the host, so it cannot be used with -d");
exit(EXIT_FAILURE);
}
// -v gives the progress lines only: the debug log has a line for every module frame
logger.Verbose(false);
@@ -454,7 +367,7 @@ int main(int argc, char **argv) {
bool ret = false;
try {
ret = RunDataset(logger, aq_devices, argv[optind], nimages, nstreams, nthreads, send_buffer_size_MiB,
i_settings, cell, puller_queue, rates, dropping, pad, verbose, downsample, space_group,
i_settings, cell, verbose, downsample, space_group,
summation);
} catch (std::exception &e) {
logger.Error(e.what());