jfjoch_fpga_stills_test: one collection, as fast as it goes; docs say no host stage discards
Removed -r (host pacing and the per-rate loop with its table), -d (the dropping run), -p (the queue size) and -U with the dummy-module padding; the blocked fraction, measured from the removed queue-wait time, is gone - the card's own wait for host buffers (pipeline_stalls_host), printed per card, says how long the chain held the source. Options: -i -s -N -B -C -S -X -t -k -m -K -v. docs/TOOLS.md rewritten to match, and docs/JFJOCH_BROKER.md no longer says a frame finding the stills hold full is dropped: it waits, and frames are lost only at the card. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HVeDQrPqZ52Tp6rzzZGxf3
This commit is contained in:
1 parent
472759ee0b
commit
0f2b89cb80
3 files changed
+120
-219
No files matched your search
@@ -383,9 +383,10 @@ Frames arriving during the pre-pass, or faster than the analysis, wait in at mos
|
||||
of its slots hold a frame each (it can be seen through the image routes while it waits), the rest of that half
|
||||
holds frames as their compressed bytes, several to a slot; the other half is for the analysed frames, which go to
|
||||
the writer from a slot each, with one slot kept per analysis thread. The receiver's image memory is the image
|
||||
buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. A frame that finds the waiting
|
||||
half full is dropped and counted as for a full image buffer, so analysis slower than the frame rate costs frames,
|
||||
not memory. Where the writer lags behind, the analysis
|
||||
buffer (`image_buffer_MiB`, `-B` of `jfjoch_fpga_stills_test`) and nothing beside it. No stage on the host discards a
|
||||
frame: a frame that finds the waiting half full waits for room, and the wait goes back to the source - from FPGA
|
||||
cards, `FPGAImagePuller` holds the cards' buffers and the cards wait for the host, so frames are lost only at the
|
||||
card. Analysis slower than the frame rate thus costs frames at the card, not memory. Where the writer lags behind, the analysis
|
||||
waits for slots to come back. A pre-pass that fails leaves the stated geometry and
|
||||
mask in use. Where the start message cannot be sent after the pre-pass (no writer listening) or the analysis
|
||||
cannot be set up, the collection ends with that error, reported by `/status` and `/wait_till_done`, and the
|
||||
|
||||
+12
-25
@@ -74,7 +74,7 @@ synthesis C model on the CPU, so no FPGA device is required.
|
||||
Measures rugnux live stills on the FPGA card(s), with a stored dataset standing in for the detector:
|
||||
|
||||
```
|
||||
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-X <indexer>] [-r <Hz>,...] [-p <slots>] [-k <F>] [-m <frames>] [-d] [-v] [-K[<MiB>]] <master.h5>
|
||||
jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C a,b,c,al,be,ga] [-S <group>] [-X <indexer>] [-t <threads>] [-k <F>] [-m <frames>] [-v] [-K[<MiB>]] <master.h5>
|
||||
```
|
||||
|
||||
- The dataset must be a stored JUNGFRAU dataset with its modules assembled (1030 x 514 each, any placement
|
||||
@@ -88,7 +88,7 @@ jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C
|
||||
- The frames are loaded into the card's internal frame generator and the run is made with the JUNGFRAU
|
||||
conversion off, so the card returns them verbatim; the host assembles them as signed 16-bit photon
|
||||
counts and rugnux live stills analyses them through `FPGAImagePuller`, as the broker does with rugnux live
|
||||
on (`-d`: not blocking, as the broker runs; `-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
|
||||
on (`-v`: each card's progress every 0.5 s; `-k <F>`: analyse 1 of every
|
||||
F images on average, the rugnux live setting `stills_downsample`; `-m <N>`: N frames summed into each image, by
|
||||
the card up to 256 and above that the largest divisor of N up to 256 on the card and the rest on the host, 32-bit
|
||||
images). Beam centre, distance, energy and the cell
|
||||
@@ -101,30 +101,17 @@ jfjoch_fpga_stills_test [-i <images>] [-s <cards>] [-N <threads>] [-B <MiB>] [-C
|
||||
apply here) and prints the mapping. These are the first ones of the file, cycled for the whole run of `-i` images
|
||||
(default: as many as the file has). **Images repeat**, so throughput, latency, spots per image and
|
||||
indexing rate are meaningful, the merge and its statistics are not.
|
||||
- The run blocks at the source, no image is dropped: `FPGAImagePuller` waits for a free slot instead of
|
||||
dropping an image (and gives no module up for being behind the others), the stills receiver waits for
|
||||
room in the image buffer, so the card keeps its frames, its host writer waits for host buffers and the
|
||||
stall reaches the internal generator through the card's AXI-stream back-pressure (the generator sits
|
||||
before the network stack; a real detector's UDP packets would be lost at that point instead). The
|
||||
firmware cannot pace the generator (it has no frame period): `-r` paces it from the host, the puller
|
||||
taking image *i* from the cards no earlier than *i / rate* after the start, so the card waits for the
|
||||
host either way. One collection is run per rate of `-r` (default `0`, as fast as it goes). `-r` is refused
|
||||
with `-d`: holding a frame on the host makes the card wait, which is blocking again, and dropping the frames
|
||||
that come early would keep a different subset on each card, as their generators run apart.
|
||||
- One collection per invocation, as fast as it goes. Nothing on the host discards an image: `FPGAImagePuller`
|
||||
waits for room in its queue to the receiver and the stills receiver for room in the image buffer, so the card
|
||||
keeps its frames, its host writer waits for host buffers and the stall reaches the internal generator through
|
||||
the card's AXI-stream back-pressure (the generator sits before the network stack; a real detector's UDP
|
||||
packets would be lost at that point instead, the only place frames are lost). The card's own count of the
|
||||
cycles it waited for host buffers is printed per card: how long the chain behind it held the source.
|
||||
- The cards' generators run free, each as fast as its modules' bandwidth allows, so cards with different
|
||||
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) drift apart - unlike the modules of a detector, which a
|
||||
common trigger keeps together - and with `-d` the images started by the cards ahead hold the puller's slots
|
||||
until the frames of the card behind arrive. Where the modules do not divide evenly, every card is therefore
|
||||
given as many modules as the largest, the rest dummy modules (zeros) the generator sends and the puller does
|
||||
not read (`FPGAImagePuller::CardExperiment`; not part of the image, nor of its efficiency); `-U` leaves them
|
||||
out. The start lines give each card's modules and dummies; the distinct images follow the padded count. With
|
||||
`-U` (or cards that still drift, `-v` shows each card's frames) give `-p` at least the frames they drift apart.
|
||||
- For each rate it reports the rate images were delivered (first to last image), the fraction of that time
|
||||
the source was blocked by the chain behind it (the time the puller's module threads waited for a free
|
||||
slot, mean over the modules), the sustained rate (images / wall time, start to the end of the analysis)
|
||||
and the analysis rate after the pre-pass, and at the end a table of them: the highest rate with ~0%
|
||||
blocked is the rate the live stills chain sustains. The card's own count of the cycles it waited for host
|
||||
buffers is printed too; with `-r` it includes the pacing.
|
||||
numbers of modules (a 9M on 4 cards: 5, 5, 4, 4) run at different speeds - unlike the modules of a detector,
|
||||
which a common trigger keeps together; the run then goes at the pace of the slowest card.
|
||||
- It reports the rate images were delivered (first to last image), the sustained rate (images / wall time,
|
||||
start to the end of the analysis) and the analysis rate after the pre-pass.
|
||||
- At the end it prints images collected/sent/dropped, receiving time and frame rate, CPU use, indexing
|
||||
rate and spots per image, per-image times of the analysis stages, the pre-pass (time, beam centre and
|
||||
distance it decided), the latency from the last image to the end of the analysis and rugnux's steps.
|
||||
|
||||
+104
-191
@@ -11,7 +11,6 @@
|
||||
#include <atomic>
|
||||
#include <getopt.h>
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <thread>
|
||||
#include <sys/resource.h>
|
||||
|
||||
@@ -40,16 +39,9 @@ void print_usage(Logger &logger) {
|
||||
logger.Info(" -S<group> Space group of the collection (a space group stored in the file is ignored)");
|
||||
logger.Info(" -X<txt> Indexing (auto|fft|fftw|ffbidx), auto is default");
|
||||
logger.Info(" -t<num> Indexing thread pool size (default: 8, as offline rugnux on stills; the broker's default is 4)");
|
||||
logger.Info(" -r<Hz,...> Rates the source is paced at, one collection each (default: 0 = as fast as it goes); the");
|
||||
logger.Info(" host takes frame i no earlier than i/rate, the card waits meanwhile: blocking runs only");
|
||||
logger.Info(" -p<num> Images FPGAImagePuller keeps compressed for the receiver (default 128)");
|
||||
logger.Info(" -U No dummy modules: where the cards carry different numbers of modules, they run apart");
|
||||
logger.Info(" -k<num> Analyse 1 of every <num> images on average (rugnux live stills_downsample, default 1)");
|
||||
logger.Info(" -m<num> Frames summed into an image (default 1): the card sums the largest divisor up to 256, the");
|
||||
logger.Info(" host the rest (1000 = 250 on the card x 4 on the host)");
|
||||
logger.Info(" -d Not blocking, as the broker runs: images the host has no room for are dropped. The cards'");
|
||||
logger.Info(" generators run free, each at a speed that goes with its modules: where the modules do not");
|
||||
logger.Info(" divide evenly, each card is given dummy modules up to the largest count (-U: not)");
|
||||
logger.Info(" -K{<MiB>} Mock of the card instead of /dev/jfjoch* (optional: its generator memory, default 64 MiB =");
|
||||
logger.Info(" 128 module frames)");
|
||||
}
|
||||
@@ -94,9 +86,8 @@ int64_t GeneratorImages(AcquisitionDeviceGroup &aq_devices, const DiffractionExp
|
||||
// A stored dataset through the card and rugnux live stills; returns false where the run failed
|
||||
bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::string &file,
|
||||
std::optional<size_t> nimages, uint16_t nstreams, uint16_t nthreads, size_t send_buffer_size_MiB,
|
||||
const IndexingSettings &indexing, const std::optional<UnitCell> &cell, size_t puller_queue,
|
||||
const std::vector<double> &rates, bool dropping, bool pad, bool verbose, int64_t downsample,
|
||||
const std::optional<gemmi::SpaceGroup> &space_group, int64_t summation) {
|
||||
const IndexingSettings &indexing, const std::optional<UnitCell> &cell, bool verbose,
|
||||
int64_t downsample, const std::optional<gemmi::SpaceGroup> &space_group, int64_t summation) {
|
||||
RegisterHDF5Filter();
|
||||
JFJochHDF5Reader reader;
|
||||
reader.ReadFile(file);
|
||||
@@ -133,18 +124,12 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
|
||||
file_x.GetGemmiSpaceGroup()->hm);
|
||||
x.ImportIndexingSettings(indexing);
|
||||
|
||||
// The generator's speed goes with the modules of its card: with the modules not dividing evenly, every card is
|
||||
// given as many as the largest, the rest dummies the puller does not read (FPGAImagePuller::CardExperiment)
|
||||
const int64_t modules_per_card = (pad && (x.GetModulesNum() % nstreams != 0)) ? x.GetModulesNum(0) : 0;
|
||||
DiffractionExperiment card_x = FPGAImagePuller::CardExperiment(x, modules_per_card);
|
||||
|
||||
const int64_t generator_images = GeneratorImages(aq_devices, card_x, reader.GetNumberOfImages());
|
||||
const int64_t generator_images = GeneratorImages(aq_devices, x, reader.GetNumberOfImages());
|
||||
if (generator_images < 1) {
|
||||
logger.Error("The card memory cannot hold one image of {} modules", card_x.GetModulesNum(0));
|
||||
logger.Error("The card memory cannot hold one image of {} modules", x.GetModulesNum(0));
|
||||
return false;
|
||||
}
|
||||
x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES);
|
||||
card_x.InternalPacketGeneratorImages(generator_images, FPGA_INTERNAL_GENERATOR_MAX_IMAGES);
|
||||
|
||||
PixelMask mask(x);
|
||||
if (dataset->pixel_mask->GetMask().size() == x.GetPixelsNum())
|
||||
@@ -153,17 +138,9 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
|
||||
logger.Info("Dataset {}: {} images of {} modules, {} distinct ones in the generator, {} in the run",
|
||||
file, reader.GetNumberOfImages(), x.GetModulesNum(), generator_images, x.GetImageNum());
|
||||
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
||||
logger.Info(" card {}: modules {}-{} and {} dummy, generator memory {} module frames, {} used", i,
|
||||
logger.Info(" card {}: modules {}-{}, generator memory {} module frames, {} used", i,
|
||||
x.GetFirstModuleOfDataStream(i), x.GetFirstModuleOfDataStream(i) + x.GetModulesNum(i) - 1,
|
||||
card_x.GetModulesNum(i) - x.GetModulesNum(i), aq_devices[i].GetInternalGeneratorModuleFrames(),
|
||||
card_x.GetModulesNum(i) * generator_images);
|
||||
if (dropping && !pad) {
|
||||
for (int i = 1; i < x.GetDataStreamsNum(); i++)
|
||||
if (x.GetModulesNum(i) != x.GetModulesNum(0)) {
|
||||
logger.Warning("The cards carry different numbers of modules: their generators run apart (-U)");
|
||||
break;
|
||||
}
|
||||
}
|
||||
aq_devices[i].GetInternalGeneratorModuleFrames(), x.GetModulesNum(i) * generator_images);
|
||||
for (int m = 0; m < x.GetModulesNum(); m++)
|
||||
logger.Debug(" module {}: pixel 0 at ({}, {}), fast step {}, slow step {}", m,
|
||||
x.GetPixel0OfModuleConv(m) % x.GetXPixelsNum(), x.GetPixel0OfModuleConv(m) / x.GetXPixelsNum(),
|
||||
@@ -175,146 +152,110 @@ bool RunDataset(Logger &logger, AcquisitionDeviceGroup &aq_devices, const std::s
|
||||
service.Indexing(x.GetIndexingSettings());
|
||||
service.RugnuxLive(true, nullptr, 4, 60, false, false, downsample);
|
||||
|
||||
// Error, gap and beam-stop pixels of the reader become error pixels of the module frame. Card by card: each
|
||||
// card's modules of the image, then its dummies (zeros)
|
||||
// Error, gap and beam-stop pixels of the reader become error pixels of the module frame
|
||||
std::vector<int16_t> frame(x.GetModulesNum() * RAW_MODULE_SIZE);
|
||||
std::vector<uint16_t> card_frame(card_x.GetModulesNum() * RAW_MODULE_SIZE);
|
||||
std::vector<uint16_t> raw_frame(x.GetModulesNum() * RAW_MODULE_SIZE);
|
||||
for (int64_t i = 0; i < generator_images; i++) {
|
||||
const auto image = reader.LoadImage(i);
|
||||
SqueezeToRawGeometry<int16_t, int32_t>(x, frame.data(), image->Image().data(), BEAM_STOP_PXL_VALUE,
|
||||
SATURATED_PXL_VALUE, INT16_MIN, INT16_MAX);
|
||||
for (int d = 0; d < x.GetDataStreamsNum(); d++)
|
||||
memcpy(card_frame.data() + card_x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE,
|
||||
frame.data() + x.GetFirstModuleOfDataStream(d) * RAW_MODULE_SIZE,
|
||||
x.GetModulesNum(d) * RAW_MODULE_SIZE * sizeof(int16_t));
|
||||
service.LoadInternalGeneratorImage(card_x, card_frame, i);
|
||||
}
|
||||
|
||||
struct Row {
|
||||
double rate_hz, delivered_hz, blocked, sustained_hz, after_prepass_hz, indexing_rate;
|
||||
uint64_t images, dropped;
|
||||
};
|
||||
std::vector<Row> rows;
|
||||
bool ok = true;
|
||||
|
||||
for (const double rate: rates) {
|
||||
// Blocking at source: the puller waits for a free slot and the receiver for room in the image buffer,
|
||||
// the cards hold their frames and their generators stall
|
||||
service.FPGAImagePullerSettings(puller_queue, !dropping, rate, false, modules_per_card);
|
||||
logger.Info("");
|
||||
logger.Info("=== FPGAImagePuller (queue {}, {}), {} ===", puller_queue, dropping ? "dropping" : "blocking",
|
||||
(rate > 0) ? fmt::format("source paced at {:.1f} Hz", rate) : std::string("source as fast as it goes"));
|
||||
|
||||
rusage usage_start{};
|
||||
getrusage(RUSAGE_SELF, &usage_start);
|
||||
const auto wall_start = std::chrono::steady_clock::now();
|
||||
|
||||
service.Start(x, mask, nullptr);
|
||||
// -v: each card's progress, the frames of its fastest and slowest module written to the host
|
||||
std::atomic<bool> running = true;
|
||||
std::thread progress([&] {
|
||||
while (running) {
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(500));
|
||||
std::string line;
|
||||
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
||||
line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(),
|
||||
aq_devices[i].Counters().GetFastestFrameNumber(),
|
||||
aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : "");
|
||||
// Where a run stands, so that one that stops shows which part waits
|
||||
if (const auto st = service.GetStatus())
|
||||
line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected,
|
||||
st->images_sent, st->images_dropped);
|
||||
const auto live = service.GetRugnuxStatus();
|
||||
if (live.stills)
|
||||
line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held,
|
||||
live.stills->images_analysed, static_cast<int>(live.stills->prepass));
|
||||
if (verbose)
|
||||
logger.Info("{:.1f} s{}", std::chrono::duration<double>(std::chrono::steady_clock::now()
|
||||
- wall_start).count(), line);
|
||||
}
|
||||
});
|
||||
const auto output = service.Stop();
|
||||
running = false;
|
||||
progress.join();
|
||||
const auto live = service.GetRugnuxStatus();
|
||||
|
||||
const double wall_time = std::chrono::duration<double>(std::chrono::steady_clock::now() - wall_start).count();
|
||||
rusage usage_end{};
|
||||
getrusage(RUSAGE_SELF, &usage_end);
|
||||
auto seconds = [](const timeval &t) { return static_cast<double>(t.tv_sec) + static_cast<double>(t.tv_usec) / 1e6; };
|
||||
const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime)
|
||||
+ seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime);
|
||||
|
||||
int64_t spots = 0, spot_images = 0;
|
||||
for (const auto &e: output.scan_result.images)
|
||||
if (e.spot_count) {
|
||||
spots += e.spot_count.value();
|
||||
spot_images++;
|
||||
}
|
||||
|
||||
// From the first image to the last one into the receiver; the source was blocked for the time the module
|
||||
// threads waited for a free slot, as a fraction of that span (mean over the modules)
|
||||
const double delivery_s = live.last_image_s.value_or(0);
|
||||
const double delivered_hz = (delivery_s > 0) ? static_cast<double>(output.status.images_collected - 1) / delivery_s : 0;
|
||||
const double blocked = (delivery_s > 0) ? service.GetFPGAImagePullerQueueWaitTime_s() / delivery_s : 0;
|
||||
|
||||
logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}",
|
||||
x.GetImageNum(), output.status.images_collected, output.status.images_sent,
|
||||
output.status.images_dropped, output.efficiency);
|
||||
logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s), source blocked {:.1f}% of it",
|
||||
delivered_hz, delivery_s, 100.0 * blocked);
|
||||
logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores"
|
||||
" max resident memory {:.1f} GiB",
|
||||
static_cast<double>(output.status.images_collected) / wall_time, wall_time, cpu_time,
|
||||
cpu_time / wall_time, static_cast<double>(usage_end.ru_maxrss) / (1024.0 * 1024.0));
|
||||
// The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host
|
||||
for (int i = 0; i < x.GetDataStreamsNum(); i++) {
|
||||
const auto status = aq_devices[i].GetDataCollectionStatus();
|
||||
logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles",
|
||||
i, status.pipeline_stalls_host / 200e6,
|
||||
(delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0,
|
||||
status.pipeline_stalls_hbm);
|
||||
}
|
||||
logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0),
|
||||
static_cast<double>(spots) / std::max<int64_t>(1, spot_images));
|
||||
logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} "
|
||||
"refinement {:.0f} integration {:.0f} total {:.0f}",
|
||||
output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6,
|
||||
output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6,
|
||||
output.processing_time.integration * 1e6, output.processing_time.processing * 1e6);
|
||||
double after_prepass_hz = 0;
|
||||
if (live.stills) {
|
||||
const auto &st = *live.stills;
|
||||
after_prepass_hz = st.analysis_rate_hz.value_or(0);
|
||||
logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})",
|
||||
st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN),
|
||||
st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm);
|
||||
logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz",
|
||||
st.images_analysed, static_cast<double>(st.images_analysed) / x.GetImageNum(),
|
||||
st.images_indexed, after_prepass_hz);
|
||||
}
|
||||
logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message,
|
||||
live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN));
|
||||
for (const auto &step: live.steps)
|
||||
logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title);
|
||||
|
||||
rows.push_back(Row{.rate_hz = rate, .delivered_hz = delivered_hz, .blocked = blocked,
|
||||
.sustained_hz = static_cast<double>(output.status.images_collected) / wall_time,
|
||||
.after_prepass_hz = after_prepass_hz,
|
||||
.indexing_rate = output.status.indexing_rate.value_or(0),
|
||||
.images = output.status.images_collected, .dropped = output.status.images_dropped});
|
||||
ok = ok && (output.efficiency > 0) && (live.state == RugnuxLiveState::Done);
|
||||
memcpy(raw_frame.data(), frame.data(), frame.size() * sizeof(int16_t));
|
||||
service.LoadInternalGeneratorImage(x, raw_frame, i);
|
||||
}
|
||||
|
||||
logger.Info("");
|
||||
logger.Info("=== Rates ({} images, {} distinct: images repeat, the merge and its statistics are not a measure of the data) ===",
|
||||
x.GetImageNum(), generator_images);
|
||||
logger.Info(" source Hz delivered Hz blocked sustained Hz after pre-pass Hz indexing images dropped");
|
||||
for (const auto &r: rows)
|
||||
logger.Info(" {:>9} {:>12.1f} {:>6.1f}% {:>12.1f} {:>17.1f} {:>8.3f} {:>6} {:>7}",
|
||||
(r.rate_hz > 0) ? fmt::format("{:.1f}", r.rate_hz) : std::string("max"), r.delivered_hz,
|
||||
100.0 * r.blocked, r.sustained_hz, r.after_prepass_hz, r.indexing_rate, r.images, r.dropped);
|
||||
return ok;
|
||||
logger.Info("=== FPGAImagePuller, {} images ===", x.GetImageNum());
|
||||
|
||||
rusage usage_start{};
|
||||
getrusage(RUSAGE_SELF, &usage_start);
|
||||
const auto wall_start = std::chrono::steady_clock::now();
|
||||
|
||||
service.Start(x, mask, nullptr);
|
||||
// -v: each card's progress, the frames of its fastest and slowest module written to the host
|
||||
std::atomic<bool> running = true;
|
||||
std::thread progress([&] {
|
||||
while (running) {
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(500));
|
||||
std::string line;
|
||||
for (int i = 0; i < x.GetDataStreamsNum(); i++)
|
||||
line += fmt::format(" card {}: {} .. {}{}", i, aq_devices[i].Counters().GetSlowestFrameNumber(),
|
||||
aq_devices[i].Counters().GetFastestFrameNumber(),
|
||||
aq_devices[i].Counters().IsAcquisitionFinished() ? " done" : "");
|
||||
// Where a run stands, so that one that stops shows which part waits
|
||||
if (const auto st = service.GetStatus())
|
||||
line += fmt::format(" receiver: collected {} sent {} dropped {}", st->images_collected,
|
||||
st->images_sent, st->images_dropped);
|
||||
const auto live = service.GetRugnuxStatus();
|
||||
if (live.stills)
|
||||
line += fmt::format(" stills: held {} analysed {} pre-pass {}", live.stills->images_held,
|
||||
live.stills->images_analysed, static_cast<int>(live.stills->prepass));
|
||||
if (verbose)
|
||||
logger.Info("{:.1f} s{}", std::chrono::duration<double>(std::chrono::steady_clock::now()
|
||||
- wall_start).count(), line);
|
||||
}
|
||||
});
|
||||
const auto output = service.Stop();
|
||||
running = false;
|
||||
progress.join();
|
||||
const auto live = service.GetRugnuxStatus();
|
||||
|
||||
const double wall_time = std::chrono::duration<double>(std::chrono::steady_clock::now() - wall_start).count();
|
||||
rusage usage_end{};
|
||||
getrusage(RUSAGE_SELF, &usage_end);
|
||||
auto seconds = [](const timeval &t) { return static_cast<double>(t.tv_sec) + static_cast<double>(t.tv_usec) / 1e6; };
|
||||
const double cpu_time = seconds(usage_end.ru_utime) - seconds(usage_start.ru_utime)
|
||||
+ seconds(usage_end.ru_stime) - seconds(usage_start.ru_stime);
|
||||
|
||||
int64_t spots = 0, spot_images = 0;
|
||||
for (const auto &e: output.scan_result.images)
|
||||
if (e.spot_count) {
|
||||
spots += e.spot_count.value();
|
||||
spot_images++;
|
||||
}
|
||||
|
||||
// From the first image to the last one into the receiver
|
||||
const double delivery_s = live.last_image_s.value_or(0);
|
||||
const double delivered_hz = (delivery_s > 0) ? static_cast<double>(output.status.images_collected - 1) / delivery_s : 0;
|
||||
|
||||
logger.Info("Images expected {} collected {} sent {} dropped {} efficiency {:.4f}",
|
||||
x.GetImageNum(), output.status.images_collected, output.status.images_sent,
|
||||
output.status.images_dropped, output.efficiency);
|
||||
logger.Info("Delivered {:.1f} Hz (first to last image {:.2f} s)", delivered_hz, delivery_s);
|
||||
logger.Info("Sustained {:.1f} Hz (images / wall time {:.2f} s, start to analysis done) CPU {:.1f} s = {:.1f} cores"
|
||||
" max resident memory {:.1f} GiB",
|
||||
static_cast<double>(output.status.images_collected) / wall_time, wall_time, cpu_time,
|
||||
cpu_time / wall_time, static_cast<double>(usage_end.ru_maxrss) / (1024.0 * 1024.0));
|
||||
// The card's own count of the cycles (200 MHz) it waited for a host buffer, i.e. was held by the host: the
|
||||
// source blocked
|
||||
for (int i = 0; i < x.GetDataStreamsNum(); i++) {
|
||||
const auto status = aq_devices[i].GetDataCollectionStatus();
|
||||
logger.Info("Card {}: waited for host buffers {:.2f} s ({:.1f}% of the delivery), stalls before HBM {} cycles",
|
||||
i, status.pipeline_stalls_host / 200e6,
|
||||
(delivery_s > 0) ? 100.0 * status.pipeline_stalls_host / 200e6 / delivery_s : 0.0,
|
||||
status.pipeline_stalls_hbm);
|
||||
}
|
||||
logger.Info("Indexing rate {:.3f} spots per image {:.1f}", output.status.indexing_rate.value_or(0),
|
||||
static_cast<double>(spots) / std::max<int64_t>(1, spot_images));
|
||||
logger.Info("Per-image time (mean; microseconds): preprocess {:.0f} spot finding {:.0f} indexing {:.0f} "
|
||||
"refinement {:.0f} integration {:.0f} total {:.0f}",
|
||||
output.processing_time.preprocessing * 1e6, output.processing_time.spot_finding * 1e6,
|
||||
output.processing_time.indexing * 1e6, output.processing_time.refinement * 1e6,
|
||||
output.processing_time.integration * 1e6, output.processing_time.processing * 1e6);
|
||||
if (live.stills) {
|
||||
const auto &st = *live.stills;
|
||||
logger.Info("Pre-pass {:.2f} s: beam {:.2f} {:.2f} px (file {:.2f} {:.2f}), distance {:.2f} mm (file {:.2f})",
|
||||
st.prepass_s.value_or(0), st.beam_x_pxl.value_or(NAN), st.beam_y_pxl.value_or(NAN),
|
||||
st.file_beam_x_pxl, st.file_beam_y_pxl, st.distance_mm.value_or(NAN), st.file_distance_mm);
|
||||
logger.Info("Analysed {} ({:.3f} of the images) indexed {} analysis rate after the pre-pass {:.1f} Hz",
|
||||
st.images_analysed, static_cast<double>(st.images_analysed) / x.GetImageNum(),
|
||||
st.images_indexed, st.analysis_rate_hz.value_or(0));
|
||||
}
|
||||
logger.Info("Rugnux: {} {} last image at {:.2f} s, done {:.2f} s after it", live.verdict, live.message,
|
||||
live.last_image_s.value_or(NAN), live.last_image_to_written_s.value_or(NAN));
|
||||
for (const auto &step: live.steps)
|
||||
logger.Info(" step {:.2f} s + {:.2f} s {}", step.started_s, step.duration_s.value_or(NAN), step.title);
|
||||
|
||||
return (output.efficiency > 0) && (live.state == RugnuxLiveState::Done);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
@@ -330,18 +271,14 @@ int main(int argc, char **argv) {
|
||||
std::optional<UnitCell> cell;
|
||||
IndexingAlgorithmEnum indexing = IndexingAlgorithmEnum::Auto;
|
||||
uint16_t indexing_threads = 8; // as offline rugnux's stills (Rugnux::StillsIndexerThreads)
|
||||
size_t puller_queue = FPGAImagePuller::DefaultQueue;
|
||||
std::optional<uint32_t> mock_card_MiB;
|
||||
bool verbose = false;
|
||||
std::vector<double> rates = {0};
|
||||
bool dropping = false;
|
||||
bool pad = true;
|
||||
int64_t downsample = 1;
|
||||
int64_t summation = 1;
|
||||
std::optional<gemmi::SpaceGroup> space_group;
|
||||
|
||||
int opt;
|
||||
while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:r:p:dUk:m:K::")) != -1) {
|
||||
while ((opt = getopt(argc, argv, "vi:s:N:B:C:S:X:t:k:m:K::")) != -1) {
|
||||
switch (opt) {
|
||||
case 'v':
|
||||
verbose = true;
|
||||
@@ -380,23 +317,6 @@ int main(int argc, char **argv) {
|
||||
case 't':
|
||||
indexing_threads = atol(optarg);
|
||||
break;
|
||||
case 'r': {
|
||||
rates.clear();
|
||||
std::stringstream list(optarg);
|
||||
std::string rate;
|
||||
while (std::getline(list, rate, ','))
|
||||
rates.push_back(atof(rate.c_str()));
|
||||
break;
|
||||
}
|
||||
case 'p':
|
||||
puller_queue = atol(optarg);
|
||||
break;
|
||||
case 'd':
|
||||
dropping = true;
|
||||
break;
|
||||
case 'U':
|
||||
pad = false;
|
||||
break;
|
||||
case 'k':
|
||||
downsample = atol(optarg);
|
||||
break;
|
||||
@@ -425,13 +345,6 @@ int main(int argc, char **argv) {
|
||||
print_usage(logger);
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
// The generator has no frame period: -r holds frames on the host, which makes the card wait - blocking again.
|
||||
// Dropping the frames that come early instead would keep a different subset on each card, as they run apart.
|
||||
if (dropping && std::any_of(rates.begin(), rates.end(), [](double r) { return r > 0; })) {
|
||||
logger.Error("-r paces the source through the card waiting for the host, so it cannot be used with -d");
|
||||
exit(EXIT_FAILURE);
|
||||
}
|
||||
|
||||
// -v gives the progress lines only: the debug log has a line for every module frame
|
||||
logger.Verbose(false);
|
||||
|
||||
@@ -454,7 +367,7 @@ int main(int argc, char **argv) {
|
||||
bool ret = false;
|
||||
try {
|
||||
ret = RunDataset(logger, aq_devices, argv[optind], nimages, nstreams, nthreads, send_buffer_size_MiB,
|
||||
i_settings, cell, puller_queue, rates, dropping, pad, verbose, downsample, space_group,
|
||||
i_settings, cell, verbose, downsample, space_group,
|
||||
summation);
|
||||
} catch (std::exception &e) {
|
||||
logger.Error(e.what());
|
||||
|
||||
Reference in new issue
Block a user