Four changes, all from measuring why this 48-core machine was SLOWER than a
quarter of itself on a single job.
The image loop now takes a worker count of its own. It is GPU-bound, every
worker builds a private analysis engine of tens of megabytes of device and
pinned memory - and every one of those allocations implicitly synchronises the
device - so past a handful per card another worker adds setup and contention and
no throughput. Measured at about 23 ms of pure setup per extra worker, which is
why the penalty is WORSE on short runs: 200 images cost 0.86 s of loop at 12
workers and 2.25 s at 48. Capped at four per GPU, floor of eight. Every other
phase still gets the full thread count, because each one starts its own workers.
Ingest built its array with a serial push_back over every observation of every
frame - 63 million of them on the largest crystal here. Each frame's block offset
is known before anything is written, so the frames convert together, each still
written by one thread in its own order.
The pass that buckets observations by h was the single most expensive thing in a
large run - 24% of all cycles, in five instructions. It strided an array of
80-byte observations to read one 4-byte field, and its store address depended on
the loaded value, so the store buffer could not retire and the misses stopped
overlapping. The sweep that already reads every observation now copies h out as
it goes, and the bucketing walks that instead.
And the post-refine passes took the raw thread count. One of them runs 134 times
inside the rotation-scale fit, starting 48 threads each time to divide 390k terms
among them; it is gated on the work now, like everything else.
Measured on one crystal, N=48: 12.80 s -> 11.45 s, which is what 12 threads used
to cost, and on the best-matched pair the two are now level. On the heaviest
crystal ingest goes 13.5 s -> 9.9 s and the run 77.8 s -> 69.4 s. Battery 9m01s
-> 8m24s, space group 21/24, no failures.
Also restores get_gpu_numa_node() - the sysfs lookup deleted with NUMAHWPolicy -
and an opt-in CPU pin to that node behind JFJOCH_PIN_CPU_TO_GPU_NODE. It is off
because it measured neutral here: all four GPUs hang off two of the four nodes,
so pinning to them costs a worker the other half of the machine. It is kept for
boxes whose GPUs are spread over every socket. The old lookup had a latent bug -
CUDA reports the PCI id with upper-case hex and sysfs paths are lower case, so on
three of this machine's four GPUs it would have silently returned "unknown".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This is an UNSTABLE release. It includes many experimental features, as well as many AI generated fixes. We recommend using rc.152 for production use.
* jfjoch_broker: Add EXPERIMENTAL pixelrefine mode for image processing
* jfjoch_broker: Allow to load user mask from 8-bit and 16-bit TIFF files
* jfjoch_broker: Add ROI calculation in non-FPGA workflow
* jfjoch_broker: Fixes to TCP image pusher
* jfjoch_broker: Remove NUMA bindings
* jfjoch_broker: Improvements to indexing
* jfjoch_broker: For PSI EIGER, trimming energies are taken from the detector configuration (now compulsory) instead of hardcoded values
* jfjoch_writer: Save ROI definitions and the per-pixel ROI bitmap in the master file; azimuthal ROIs support phi (angular) sectors
* jfjoch_viewer: Major redesign with dockable panels and saved layouts, plus on-canvas creation/move/resize of box, circle and azimuthal ROIs
* jfjoch_viewer: Run jfjoch_process reprocessing jobs from inside the GUI and overlay per-run results
Reviewed-on: #63
leonarski_f