jungfrauandClaude Opus 5
088ba1cff8
Build Packages / build:viewer-tgz:cpu (push) Successful in 19m27s
Build Packages / build:windows:nocuda (push) Successful in 20m0s
Build Packages / build:viewer-tgz:cuda (push) Successful in 21m29s
Build Packages / build:rpm (rocky8) (push) Failing after 17s
Build Packages / build:rpm (ubuntu2404_nocuda) (push) Successful in 23m38s
Build Packages / build:rpm (rocky9_nocuda) (push) Successful in 24m41s
Build Packages / build:rpm (ubuntu2204_nocuda) (push) Successful in 29m28s
Build Packages / build:rpm (rocky8_nocuda) (push) Successful in 29m40s
Build Packages / build:rpm (rocky8_sls9) (push) Successful in 30m14s
Build Packages / build:rpm (rocky9_sls9) (push) Successful in 20m32s
Build Packages / XDS test (durin plugin) (push) Successful in 11m57s
Build Packages / build:windows:cuda (push) Successful in 21m56s
Build Packages / Generate python client (push) Successful in 31s
Build Packages / XDS test (JFJoch plugin) (push) Successful in 12m34s
Build Packages / Create release (push) Skipped
Build Packages / build:rpm (rocky9) (push) Successful in 21m12s
Build Packages / Build documentation (push) Successful in 57s
Build Packages / build:rpm (ubuntu2404) (push) Successful in 19m21s
Build Packages / build:rpm (ubuntu2204) (push) Successful in 22m0s
Build Packages / DIALS test (push) Successful in 17m22s
Build Packages / XDS test (neggia plugin) (push) Successful in 7m53s
Build Packages / Unit tests (push) Successful in 1h17m43s
Four changes, all from measuring why this 48-core machine was SLOWER than a quarter of itself on a single job. The image loop now takes a worker count of its own. It is GPU-bound, every worker builds a private analysis engine of tens of megabytes of device and pinned memory - and every one of those allocations implicitly synchronises the device - so past a handful per card another worker adds setup and contention and no throughput. Measured at about 23 ms of pure setup per extra worker, which is why the penalty is WORSE on short runs: 200 images cost 0.86 s of loop at 12 workers and 2.25 s at 48. Capped at four per GPU, floor of eight. Every other phase still gets the full thread count, because each one starts its own workers. Ingest built its array with a serial push_back over every observation of every frame - 63 million of them on the largest crystal here. Each frame's block offset is known before anything is written, so the frames convert together, each still written by one thread in its own order. The pass that buckets observations by h was the single most expensive thing in a large run - 24% of all cycles, in five instructions. It strided an array of 80-byte observations to read one 4-byte field, and its store address depended on the loaded value, so the store buffer could not retire and the misses stopped overlapping. The sweep that already reads every observation now copies h out as it goes, and the bucketing walks that instead. And the post-refine passes took the raw thread count. One of them runs 134 times inside the rotation-scale fit, starting 48 threads each time to divide 390k terms among them; it is gated on the work now, like everything else. Measured on one crystal, N=48: 12.80 s -> 11.45 s, which is what 12 threads used to cost, and on the best-matched pair the two are now level. On the heaviest crystal ingest goes 13.5 s -> 9.9 s and the run 77.8 s -> 69.4 s. Battery 9m01s -> 8m24s, space group 21/24, no failures. Also restores get_gpu_numa_node() - the sysfs lookup deleted with NUMAHWPolicy - and an opt-in CPU pin to that node behind JFJOCH_PIN_CPU_TO_GPU_NODE. It is off because it measured neutral here: all four GPUs hang off two of the four nodes, so pinning to them costs a worker the other half of the machine. It is kept for boxes whose GPUs are spread over every socket. The old lookup had a latent bug - CUDA reports the PCI id with upper-case hex and sysfs paths are lower case, so on three of this machine's four GPUs it would have silently returned "unknown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
24 lines
1.0 KiB
C++
24 lines
1.0 KiB
C++
// SPDX-FileCopyrightText: 2024 Filip Leonarski, Paul Scherrer Institute <filip.leonarski@psi.ch>
|
|
// SPDX-License-Identifier: GPL-3.0-only
|
|
|
|
#pragma once
|
|
|
|
#include <cstdint>
|
|
|
|
int32_t get_gpu_count();
|
|
void set_gpu(int32_t dev_id);
|
|
|
|
// NUMA node the given GPU is attached to, read from sysfs; -1 when unknown, when there is no such
|
|
// device, or on a platform where the question does not apply.
|
|
int get_gpu_numa_node(int32_t dev_id);
|
|
|
|
// Pin the calling thread to the next GPU in round-robin order, using a process-wide counter
|
|
// (counter++ % get_gpu_count()). Call once per thread; no thread id needed. No-op when no GPU
|
|
// is visible. Honours CUDA_VISIBLE_DEVICES via get_gpu_count().
|
|
//
|
|
// With JFJOCH_PIN_CPU_TO_GPU_NODE set in the environment it also confines the thread to the cores of
|
|
// that GPU's NUMA node, so the memory it allocates lands beside the GPU it feeds. Off by default -
|
|
// it also takes the other sockets' cores away from the thread, and which way that trade goes depends
|
|
// on the machine.
|
|
void pin_gpu();
|