mirror of
https://github.com/slsdetectorgroup/aare.git
synced 2026-10-31 08:31:28 +01:00
perf: move ClusterFinderCUDA probes into perf/cf/
Make room for probes of other CUDA classes. common.py and gpu_span.py stay in perf/ and become shared; the cluster-finder driver, traced process, README and results move to perf/cf/. - common.py: assert_build_fresh() takes the header list, capture_env() takes extra fields, results_dir() takes the probe folder. - cf/cf_common.py: pedestal frames, n_sigma, rebuild-guard headers, DEVICE_PED_TYPE and the <date>_<f32|f64> results naming. - gpu_span.py: optional kernel-name filter; tolerates an engine with no ops. Output without a filter is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e969478241
commit
b7b786f423
11 files changed
+200
-140
No files matched your search
+12
-95
@@ -1,102 +1,19 @@
|
||||
# ClusterFinderCUDA nsys probes
|
||||
# GPU performance probes
|
||||
|
||||
Per-engine GPU time per frame (kernel, H2D, D2H) for the CUDA cluster finder,
|
||||
traced with Nsight Systems on real MOENCH data. The tallest engine is the
|
||||
roofline: the engines overlap and PCIe is full-duplex, so throughput is bound by
|
||||
the max, not the sum.
|
||||
Nsight Systems probes for the CUDA classes: per-engine GPU time per frame
|
||||
(kernel, H2D, D2H), duty cycles and the roofline. One folder per class, each
|
||||
with its own driver, traced process, README and `results/`:
|
||||
|
||||
These are *not* throughput numbers. nsys inflates wall clock several-fold by
|
||||
tracing every CUDA call. Use them to answer "what does the kernel cost" and
|
||||
"which engine binds", and measure sustained FPS separately without a profiler.
|
||||
| folder | class |
|
||||
|---|---|
|
||||
| [`cf/`](cf/README.md) | `ClusterFinderCUDA` |
|
||||
|
||||
## Files
|
||||
Shared by all of them:
|
||||
|
||||
| file | role |
|
||||
|---|---|
|
||||
| `run_probes.py` | driver: runs each config under nsys, exports SQLite, writes `probes.csv` |
|
||||
| `nsys_kernel_probe.py` | the traced process: pedestal, one batched pass, exit |
|
||||
| `gpu_span.py` | reads the SQLite export, computes busy time, duty cycle, roofline |
|
||||
| `common.py` | paths, guards (fresh build, idle GPU), `env.json` stamp |
|
||||
| `common.py` | paths (`AARE_REPO`, `AARE_BUILD`, `AARE_PERF_DATA`, `NSYS`), guards (fresh build, idle GPU), `env.json` stamp, results folder |
|
||||
| `gpu_span.py` | reads an nsys SQLite export: busy time, duty cycle and roofline per engine, optionally for one kernel name only |
|
||||
|
||||
## Requirements
|
||||
|
||||
- A built tree with the CUDA extension: `build/aare/_aare_cuda*.so`. Run the
|
||||
scripts with a Python of the same minor version as the build, with numpy and
|
||||
matplotlib installed (`aare/__init__.py` imports both). On pc-moench-04 the
|
||||
build is 3.11 and the `py` conda environment works:
|
||||
`/home/ferjao_k/.conda/envs/py/bin/python`.
|
||||
- Nsight Systems. Default path `/opt/nvidia/nsight-systems/2024.5.1/bin/nsys`,
|
||||
override with `NSYS=...`.
|
||||
- The MOENCH dataset under `/mnt/sls_det_storage/.../2026032408/process/xrf/`
|
||||
(override with `AARE_PERF_DATA=<dir>`). 1000 pedestal frames, 20 000 data
|
||||
frames, 5 sigma.
|
||||
- An idle GPU. The driver aborts above 5 % utilisation: a competing process
|
||||
leaves per-op averages intact but ruins duty cycles and rooflines.
|
||||
|
||||
Paths are resolved relative to this folder. `AARE_REPO` and `AARE_BUILD`
|
||||
override the repo root and build directory.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd python/tests/perf
|
||||
python run_probes.py # full sweep: 4 configs, 20k frames each (~5 min)
|
||||
python run_probes.py --frames 2000 # smoke test only; clocks do not ramp, kernel reads ~7 % slow
|
||||
python run_probes.py --only 3x3_s1_uncontended 9x9_s1_uncontended --tag mychange
|
||||
```
|
||||
|
||||
Configs, matching the campaign in `docs/ClusterFinderCUDA_benchmark_results.md`
|
||||
on branch `bench/cuda_cf`:
|
||||
|
||||
| label | cluster | cap | streams | batch | frames |
|
||||
|---|---|---|---|---|---|
|
||||
| `3x3_s4` | 3x3 | 3000 | 4 | 2000 | 20 000 |
|
||||
| `3x3_s1_uncontended` | 3x3 | 3000 | 1 | 2000 | 20 000 |
|
||||
| `9x9_s4` | 9x9 | 1700 | 4 | 2000 | 20 000 |
|
||||
| `9x9_s1_uncontended` | 9x9 | 1700 | 1 | 2000 | 20 000 |
|
||||
|
||||
Keep the cap: at 9x9 the D2H slot is `4 + cap * sizeof(cluster)` and is copied
|
||||
whole, so the cap sets the D2H bar.
|
||||
|
||||
Output goes to `results/<date>_<f32|f64>[_tag]/`: `env.json` (git rev, dirty
|
||||
flag, driver, pedestal type), one `.nsys-rep` + `.sqlite` per config, and
|
||||
`probes.csv`. Re-running one config replaces its own row and keeps the others.
|
||||
|
||||
To inspect one trace by hand:
|
||||
|
||||
```bash
|
||||
nsys stats --report cuda_gpu_sum results/<dir>/probe_9x9_s1_uncontended_cap1700.nsys-rep
|
||||
python gpu_span.py results/<dir>/probe_9x9_s1_uncontended_cap1700.sqlite 20000
|
||||
```
|
||||
|
||||
`probes.csv` and `env.json` are small and can be committed with a result;
|
||||
the `.nsys-rep` and `.sqlite` traces are tens of MB each and are gitignored.
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
- **Quote kernel changes from the `s1_uncontended` rows.** With one stream
|
||||
nothing overlaps, so the kernel column is the kernel alone. The 1-stream
|
||||
kernel and transfer numbers reproduce across sessions to about 1 %.
|
||||
- **4-stream transfer columns drift by up to 20 % between sessions** on an
|
||||
identical build (H2D and D2H fight for the PCIe link). Only compare s4
|
||||
rows taken in the same session, and read them for the bottleneck, not for
|
||||
a percentage.
|
||||
- A kernel below the tallest transfer bar is invisible end to end. Further
|
||||
kernel work only pays once it is the tallest bar.
|
||||
|
||||
## Reference, 2026-09-10, RTX 4090, driver 595.71.05, float pedestal
|
||||
|
||||
Kernel us/frame. "before" is commit 7177f00 (pre-refactor kernel, rebuilt and
|
||||
re-probed the same day, matching its August numbers): `results/2026-09-10_f32_ref7177/`.
|
||||
"after" is the kernel with the cooperative tile load, size-dependent unroll
|
||||
and two-phase load: `results/2026-09-10_f32/`.
|
||||
|
||||
| config | before | after | roofline after | binds |
|
||||
|---|--:|--:|--:|---|
|
||||
| 3x3, 1 stream | 4.32 | 4.65 | 13.13 | H2D |
|
||||
| 3x3, 4 streams | 5.00 | 5.64 | 16.35 | H2D |
|
||||
| 9x9, 1 stream | 23.80 | 15.29 | 22.00 | D2H |
|
||||
| 9x9, 4 streams | 24.72 | 15.59 | 25.39 | D2H |
|
||||
|
||||
Both sizes are transfer-bound; sustained throughput is unchanged (3x3 about
|
||||
61 400 FPS, 9x9 about 39 600 FPS on the zero-copy path).
|
||||
Requirements (a CUDA build of the Python extension, Nsight Systems, the
|
||||
dataset, an idle GPU) are listed in each folder's README.
|
||||
@@ -0,0 +1,103 @@
|
||||
# ClusterFinderCUDA nsys probes
|
||||
|
||||
Per-engine GPU time per frame (kernel, H2D, D2H) for the CUDA cluster finder,
|
||||
traced with Nsight Systems on real MOENCH data. The tallest engine is the
|
||||
roofline: the engines overlap and PCIe is full-duplex, so throughput is bound by
|
||||
the max, not the sum.
|
||||
|
||||
These are *not* throughput numbers. nsys inflates wall clock several-fold by
|
||||
tracing every CUDA call. Use them to answer "what does the kernel cost" and
|
||||
"which engine binds", and measure sustained FPS separately without a profiler.
|
||||
|
||||
## Files
|
||||
|
||||
| file | role |
|
||||
|---|---|
|
||||
| `run_probes.py` | driver: runs each config under nsys, exports SQLite, writes `probes.csv` |
|
||||
| `nsys_kernel_probe.py` | the traced process: pedestal, one batched pass, exit |
|
||||
| `cf_common.py` | cluster-finder settings: pedestal frames, n_sigma, headers for the rebuild guard, `DEVICE_PED_TYPE` |
|
||||
| `../gpu_span.py` | shared: reads the SQLite export, computes busy time, duty cycle, roofline |
|
||||
| `../common.py` | shared: paths, guards (fresh build, idle GPU), `env.json` stamp |
|
||||
|
||||
## Requirements
|
||||
|
||||
- A built tree with the CUDA extension: `build/aare/_aare_cuda*.so`. Run the
|
||||
scripts with a Python of the same minor version as the build, with numpy and
|
||||
matplotlib installed (`aare/__init__.py` imports both). On pc-moench-04 the
|
||||
build is 3.11 and the `py` conda environment works:
|
||||
`/home/ferjao_k/.conda/envs/py/bin/python`.
|
||||
- Nsight Systems. Default path `/opt/nvidia/nsight-systems/2024.5.1/bin/nsys`,
|
||||
override with `NSYS=...`.
|
||||
- The MOENCH dataset under `/mnt/sls_det_storage/.../2026032408/process/xrf/`
|
||||
(override with `AARE_PERF_DATA=<dir>`). 1000 pedestal frames, 20 000 data
|
||||
frames, 5 sigma.
|
||||
- An idle GPU. The driver aborts above 5 % utilisation: a competing process
|
||||
leaves per-op averages intact but ruins duty cycles and rooflines.
|
||||
|
||||
Paths are resolved relative to `python/tests/perf/`. `AARE_REPO` and `AARE_BUILD`
|
||||
override the repo root and build directory.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
cd python/tests/perf/cf
|
||||
python run_probes.py # full sweep: 4 configs, 20k frames each (~5 min)
|
||||
python run_probes.py --frames 2000 # smoke test only; clocks do not ramp, kernel reads ~7 % slow
|
||||
python run_probes.py --only 3x3_s1_uncontended 9x9_s1_uncontended --tag mychange
|
||||
```
|
||||
|
||||
Configs, matching the campaign in `docs/ClusterFinderCUDA_benchmark_results.md`
|
||||
on branch `bench/cuda_cf`:
|
||||
|
||||
| label | cluster | cap | streams | batch | frames |
|
||||
|---|---|---|---|---|---|
|
||||
| `3x3_s4` | 3x3 | 3000 | 4 | 2000 | 20 000 |
|
||||
| `3x3_s1_uncontended` | 3x3 | 3000 | 1 | 2000 | 20 000 |
|
||||
| `9x9_s4` | 9x9 | 1700 | 4 | 2000 | 20 000 |
|
||||
| `9x9_s1_uncontended` | 9x9 | 1700 | 1 | 2000 | 20 000 |
|
||||
|
||||
Keep the cap: at 9x9 the D2H slot is `4 + cap * sizeof(cluster)` and is copied
|
||||
whole, so the cap sets the D2H bar.
|
||||
|
||||
Output goes to `results/<date>_<f32|f64>[_tag]/`: `env.json` (git rev, dirty
|
||||
flag, driver, pedestal type), one `.nsys-rep` + `.sqlite` per config, and
|
||||
`probes.csv`. Re-running one config replaces its own row and keeps the others.
|
||||
|
||||
To inspect one trace by hand:
|
||||
|
||||
```bash
|
||||
nsys stats --report cuda_gpu_sum results/<dir>/probe_9x9_s1_uncontended_cap1700.nsys-rep
|
||||
python ../gpu_span.py results/<dir>/probe_9x9_s1_uncontended_cap1700.sqlite 20000
|
||||
```
|
||||
|
||||
`probes.csv` and `env.json` are small and can be committed with a result;
|
||||
the `.nsys-rep` and `.sqlite` traces are tens of MB each and are gitignored.
|
||||
|
||||
## Reading the numbers
|
||||
|
||||
- **Quote kernel changes from the `s1_uncontended` rows.** With one stream
|
||||
nothing overlaps, so the kernel column is the kernel alone. The 1-stream
|
||||
kernel and transfer numbers reproduce across sessions to about 1 %.
|
||||
- **4-stream transfer columns drift by up to 20 % between sessions** on an
|
||||
identical build (H2D and D2H fight for the PCIe link). Only compare s4
|
||||
rows taken in the same session, and read them for the bottleneck, not for
|
||||
a percentage.
|
||||
- A kernel below the tallest transfer bar is invisible end to end. Further
|
||||
kernel work only pays once it is the tallest bar.
|
||||
|
||||
## Reference, 2026-09-10, RTX 4090, driver 595.71.05, float pedestal
|
||||
|
||||
Kernel us/frame. "before" is commit 7177f00 (pre-refactor kernel, rebuilt and
|
||||
re-probed the same day, matching its August numbers): `results/2026-09-10_f32_ref7177/`.
|
||||
"after" is the kernel with the cooperative tile load, size-dependent unroll
|
||||
and two-phase load: `results/2026-09-10_f32/`.
|
||||
|
||||
| config | before | after | roofline after | binds |
|
||||
|---|--:|--:|--:|---|
|
||||
| 3x3, 1 stream | 4.32 | 4.65 | 13.13 | H2D |
|
||||
| 3x3, 4 streams | 5.00 | 5.64 | 16.35 | H2D |
|
||||
| 9x9, 1 stream | 23.80 | 15.29 | 22.00 | D2H |
|
||||
| 9x9, 4 streams | 24.72 | 15.59 | 25.39 | D2H |
|
||||
|
||||
Both sizes are transfer-bound; sustained throughput is unchanged (3x3 about
|
||||
61 400 FPS, 9x9 about 39 600 FPS on the zero-copy path).
|
||||
@@ -0,0 +1,35 @@
|
||||
"""ClusterFinderCUDA-specific settings for the probes in this folder.
|
||||
|
||||
Importing this module also makes ../common.py and ../gpu_span.py importable.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE.parent))
|
||||
import common # noqa: E402
|
||||
|
||||
N_PEDESTAL_FRAMES = 1000
|
||||
N_SIGMA = 5
|
||||
|
||||
# Rebuild-guard: the extension must be newer than all of these.
|
||||
HEADERS = ["include/aare/cf/clusterfinder_kernel.cuh",
|
||||
"include/aare/cf/clusterfinder_algo.cuh",
|
||||
"include/aare/cf/ClusterFinderCUDA.hpp"]
|
||||
|
||||
|
||||
def device_ped_type() -> str:
|
||||
"""DEVICE_PED_TYPE from the kernel header (the float/double build axis)."""
|
||||
hdr = common.REPO / "include/aare/cf/clusterfinder_kernel.cuh"
|
||||
for line in hdr.read_text().splitlines()[:40]:
|
||||
if "using DEVICE_PED_TYPE" in line and not line.strip().startswith("//"):
|
||||
return line.split("=")[1].split(";")[0].strip()
|
||||
return "unknown"
|
||||
|
||||
|
||||
def results_dir(tag: str = "") -> Path:
|
||||
"""cf/results/<date>_<f32|f64>[_tag]/"""
|
||||
short = {"float": "f32", "double": "f64"}.get(device_ped_type(), "unk")
|
||||
return common.results_dir(HERE, short, tag)
|
||||
@@ -10,7 +10,8 @@ import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import common # noqa: E402 (puts the build dir on sys.path)
|
||||
import cf_common # noqa: E402 (puts ../ and the build dir on sys.path)
|
||||
from cf_common import common # noqa: E402
|
||||
|
||||
from aare import File, ClusterFinderCUDA # noqa: E402
|
||||
|
||||
@@ -20,9 +21,9 @@ f = File(common.DATA_FILE)
|
||||
pd = File(common.PEDESTAL_FILE)
|
||||
|
||||
cf = ClusterFinderCUDA((f.rows, f.cols), cluster_size=(cdim, cdim),
|
||||
n_sigma=common.N_SIGMA, max_clusters_per_frame=cap,
|
||||
n_sigma=cf_common.N_SIGMA, max_clusters_per_frame=cap,
|
||||
n_streams=n_streams)
|
||||
for _ in range(common.N_PEDESTAL_FRAMES):
|
||||
for _ in range(cf_common.N_PEDESTAL_FRAMES):
|
||||
cf.push_pedestal_frame(pd.read_frame().copy())
|
||||
|
||||
data = f.read_n(N)
|
||||
File renamed without changes.
File renamed without changes.
|
500 Internal Server Error
An error occurred: An error occurred Gitea Version: 28.2.0 |