perf: move ClusterFinderCUDA probes into perf/cf/
Build on RHEL8 / build (push) Successful in 4m1s
Build on RHEL9 / build (push) Successful in 4m15s
Run tests using data on local RHEL8 / build (push) Failing after 4m56s

Make room for probes of other CUDA classes. common.py and gpu_span.py
stay in perf/ and become shared; the cluster-finder driver, traced
process, README and results move to perf/cf/.

- common.py: assert_build_fresh() takes the header list, capture_env()
  takes extra fields, results_dir() takes the probe folder.
- cf/cf_common.py: pedestal frames, n_sigma, rebuild-guard headers,
  DEVICE_PED_TYPE and the <date>_<f32|f64> results naming.
- gpu_span.py: optional kernel-name filter; tolerates an engine with no
  ops. Output without a filter is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kferjaouiandClaude Opus 5.5 committed 2026-10-27 12:22:18 +01:00
1 parent e969478241
commit b7b786f423
11 files changed
+200 -140

No files matched your search

+12 -95
View File
@@ -1,102 +1,19 @@
# ClusterFinderCUDA nsys probes
# GPU performance probes
Per-engine GPU time per frame (kernel, H2D, D2H) for the CUDA cluster finder,
traced with Nsight Systems on real MOENCH data. The tallest engine is the
roofline: the engines overlap and PCIe is full-duplex, so throughput is bound by
the max, not the sum.
Nsight Systems probes for the CUDA classes: per-engine GPU time per frame
(kernel, H2D, D2H), duty cycles and the roofline. One folder per class, each
with its own driver, traced process, README and `results/`:
These are *not* throughput numbers. nsys inflates wall clock several-fold by
tracing every CUDA call. Use them to answer "what does the kernel cost" and
"which engine binds", and measure sustained FPS separately without a profiler.
| folder | class |
|---|---|
| [`cf/`](cf/README.md) | `ClusterFinderCUDA` |
## Files
Shared by all of them:
| file | role |
|---|---|
| `run_probes.py` | driver: runs each config under nsys, exports SQLite, writes `probes.csv` |
| `nsys_kernel_probe.py` | the traced process: pedestal, one batched pass, exit |
| `gpu_span.py` | reads the SQLite export, computes busy time, duty cycle, roofline |
| `common.py` | paths, guards (fresh build, idle GPU), `env.json` stamp |
| `common.py` | paths (`AARE_REPO`, `AARE_BUILD`, `AARE_PERF_DATA`, `NSYS`), guards (fresh build, idle GPU), `env.json` stamp, results folder |
| `gpu_span.py` | reads an nsys SQLite export: busy time, duty cycle and roofline per engine, optionally for one kernel name only |
## Requirements
- A built tree with the CUDA extension: `build/aare/_aare_cuda*.so`. Run the
scripts with a Python of the same minor version as the build, with numpy and
matplotlib installed (`aare/__init__.py` imports both). On pc-moench-04 the
build is 3.11 and the `py` conda environment works:
`/home/ferjao_k/.conda/envs/py/bin/python`.
- Nsight Systems. Default path `/opt/nvidia/nsight-systems/2024.5.1/bin/nsys`,
override with `NSYS=...`.
- The MOENCH dataset under `/mnt/sls_det_storage/.../2026032408/process/xrf/`
(override with `AARE_PERF_DATA=<dir>`). 1000 pedestal frames, 20 000 data
frames, 5 sigma.
- An idle GPU. The driver aborts above 5 % utilisation: a competing process
leaves per-op averages intact but ruins duty cycles and rooflines.
Paths are resolved relative to this folder. `AARE_REPO` and `AARE_BUILD`
override the repo root and build directory.
## Run
```bash
cd python/tests/perf
python run_probes.py # full sweep: 4 configs, 20k frames each (~5 min)
python run_probes.py --frames 2000 # smoke test only; clocks do not ramp, kernel reads ~7 % slow
python run_probes.py --only 3x3_s1_uncontended 9x9_s1_uncontended --tag mychange
```
Configs, matching the campaign in `docs/ClusterFinderCUDA_benchmark_results.md`
on branch `bench/cuda_cf`:
| label | cluster | cap | streams | batch | frames |
|---|---|---|---|---|---|
| `3x3_s4` | 3x3 | 3000 | 4 | 2000 | 20 000 |
| `3x3_s1_uncontended` | 3x3 | 3000 | 1 | 2000 | 20 000 |
| `9x9_s4` | 9x9 | 1700 | 4 | 2000 | 20 000 |
| `9x9_s1_uncontended` | 9x9 | 1700 | 1 | 2000 | 20 000 |
Keep the cap: at 9x9 the D2H slot is `4 + cap * sizeof(cluster)` and is copied
whole, so the cap sets the D2H bar.
Output goes to `results/<date>_<f32|f64>[_tag]/`: `env.json` (git rev, dirty
flag, driver, pedestal type), one `.nsys-rep` + `.sqlite` per config, and
`probes.csv`. Re-running one config replaces its own row and keeps the others.
To inspect one trace by hand:
```bash
nsys stats --report cuda_gpu_sum results/<dir>/probe_9x9_s1_uncontended_cap1700.nsys-rep
python gpu_span.py results/<dir>/probe_9x9_s1_uncontended_cap1700.sqlite 20000
```
`probes.csv` and `env.json` are small and can be committed with a result;
the `.nsys-rep` and `.sqlite` traces are tens of MB each and are gitignored.
## Reading the numbers
- **Quote kernel changes from the `s1_uncontended` rows.** With one stream
nothing overlaps, so the kernel column is the kernel alone. The 1-stream
kernel and transfer numbers reproduce across sessions to about 1 %.
- **4-stream transfer columns drift by up to 20 % between sessions** on an
identical build (H2D and D2H fight for the PCIe link). Only compare s4
rows taken in the same session, and read them for the bottleneck, not for
a percentage.
- A kernel below the tallest transfer bar is invisible end to end. Further
kernel work only pays once it is the tallest bar.
## Reference, 2026-09-10, RTX 4090, driver 595.71.05, float pedestal
Kernel us/frame. "before" is commit 7177f00 (pre-refactor kernel, rebuilt and
re-probed the same day, matching its August numbers): `results/2026-09-10_f32_ref7177/`.
"after" is the kernel with the cooperative tile load, size-dependent unroll
and two-phase load: `results/2026-09-10_f32/`.
| config | before | after | roofline after | binds |
|---|--:|--:|--:|---|
| 3x3, 1 stream | 4.32 | 4.65 | 13.13 | H2D |
| 3x3, 4 streams | 5.00 | 5.64 | 16.35 | H2D |
| 9x9, 1 stream | 23.80 | 15.29 | 22.00 | D2H |
| 9x9, 4 streams | 24.72 | 15.59 | 25.39 | D2H |
Both sizes are transfer-bound; sustained throughput is unchanged (3x3 about
61 400 FPS, 9x9 about 39 600 FPS on the zero-copy path).
Requirements (a CUDA build of the Python extension, Nsight Systems, the
dataset, an idle GPU) are listed in each folder's README.
+103
View File
@@ -0,0 +1,103 @@
# ClusterFinderCUDA nsys probes
Per-engine GPU time per frame (kernel, H2D, D2H) for the CUDA cluster finder,
traced with Nsight Systems on real MOENCH data. The tallest engine is the
roofline: the engines overlap and PCIe is full-duplex, so throughput is bound by
the max, not the sum.
These are *not* throughput numbers. nsys inflates wall clock several-fold by
tracing every CUDA call. Use them to answer "what does the kernel cost" and
"which engine binds", and measure sustained FPS separately without a profiler.
## Files
| file | role |
|---|---|
| `run_probes.py` | driver: runs each config under nsys, exports SQLite, writes `probes.csv` |
| `nsys_kernel_probe.py` | the traced process: pedestal, one batched pass, exit |
| `cf_common.py` | cluster-finder settings: pedestal frames, n_sigma, headers for the rebuild guard, `DEVICE_PED_TYPE` |
| `../gpu_span.py` | shared: reads the SQLite export, computes busy time, duty cycle, roofline |
| `../common.py` | shared: paths, guards (fresh build, idle GPU), `env.json` stamp |
## Requirements
- A built tree with the CUDA extension: `build/aare/_aare_cuda*.so`. Run the
scripts with a Python of the same minor version as the build, with numpy and
matplotlib installed (`aare/__init__.py` imports both). On pc-moench-04 the
build is 3.11 and the `py` conda environment works:
`/home/ferjao_k/.conda/envs/py/bin/python`.
- Nsight Systems. Default path `/opt/nvidia/nsight-systems/2024.5.1/bin/nsys`,
override with `NSYS=...`.
- The MOENCH dataset under `/mnt/sls_det_storage/.../2026032408/process/xrf/`
(override with `AARE_PERF_DATA=<dir>`). 1000 pedestal frames, 20 000 data
frames, 5 sigma.
- An idle GPU. The driver aborts above 5 % utilisation: a competing process
leaves per-op averages intact but ruins duty cycles and rooflines.
Paths are resolved relative to `python/tests/perf/`. `AARE_REPO` and `AARE_BUILD`
override the repo root and build directory.
## Run
```bash
cd python/tests/perf/cf
python run_probes.py # full sweep: 4 configs, 20k frames each (~5 min)
python run_probes.py --frames 2000 # smoke test only; clocks do not ramp, kernel reads ~7 % slow
python run_probes.py --only 3x3_s1_uncontended 9x9_s1_uncontended --tag mychange
```
Configs, matching the campaign in `docs/ClusterFinderCUDA_benchmark_results.md`
on branch `bench/cuda_cf`:
| label | cluster | cap | streams | batch | frames |
|---|---|---|---|---|---|
| `3x3_s4` | 3x3 | 3000 | 4 | 2000 | 20 000 |
| `3x3_s1_uncontended` | 3x3 | 3000 | 1 | 2000 | 20 000 |
| `9x9_s4` | 9x9 | 1700 | 4 | 2000 | 20 000 |
| `9x9_s1_uncontended` | 9x9 | 1700 | 1 | 2000 | 20 000 |
Keep the cap: at 9x9 the D2H slot is `4 + cap * sizeof(cluster)` and is copied
whole, so the cap sets the D2H bar.
Output goes to `results/<date>_<f32|f64>[_tag]/`: `env.json` (git rev, dirty
flag, driver, pedestal type), one `.nsys-rep` + `.sqlite` per config, and
`probes.csv`. Re-running one config replaces its own row and keeps the others.
To inspect one trace by hand:
```bash
nsys stats --report cuda_gpu_sum results/<dir>/probe_9x9_s1_uncontended_cap1700.nsys-rep
python ../gpu_span.py results/<dir>/probe_9x9_s1_uncontended_cap1700.sqlite 20000
```
`probes.csv` and `env.json` are small and can be committed with a result;
the `.nsys-rep` and `.sqlite` traces are tens of MB each and are gitignored.
## Reading the numbers
- **Quote kernel changes from the `s1_uncontended` rows.** With one stream
nothing overlaps, so the kernel column is the kernel alone. The 1-stream
kernel and transfer numbers reproduce across sessions to about 1 %.
- **4-stream transfer columns drift by up to 20 % between sessions** on an
identical build (H2D and D2H fight for the PCIe link). Only compare s4
rows taken in the same session, and read them for the bottleneck, not for
a percentage.
- A kernel below the tallest transfer bar is invisible end to end. Further
kernel work only pays once it is the tallest bar.
## Reference, 2026-09-10, RTX 4090, driver 595.71.05, float pedestal
Kernel us/frame. "before" is commit 7177f00 (pre-refactor kernel, rebuilt and
re-probed the same day, matching its August numbers): `results/2026-09-10_f32_ref7177/`.
"after" is the kernel with the cooperative tile load, size-dependent unroll
and two-phase load: `results/2026-09-10_f32/`.
| config | before | after | roofline after | binds |
|---|--:|--:|--:|---|
| 3x3, 1 stream | 4.32 | 4.65 | 13.13 | H2D |
| 3x3, 4 streams | 5.00 | 5.64 | 16.35 | H2D |
| 9x9, 1 stream | 23.80 | 15.29 | 22.00 | D2H |
| 9x9, 4 streams | 24.72 | 15.59 | 25.39 | D2H |
Both sizes are transfer-bound; sustained throughput is unchanged (3x3 about
61 400 FPS, 9x9 about 39 600 FPS on the zero-copy path).
+35
View File
@@ -0,0 +1,35 @@
"""ClusterFinderCUDA-specific settings for the probes in this folder.
Importing this module also makes ../common.py and ../gpu_span.py importable.
"""
from __future__ import annotations
import sys
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
import common # noqa: E402
N_PEDESTAL_FRAMES = 1000
N_SIGMA = 5
# Rebuild-guard: the extension must be newer than all of these.
HEADERS = ["include/aare/cf/clusterfinder_kernel.cuh",
"include/aare/cf/clusterfinder_algo.cuh",
"include/aare/cf/ClusterFinderCUDA.hpp"]
def device_ped_type() -> str:
"""DEVICE_PED_TYPE from the kernel header (the float/double build axis)."""
hdr = common.REPO / "include/aare/cf/clusterfinder_kernel.cuh"
for line in hdr.read_text().splitlines()[:40]:
if "using DEVICE_PED_TYPE" in line and not line.strip().startswith("//"):
return line.split("=")[1].split(";")[0].strip()
return "unknown"
def results_dir(tag: str = "") -> Path:
"""cf/results/<date>_<f32|f64>[_tag]/"""
short = {"float": "f32", "double": "f64"}.get(device_ped_type(), "unk")
return common.results_dir(HERE, short, tag)
@@ -10,7 +10,8 @@ import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import common # noqa: E402 (puts the build dir on sys.path)
import cf_common # noqa: E402 (puts ../ and the build dir on sys.path)
from cf_common import common # noqa: E402
from aare import File, ClusterFinderCUDA # noqa: E402
@@ -20,9 +21,9 @@ f = File(common.DATA_FILE)
pd = File(common.PEDESTAL_FILE)
cf = ClusterFinderCUDA((f.rows, f.cols), cluster_size=(cdim, cdim),
n_sigma=common.N_SIGMA, max_clusters_per_frame=cap,
n_sigma=cf_common.N_SIGMA, max_clusters_per_frame=cap,
n_streams=n_streams)
for _ in range(common.N_PEDESTAL_FRAMES):
for _ in range(cf_common.N_PEDESTAL_FRAMES):
cf.push_pedestal_frame(pd.read_frame().copy())
data = f.read_n(N)
File renamed without changes.
Internal Server Error - PSI GIT Service
500 Internal Server Error

An error occurred:

An error occurred

Gitea Version: 28.2.0