mirror of
https://github.com/slsdetectorgroup/aare.git
synced 2026-09-03 22:40:42 +02:00
174 lines
8.7 KiB
Markdown
174 lines
8.7 KiB
Markdown
# ClusterFinderCUDA measurement campaign
|
||
|
||
Everything that produces a number in
|
||
[`docs/ClusterFinderCUDA_benchmark_results.md`](../../../docs/ClusterFinderCUDA_benchmark_results.md)
|
||
lives here, and everything it produces lands in `results/<date>_<build>/` with a
|
||
manifest, so any figure in the report or the deck can be traced back to the run
|
||
and the build that made it.
|
||
|
||
## Scripts
|
||
|
||
| script | produces | answers |
|
||
|---|---|---|
|
||
| `run_campaign.sh` | both arms | the whole thing: f32 ladder+probes → rebuild → f64 ladder+probes → rebuild back |
|
||
| `run_ladder.py` | `ladder_<dim>x<dim>.csv` | end-to-end wall / FPS / faults / counts for every step |
|
||
| `run_probes.py` | `probes.csv` + `.nsys-rep`/`.sqlite` | per-engine GPU times, duty cycles, **the rooflines** |
|
||
| `ladder.py` | — | the ladder *as a config matrix*; imported, not run |
|
||
| `common.py` | — | dataset, pedestal, fault bracketing, env capture, CSV format |
|
||
| `nsys_kernel_probe.py` | — | the workload `run_probes.py` traces; runnable alone under nsys |
|
||
| `gpu_span.py` | a duty-cycle table | *engine busy* vs *engine idle*; `analyze()` is importable |
|
||
|
||
## Running it
|
||
|
||
```bash
|
||
./run_campaign.sh # both arms, ~1.5 h, unattended
|
||
./run_campaign.sh f32 # one arm
|
||
|
||
python run_ladder.py --dry-run # 2000 frames, 1 rep — proves it executes
|
||
python run_ladder.py # 3x3 @ 100k, 9x9 @ 20k, 5 reps
|
||
python run_ladder.py --dims 9 --steps opt7 opt8 # harness labels — see the map below
|
||
python run_ladder.py --retain # keep every ClusterVector (notebook behaviour)
|
||
|
||
python run_probes.py # 4 configs at 20k frames
|
||
python run_probes.py --only 9x9_s4
|
||
python gpu_span.py <rep>.sqlite 20000 # re-read duty cycles from an existing profile
|
||
|
||
# per-operation durations from a committed profile — no GPU, no rerun.
|
||
# Pass the .sqlite: the reps are newer than their exports, so the .nsys-rep path
|
||
# demands --force-export=true and rewrites the export gpu_span.py reads.
|
||
nsys stats --report cuda_gpu_kern_sum --report cuda_gpu_mem_time_sum \
|
||
--format table results/2026-08-18_f64/probe_9x9_s1_uncontended.sqlite
|
||
```
|
||
|
||
`nsys stats` gives durations, not duty cycles — it cannot tell "engine busy" from
|
||
"engine idle waiting". Use it for *how long is one kernel* (`s1`); use `gpu_span.py`
|
||
for overlap, duty and the roofline.
|
||
|
||
**The GPU must be idle.** Both drivers abort above 5 % utilisation: a competing
|
||
process leaves per-operation averages intact while destroying the duty cycle and
|
||
the wall clock, so the failure is silent if you don't check. Close any notebook
|
||
first. `--allow-busy-gpu` exists but the numbers are not quotable.
|
||
|
||
## Step labels
|
||
|
||
The `--steps` flag and the `step` column of every CSV use the harness's internal labels,
|
||
which predate the report's numbering. The report carries the same map in its §15.
|
||
|
||
| harness label | step in the report | act |
|
||
|---|---|---|
|
||
| `cpu` | baseline | — |
|
||
| `opt1` `opt2` `opt3` `opt4` | opt1 opt2 opt3 opt4 | **I** — feeding the GPU |
|
||
| `opt5` | **route A** — CUDA Graphs, *rejected* | (fork after opt4) |
|
||
| `opt7` | **opt5** — chunked host↔GPU overlap | **II** — getting results back |
|
||
| `opt8` | **opt6** — zero-copy `collect_view()` | **II** |
|
||
| *(build axis, not a row)* | **opt7** — f32 device pedestal | **III** — the kernel |
|
||
|
||
Renaming the labels would rewrite recorded data, so they stay as they are; translate on
|
||
the way out.
|
||
|
||
## Fixed parameters
|
||
|
||
| | 3×3 | 9×9 |
|
||
|---|--:|--:|
|
||
| `N` | 100 000 | 20 000 |
|
||
| `max_clusters_per_frame` | 3 000 | 1 500 |
|
||
| `n_streams` | 4 | 4 |
|
||
| `BATCH_SIZE` | 2 000 | 2 000 |
|
||
| pedestal frames / `n_sigma` | 1 000 / 5 | 1 000 / 5 |
|
||
| reps | 5 | 5 |
|
||
| nsys probe frames | 20 000 | 20 000 |
|
||
|
||
Optimising over these is a separate exercise; what matters here is that every
|
||
step sees the same ones. Two are deliberate departures from earlier campaigns:
|
||
`n_streams` was 8 at 9×9 (8 streams buy no kernel concurrency there — instance
|
||
time +1 % — while inflating the event timer 3.5×), and probes were 2 000 frames
|
||
(too short for the clocks to ramp, which is how a 26.7 µs roofline was published
|
||
for a pipeline that sustains 24.25).
|
||
|
||
9×9 is held at N=20 000 because its result heap is ~5× larger per frame
|
||
(1422 × 328 B = 466 kB vs 2330 × 40 B = 93 kB); 100 k would need 46.6 GB to
|
||
retain against 98 GB free with no swap.
|
||
|
||
## Reading the output
|
||
|
||
**`cold` = rep 0 in a fresh process. `warm` = best of the remaining reps.**
|
||
Not the last rep — `collect()` does not converge, it oscillates between allocator
|
||
states. Measured at 9×9, opt4, one run: 85.8 / 73.7 / 86.6 µs with faults
|
||
520 k / 127 k / 519 k. Quoting the last rep there reports 86.6 when 73.7 was
|
||
achieved in the same run; the choice of rep would be doing the work, not the code.
|
||
|
||
**The `spread` column is a result, not noise.** Paths that allocate per frame
|
||
vary 3–28 % run to run; `collect_view()`, which allocates nothing, is
|
||
reproducible to 0.0–0.2 %. That contrast is the strongest argument for opt6
|
||
(harness `opt8`) — it is the only path whose throughput is *reproducible*.
|
||
|
||
**Each step runs in its own process.** The heap is process-wide, so in a shared
|
||
process every step inherits what the previous ones grew: `opt3` reports **2**
|
||
faults after opt1/opt2 have run and **92 251** on its own. `--no-isolate` restores
|
||
the fast path and makes the fault columns meaningless; throughput is unaffected
|
||
either way.
|
||
|
||
**Reps share a finder within a process** — a new one would reset the heap. The
|
||
device pedestal advances by `n_frames` each pass, so `n_clusters` drifts ~0.002 %
|
||
between reps. Compare steps at the same rep index, never across reps.
|
||
|
||
**opt7 is not a row.** It is `DEVICE_PED_TYPE` in
|
||
`include/aare/clusterfinder_kernel.cuh`. Run the matrix once per build and compare
|
||
directories; `env.json` records which arm you are in. The f64 arm is the report's
|
||
Acts I–II, the f32 arm is Act III.
|
||
|
||
**opt1/opt2 are 3×3 only** — `ClusterFinderCUDAOpt2` is registered for 3×3 in
|
||
`cuda_bindings.cu`, so the 9×9 ladder starts at opt3. They are also unaffected by
|
||
the opt7 flip by construction: their binding pins `PEDESTAL_TYPE` to `double`.
|
||
If they move between arms, something is wrong.
|
||
|
||
**The CPU baseline is first-pass only** and forced to 1 rep: `ClusterFinderMT`
|
||
cannot restart after `stop()`. Both speedup columns therefore divide by a *cold*
|
||
CPU number, which reads ~9 % generous.
|
||
|
||
**At 4 streams the probe's `kernel_us_per_frame` is engine occupancy**, the union
|
||
of kernel intervals over frames — not per-kernel duration. That is why f64 9×9
|
||
reads 32.08 µs at `s4` while each kernel is ~39.9 µs. Use `s1` for exclusive
|
||
kernel times (the opt7 claim), `s4` for "% of roofline".
|
||
|
||
**Expect zero-copy to land 2–3 % *under* its roofline.** The roofline is measured
|
||
under CUPTI, which dilates GPU op durations slightly, so it is a mild
|
||
over-estimate. "≥100 % of roofline" means *at the floor, within the profiler's
|
||
own systematic error* — not a measurement error. Take percentages from the **f32**
|
||
arm: the f64 9×9 `s4` kernel column is an interval union at `overlap = 1.36`, which
|
||
CUPTI inflates further, and zero-copy reads 6.4 % under it there.
|
||
|
||
## Two policies enforced in code
|
||
|
||
1. **Never warm up by processing frames.** The kernel pushes a pedestal update
|
||
per pixel per frame, so a finder that has seen extra frames is no longer
|
||
comparable with one that has not. Slots are pre-pinned with
|
||
`reserve_output_slots()`, which allocates without transferring or launching —
|
||
verified to leave cluster counts bit-identical.
|
||
2. **`time_kernels=False` everywhere**, including `ClusterFinderCUDAOpt2`, which
|
||
gained the flag for this reason. With events on for one finder and off for
|
||
another, the instrumented one pays a per-frame tax the other does not and the
|
||
step between them absorbs it. Kernel times come from nsys, which is the only
|
||
source correct under multi-stream load anyway.
|
||
|
||
## Results directories
|
||
|
||
```
|
||
results/<date>_<f32|f64>[_tag]/
|
||
env.json build, git rev, driver, GPU, DEVICE_PED_TYPE, timestamp
|
||
manifest.csv artifact -> config -> build -> which report section cites it
|
||
ladder_3x3.csv one row per (step, rep) — every rep kept, nothing averaged
|
||
ladder_9x9.csv
|
||
probes.csv per-engine us/frame, duty %, overlap, bottleneck, roofline
|
||
probe_*.nsys-rep openable in nsys-ui
|
||
probe_*.sqlite input to gpu_span.py
|
||
```
|
||
|
||
Current campaign: **`2026-08-18_f32/`** and **`2026-08-18_f64/`**.
|
||
`2026-08-12_f32_legacy/` is retired — see its `SUPERSEDED.md`.
|
||
|
||
> **Known wart:** `results_dir()` stamps *today's* date, so a campaign that spans
|
||
> midnight splits across two directories. That happened once already (the f32
|
||
> ladder and its probes landed a day apart) and had to be merged by hand, with a
|
||
> note added to `env.json`. Prefer a campaign tag over a date if this recurs.
|