battery README: cut the sweep for compute-sanitizer stress runs; read the Xid after a device fault
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This commit is contained in:
1 parent
d495a01de4
commit
7f4c9e2f8a
1 file changed
+9
-4
@@ -312,8 +312,8 @@ directory per run) and never touches the run list, a baseline or a report.
|
||||
G=/data/battery/gpulock.sh # take the GPU once for the whole stress run, not per rugnux
|
||||
$G $B stress --rugnux build/rugnux/rugnux --tier ci --repeat 3 # ci tier, 3 times
|
||||
$G $B stress --rugnux build/rugnux/rugnux --tier ci --concurrent 2 # two sets side by side on the card
|
||||
$G $B stress --rugnux build/rugnux/rugnux --only 5reo,lyso_ref --sanitizer memcheck
|
||||
$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --timeout 36000
|
||||
$G $B stress --rugnux build/rugnux/rugnux --only 5reo,11if --sanitizer memcheck --extra "-e 60 -N 8"
|
||||
$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --extra "-e 30 -N 4" --timeout 36000
|
||||
```
|
||||
|
||||
- `--repeat N` runs every set N times; `--concurrent K` keeps K rugnux processes running at once,
|
||||
@@ -322,12 +322,17 @@ $G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --ti
|
||||
makes a race inside one process show. A GPU out-of-memory failure under `--concurrent` means the
|
||||
card is too small for K of those sets, not a defect.
|
||||
- `--sanitizer memcheck|racecheck|synccheck|initcheck` runs every rugnux under NVIDIA's
|
||||
compute-sanitizer (API errors rugnux handles itself are not reported). memcheck is 10-50x slower,
|
||||
racecheck far more: use the smoke tier's fastest sets (5reo, 11if) and a long `--timeout`.
|
||||
compute-sanitizer (API errors rugnux handles itself are not reported). It is slow - the whole of
|
||||
5reo under memcheck was still in its first pass after ten minutes - so cut the sweep and the
|
||||
threads with `--extra "-e 60 -N 8"` (fewer images still runs every GPU engine) on the smoke
|
||||
tier's fastest sets, give a long `--timeout`, and remember it holds the GPU lock all that time.
|
||||
- `--launch-blocking` sets `CUDA_LAUNCH_BLOCKING=1`, so a kernel's fault is reported by that kernel's
|
||||
own launch and not by whichever thread next touched the device - the message then names the culprit.
|
||||
|
||||
Before a release: the ci tier with `--repeat 3 --concurrent 2`, and memcheck on two small sets.
|
||||
After a device fault, `journalctl -k | grep Xid` says which engine of the card faulted (a copy
|
||||
engine `CE`, or the graphics/compute engine for a kernel) and at what address - evidence the
|
||||
rugnux log, which names only the first thread to notice, does not have.
|
||||
|
||||
**CI.** `.gitea/workflows/battery.yml` builds rugnux with the CI flags and runs the ci tier on every
|
||||
push, or by hand any subset (empty: the full open + inhouse battery), on the machine that holds the
|
||||
|
||||
Reference in new issue
Block a user