battery README: cut the sweep for compute-sanitizer stress runs; read the Xid after a device fault

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi
This commit is contained in:
leonarski_fandClaude Opus 5.5 committed 2026-10-10 09:46:06 +02:00
1 parent d495a01de4
commit 7f4c9e2f8a
1 file changed
+9 -4
+9 -4
View File
@@ -312,8 +312,8 @@ directory per run) and never touches the run list, a baseline or a report.
G=/data/battery/gpulock.sh # take the GPU once for the whole stress run, not per rugnux
$G $B stress --rugnux build/rugnux/rugnux --tier ci --repeat 3 # ci tier, 3 times
$G $B stress --rugnux build/rugnux/rugnux --tier ci --concurrent 2 # two sets side by side on the card
$G $B stress --rugnux build/rugnux/rugnux --only 5reo,lyso_ref --sanitizer memcheck
$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --timeout 36000
$G $B stress --rugnux build/rugnux/rugnux --only 5reo,11if --sanitizer memcheck --extra "-e 60 -N 8"
$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --extra "-e 30 -N 4" --timeout 36000
```
- `--repeat N` runs every set N times; `--concurrent K` keeps K rugnux processes running at once,
@@ -322,12 +322,17 @@ $G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --ti
makes a race inside one process show. A GPU out-of-memory failure under `--concurrent` means the
card is too small for K of those sets, not a defect.
- `--sanitizer memcheck|racecheck|synccheck|initcheck` runs every rugnux under NVIDIA's
compute-sanitizer (API errors rugnux handles itself are not reported). memcheck is 10-50x slower,
racecheck far more: use the smoke tier's fastest sets (5reo, 11if) and a long `--timeout`.
compute-sanitizer (API errors rugnux handles itself are not reported). It is slow - the whole of
5reo under memcheck was still in its first pass after ten minutes - so cut the sweep and the
threads with `--extra "-e 60 -N 8"` (fewer images still runs every GPU engine) on the smoke
tier's fastest sets, give a long `--timeout`, and remember it holds the GPU lock all that time.
- `--launch-blocking` sets `CUDA_LAUNCH_BLOCKING=1`, so a kernel's fault is reported by that kernel's
own launch and not by whichever thread next touched the device - the message then names the culprit.
Before a release: the ci tier with `--repeat 3 --concurrent 2`, and memcheck on two small sets.
After a device fault, `journalctl -k | grep Xid` says which engine of the card faulted (a copy
engine `CE`, or the graphics/compute engine for a kernel) and at what address - evidence the
rugnux log, which names only the first thread to notice, does not have.
**CI.** `.gitea/workflows/battery.yml` builds rugnux with the CI flags and runs the ci tier on every
push, or by hand any subset (empty: the full open + inhouse battery), on the machine that holds the