From 7f4c9e2f8a44791726bba2a08c2a5eca4086cb0a Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Sat, 10 Oct 2026 09:46:06 +0200 Subject: [PATCH] battery README: cut the sweep for compute-sanitizer stress runs; read the Xid after a device fault Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_01SVmAWnzCmRKAXVUCdc4iNi --- tools/battery/README.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tools/battery/README.md b/tools/battery/README.md index 3dc260c3d..cd3cc6d0f 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -312,8 +312,8 @@ directory per run) and never touches the run list, a baseline or a report. G=/data/battery/gpulock.sh # take the GPU once for the whole stress run, not per rugnux $G $B stress --rugnux build/rugnux/rugnux --tier ci --repeat 3 # ci tier, 3 times $G $B stress --rugnux build/rugnux/rugnux --tier ci --concurrent 2 # two sets side by side on the card -$G $B stress --rugnux build/rugnux/rugnux --only 5reo,lyso_ref --sanitizer memcheck -$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --timeout 36000 +$G $B stress --rugnux build/rugnux/rugnux --only 5reo,11if --sanitizer memcheck --extra "-e 60 -N 8" +$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --extra "-e 30 -N 4" --timeout 36000 ``` - `--repeat N` runs every set N times; `--concurrent K` keeps K rugnux processes running at once, @@ -322,12 +322,17 @@ $G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --ti makes a race inside one process show. A GPU out-of-memory failure under `--concurrent` means the card is too small for K of those sets, not a defect. - `--sanitizer memcheck|racecheck|synccheck|initcheck` runs every rugnux under NVIDIA's - compute-sanitizer (API errors rugnux handles itself are not reported). memcheck is 10-50x slower, - racecheck far more: use the smoke tier's fastest sets (5reo, 11if) and a long `--timeout`. + compute-sanitizer (API errors rugnux handles itself are not reported). It is slow - the whole of + 5reo under memcheck was still in its first pass after ten minutes - so cut the sweep and the + threads with `--extra "-e 60 -N 8"` (fewer images still runs every GPU engine) on the smoke + tier's fastest sets, give a long `--timeout`, and remember it holds the GPU lock all that time. - `--launch-blocking` sets `CUDA_LAUNCH_BLOCKING=1`, so a kernel's fault is reported by that kernel's own launch and not by whichever thread next touched the device - the message then names the culprit. Before a release: the ci tier with `--repeat 3 --concurrent 2`, and memcheck on two small sets. +After a device fault, `journalctl -k | grep Xid` says which engine of the card faulted (a copy +engine `CE`, or the graphics/compute engine for a kernel) and at what address - evidence the +rugnux log, which names only the first thread to notice, does not have. **CI.** `.gitea/workflows/battery.yml` builds rugnux with the CI flags and runs the ci tier on every push, or by hand any subset (empty: the full open + inhouse battery), on the machine that holds the