diff --git a/tools/battery/README.md b/tools/battery/README.md index 3dc260c3d..cd3cc6d0f 100644 --- a/tools/battery/README.md +++ b/tools/battery/README.md @@ -312,8 +312,8 @@ directory per run) and never touches the run list, a baseline or a report. G=/data/battery/gpulock.sh # take the GPU once for the whole stress run, not per rugnux $G $B stress --rugnux build/rugnux/rugnux --tier ci --repeat 3 # ci tier, 3 times $G $B stress --rugnux build/rugnux/rugnux --tier ci --concurrent 2 # two sets side by side on the card -$G $B stress --rugnux build/rugnux/rugnux --only 5reo,lyso_ref --sanitizer memcheck -$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --timeout 36000 +$G $B stress --rugnux build/rugnux/rugnux --only 5reo,11if --sanitizer memcheck --extra "-e 60 -N 8" +$G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --extra "-e 30 -N 4" --timeout 36000 ``` - `--repeat N` runs every set N times; `--concurrent K` keeps K rugnux processes running at once, @@ -322,12 +322,17 @@ $G $B stress --rugnux build/rugnux/rugnux --only 5reo --sanitizer racecheck --ti makes a race inside one process show. A GPU out-of-memory failure under `--concurrent` means the card is too small for K of those sets, not a defect. - `--sanitizer memcheck|racecheck|synccheck|initcheck` runs every rugnux under NVIDIA's - compute-sanitizer (API errors rugnux handles itself are not reported). memcheck is 10-50x slower, - racecheck far more: use the smoke tier's fastest sets (5reo, 11if) and a long `--timeout`. + compute-sanitizer (API errors rugnux handles itself are not reported). It is slow - the whole of + 5reo under memcheck was still in its first pass after ten minutes - so cut the sweep and the + threads with `--extra "-e 60 -N 8"` (fewer images still runs every GPU engine) on the smoke + tier's fastest sets, give a long `--timeout`, and remember it holds the GPU lock all that time. - `--launch-blocking` sets `CUDA_LAUNCH_BLOCKING=1`, so a kernel's fault is reported by that kernel's own launch and not by whichever thread next touched the device - the message then names the culprit. Before a release: the ci tier with `--repeat 3 --concurrent 2`, and memcheck on two small sets. +After a device fault, `journalctl -k | grep Xid` says which engine of the card faulted (a copy +engine `CE`, or the graphics/compute engine for a kernel) and at what address - evidence the +rugnux log, which names only the first thread to notice, does not have. **CI.** `.gitea/workflows/battery.yml` builds rugnux with the CI flags and runs the ci tier on every push, or by hand any subset (empty: the full open + inhouse battery), on the machine that holds the