From 8fa1ab7b05c91df5c76cd4e9b8c5eeb37a75df43 Mon Sep 17 00:00:00 2001 From: Filip Leonarski Date: Tue, 25 Aug 2026 21:21:11 +0200 Subject: [PATCH] CI: make the artifact checks capable of failing, and of passing The rugnux tarball built and packaged correctly; the step that verifies it did not. Two shell mistakes, both mine, both of the same family. `! find /tmp/rgx -name 'libcufft*'` can never pass: find exits 0 whether or not it matched anything, so the negation always fails. That is what broke the x86_64 job after a successful build. The aarch64 checks were worse, in the way that matters. Piping a large producer into `grep -q` kills the producer with SIGPIPE as soon as grep exits on its first match, and `set -o pipefail` promotes that to the pipeline's status: on `cuobjdump --list-elf | grep -q sm_90` over a 225 MB binary it fails a check that should pass, and on `! ... | grep -q "x86-64"` the leading `!` inverts it into a PASS -- so an x86 file leaking into an aarch64 tarball, the exact thing that check exists to catch, would have been reported as clean. Both now capture each producer's output to a file and grep the file, with explicit if/exit rather than exit-status negation. Verified by running the checks verbatim against the real artifacts rather than only reading them: both tarballs pass, and the checks were confirmed to FAIL when they should -- an x86_64 file planted in the extracted tree is caught, and an architecture absent from the fatbin is reported missing. The toolchain file now comes from the checkout rather than /opt/cross in the image. It describes how to build this source, so it belongs with the source: baked into the image, the Eigen fix in the previous commit could not reach CI without rebuilding and re-pushing the image, and appeared to have no effect. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01SQjneRUssfhi1k9rq8Ts3h --- .gitea/workflows/build_and_test.yml | 45 ++++++++++++++++++++--------- docker/ubuntu2404/Dockerfile | 3 +- 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/.gitea/workflows/build_and_test.yml b/.gitea/workflows/build_and_test.yml index eb1c7f0f..0982ec08 100644 --- a/.gitea/workflows/build_and_test.yml +++ b/.gitea/workflows/build_and_test.yml @@ -212,9 +212,15 @@ jobs: rm -rf /tmp/rgx && mkdir -p /tmp/rgx tar xf build-rugnux/rugnux-*.tar.gz -C /tmp/rgx bin=$(find /tmp/rgx -name rugnux -type f) - # cuFFT is static now: no libcufft beside the binary and none needed at run time. - ! ldd "$bin" | grep -q cufft - ! find /tmp/rgx -name 'libcufft*' + # Each producer writes to a file and the greps read files, rather than piping into + # `grep -q`: grep exits at the first match, the producer then dies of SIGPIPE, and under + # `set -o pipefail` that becomes the pipeline's status -- which a leading `!` would turn + # into a pass, hiding the very thing being tested. + ldd "$bin" > /tmp/rgx.ldd 2>&1 || true + if grep -q cufft /tmp/rgx.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi + find /tmp/rgx -name 'libcufft*' > /tmp/rgx.cufft + if [ -s /tmp/rgx.cufft ]; then echo "a libcufft is shipped in the tarball"; exit 1; fi + echo "self-contained: static cuFFT, nothing to ship beside it" - name: Upload rugnux tgz to release if: github.ref_type == 'tag' shell: bash @@ -284,6 +290,10 @@ jobs: # Cross-compiled on an x86_64 runner: nvcc, gcc and cmake all run native and only emit aarch64, # so this costs about what a normal build costs. Uses the ordinary ubuntu2404 image, which now # also carries the aarch64 toolchain and the CUDA sbsa cross tree. + # + # The toolchain file comes from the CHECKOUT, not from the image. It describes how to build this + # source, so it belongs with the source: baked into the image it could only be changed by + # rebuilding and re-pushing the image, and a fix to it would appear to have no effect. runs-on: jfjoch_ubuntu2404 container: image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2508 @@ -300,7 +310,7 @@ jobs: # sm_90 = GH200 (Grace Hopper, e.g. ALPS), sm_121 = GB10 (DGX Spark). One tarball, both. run: | cmake -G Ninja -S . -B build-aarch64 \ - -DCMAKE_TOOLCHAIN_FILE=/opt/cross/aarch64-sbsa.cmake \ + -DCMAKE_TOOLCHAIN_FILE="$PWD/docker/ubuntu2404/aarch64-sbsa.cmake" \ -DCMAKE_BUILD_TYPE=Release \ -DJFJOCH_RUGNUX_ONLY=ON \ -DJFJOCH_USE_CUDA=ON \ @@ -313,20 +323,29 @@ jobs: shell: bash run: | set -euo pipefail - tgz=$(ls build-aarch64/*.tar.gz | head -1) + tgz=$(ls build-aarch64/rugnux-*.tar.gz | head -1) echo "checking $tgz" rm -rf /tmp/tgzcheck && mkdir -p /tmp/tgzcheck tar xf "$tgz" -C /tmp/tgzcheck bin=$(find /tmp/tgzcheck -name rugnux -type f | head -1) - file "$bin" - file "$bin" | grep -q "ARM aarch64" + # Everything is captured to a file first and the greps read files. Piping a large + # producer into `grep -q` makes it die of SIGPIPE once grep exits, which `set -o pipefail` + # turns into a failure -- and on a negated check into a spurious PASS. + file "$bin" > /tmp/tc.bin + cat /tmp/tc.bin + if ! grep -q "ARM aarch64" /tmp/tc.bin; then echo "not an aarch64 binary"; exit 1; fi # nothing x86 may have leaked in (libjpeg-turbo's ExternalProject is the classic culprit) - ! find /tmp/tgzcheck -type f -exec file {} + | grep -q "x86-64" - # cuFFT must be linked statically: no libcufft beside the binary, none needed at run time - ! ldd "$bin" 2>/dev/null | grep -q cufft - # both GPU targets present in the fatbin - cuobjdump --list-elf "$bin" | grep -q sm_90 - cuobjdump --list-elf "$bin" | grep -q sm_121 + find /tmp/tgzcheck -type f -exec file {} + > /tmp/tc.arch + if grep -q "x86-64" /tmp/tc.arch; then echo "an x86-64 file leaked into the tarball"; exit 1; fi + # cuFFT must be linked statically: nothing to ship beside the binary, nothing to find + ldd "$bin" > /tmp/tc.ldd 2>&1 || true + if grep -q cufft /tmp/tc.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi + # both GPU targets present in the fatbin: sm_90 = GH200, sm_121 = GB10 + cuobjdump --list-elf "$bin" > /tmp/tc.elf + for arch in sm_90 sm_121; do + if ! grep -q "$arch" /tmp/tc.elf; then echo "fatbin is missing $arch"; exit 1; fi + done + echo "aarch64, self-contained, both GPU targets present" - name: Upload tarball to release if: github.ref_type == 'tag' shell: bash diff --git a/docker/ubuntu2404/Dockerfile b/docker/ubuntu2404/Dockerfile index 1f8f00e7..8fcc26ce 100644 --- a/docker/ubuntu2404/Dockerfile +++ b/docker/ubuntu2404/Dockerfile @@ -239,7 +239,8 @@ RUN set -eux; \ apt-get clean; \ rm -rf /var/lib/apt/lists/* -COPY aarch64-sbsa.cmake /opt/cross/aarch64-sbsa.cmake +# aarch64-sbsa.cmake is deliberately NOT copied in: the build passes it from the checkout, so the +# toolchain is versioned with the source it configures and needs no image rebuild to change. # Set workdir for your project WORKDIR /workspace