mx/Jungfraujoch
3
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 81 Wiki Activity

TEST BRANCH ONLY: run the aarch64 bitshuffle exactness test and decode benchmark on the macOS runner
arm64 bitshuffle bench / test:bitshuffle:macos-arm64 (push) Successful in 8s
Details

Browse Source 
Not for merging. The main workflow is removed on this branch so a push runs only this one job:
build the standalone bitshuffle test and benchmark with clang, check the decoded image hashes of one
EIGER2 and one PILATUS4 chunk against the x86 decoder's, and time classic NEON vs the per-block-
malloc fix vs the NEON port of hperf.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D1G8gJVAy6gp1K5Dz3NE5C
This commit is contained in:
leonarski_f
2026-09-27 21:26:28 +02:00
co-authored by Claude Opus 5.5
parent e886189829
commit 488f3e4dcf
7 changed files with 344 additions and 840 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/arm64_bitshuffle_bench.yml
+38
View File
@@ -0,0 +1,38 @@
name: arm64 bitshuffle bench
# Test branch only: exactness and speed of the aarch64 bitshuffle decode on the Mac mini runner.
on:
push:
branches:
- arm64-bitshuffle-bench
workflow_dispatch:
jobs:
bench:
name: test:bitshuffle:macos-arm64
runs-on: macos-arm64
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Machine
shell: bash
run: |
uname -a
sysctl -n machdep.cpu.brand_string hw.perflevel0.logicalcpu hw.perflevel1.logicalcpu || true
clang --version | head -1
- name: Build
shell: bash
run: tools/bitshuffle_bench/build.sh "$GITHUB_WORKSPACE" clang clang++
- name: Exactness (image hashes must match the x86 decoder)
shell: bash
run: |
cd tools/bitshuffle_bench
./bs_exact_test eiger2_0.bslz4 pilatus4_0.bslz4 | tee exact.log
grep -q 'image hash 4e45f1d27c1f8992' exact.log
grep -q 'image hash 99158ebdd8ebe15e' exact.log
grep -q ' 0 failures' exact.log
- name: Decode speed (classic NEON vs S1 vs hperf NEON port)
shell: bash
run: |
cd tools/bitshuffle_bench
./bs_bench eiger2_0.bslz4 pilatus4_0.bslz4
.gitea/workflows/build_and_test.yml
-840
View File
@@ -1,840 +0,0 @@
name: Build Packages
env:
# Architecture and link-time optimisation for every Linux configure. Both are deliberately absent
# from CMakeLists.txt so a site can pick its own (x86-64-v4 on an AVX-512 cluster, -march=native,
# or the plain baseline); CI pins what it ships.
#
# -flto=auto needs GCC >= 10 and both CI images put gcc-toolset-13 on PATH. Measured on rugnux
# against an otherwise identical build: 7-10% fewer retired instructions and a 9% smaller binary,
# but only ~1.5% off the wall clock, because the offline pipeline is GPU- and I/O-bound rather
# than CPU-instruction-bound. It costs about 3x on an INCREMENTAL rebuild (one file plus link,
# 9.8 s -> 30.1 s), which is why it belongs here and not in CMakeLists: CI always builds from
# scratch and ships the result, so it pays the link once and a developer never pays it at all.
#
# Not CMAKE_INTERPROCEDURAL_OPTIMIZATION, which is the tidier spelling and would cover MSVC too:
# built and linked fine (CUDA included, and it does not reach .cu either way), but measured 3%
# MORE retired instructions than -flto=auto - 396.9 G vs 384.9 G over three runs each against a
# 0.45% run-to-run spread, so the gap is real. It also leaves two dependencies (FFTW, libzmq) out
# of LTO that the flag reaches, and it did not switch CMAKE_AR to gcc-ar here, so the one
# correctness argument for it did not materialise either.
LINUX_CMAKE_FLAGS: '-DCMAKE_CXX_FLAGS="-march=x86-64-v3 -flto=auto" -DCMAKE_C_FLAGS="-march=x86-64-v3 -flto=auto"'
# MSVC has no spelling for the x86-64-v2 level; /arch:AVX is the nearest and implies SSE4.1/4.2,
# which is what matters here - without it Eigen has no vectorised round and falls back to a libm
# call per element. AVX is Sandy Bridge (2011) and up, a safe floor for a desktop viewer.
MSVC_ARCH_CMAKE_FLAGS: '-DCMAKE_CXX_FLAGS="/arch:AVX" -DCMAKE_C_FLAGS="/arch:AVX"'
on:
push:
branches:
- '**'
# No pull_request trigger: branches live in this repository, so the push above has already run
# the whole workflow on the very same commit. Adding the PR event only ran it a second time.
workflow_dispatch:
inputs:
create_release:
type: boolean
description: 'Create release'
required: false
default: false
jobs:
# ---- Viewer --------------------------------------------------------------------------------------
build-viewer-tgz:
name: build:viewer:linux-x86_64:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
include:
- variant: cuda
use_cuda: 'ON'
- variant: nocuda
use_cuda: 'OFF'
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build # self-hosted runners reuse the workspace volume; a stale build dir
# can leave the viewer linking a system libcurl instead of the vendored one
- name: Configure viewer build
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DJFJOCH_VIEWER_ONLY=ON -DJFJOCH_USE_CUDA=${{ matrix.use_cuda }} ${{ env.LINUX_CMAKE_FLAGS }} -DCMAKE_BUILD_TYPE=Release ..
- name: Build viewer tgz
shell: bash
run: |
cd build
# The "viewer" component is now jfjoch_viewer alone -- rugnux ships as its own artifact
# (build:rugnux-tgz) and the two CLI tools are no longer packaged at all. The full tree is
# still built because the viewer links the JFJochRugnux library for its in-process jobs.
ninja -j16
cpack
- name: Upload viewer tgz to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names the archive jfjoch_viewer-<version>-linux-{cuda<major>|cpu}.tar.gz;
# rename to .tgz (matching the Windows installer's release naming) before upload.
files=(build/jfjoch_viewer-*.tar.gz)
if [ ${#files[@]} -eq 0 ]; then
echo "No viewer archive found in build/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
build-windows:
name: build:viewer:windows-x86_64:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: windows-11-cuda-qt
timeout-minutes: 120
strategy:
fail-fast: false
matrix:
include:
- variant: cuda
use_cuda: 'ON'
- variant: nocuda
use_cuda: 'OFF'
steps:
- uses: actions/checkout@v4
- name: Configure viewer build
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake -G Ninja -B build -DJFJOCH_USE_CUDA=${{ matrix.use_cuda }} -DCMAKE_BUILD_TYPE=Release ${{ env.MSVC_ARCH_CMAKE_FLAGS }} -DCMAKE_PREFIX_PATH="C:/deps;C:/Qt/6.11.1/msvc2022_64"
- name: Build viewer
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake --build build
- name: Build installer (NSIS)
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cd build
cpack
- name: Upload installer to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: powershell
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
# NSIS installer named jfjoch-viewer-<version>-win64-{cuda<major>|cpu}.exe (see CMakeLists.txt).
$files = Get-ChildItem -Path build -Filter 'jfjoch-viewer-*-win64-*.exe'
if ($files.Count -eq 0) { throw 'No Windows installer found in build/' }
foreach ($file in $files) {
python gitea_upload_file.py $file.FullName
if ($LASTEXITCODE -ne 0) { throw "Upload failed for $($file.Name)" }
}
build-macos-viewer:
name: build:viewer:macos-arm64:nocuda
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
# A Mac mini running act_runner in host mode (no containers on macOS): the job runs as the runner's
# user with that user's tools - Xcode command line tools, CMake.app and Qt under ~/Qt - so the
# paths below are this machine's, like the Windows runner's C:/Qt.
runs-on: macos-arm64
timeout-minutes: 120
env:
QT_PREFIX: /Users/jungfrau/Qt/6.11.2/macos
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-macos # host runners keep the workspace between runs
- name: Configure viewer build
shell: bash
# JFJOCH_VIEWER_ONLY is forced on macOS. No -mcpu: the arm64 baseline is what every Apple
# Silicon Mac runs, and NEON is part of it.
run: cmake -S . -B build-macos -DCMAKE_BUILD_TYPE=Release -DJFJOCH_USE_CUDA=OFF -DCMAKE_PREFIX_PATH="$QT_PREFIX"
- name: Build viewer and DMG
shell: bash
run: |
cmake --build build-macos -j "$(sysctl -n hw.ncpu)"
cd build-macos && cpack
- name: Verify the bundle is self-contained
shell: bash
run: |
set -euo pipefail
app=$(ls -d build-macos/_CPack_Packages/Darwin/DragNDrop/*/ALL_IN_ONE/jfjoch_viewer.app)
# The frameworks sit in the bundle; the executable must look for them there ...
otool -l "$app/Contents/MacOS/jfjoch_viewer" > /tmp/jfjv.load
if ! grep -q "@executable_path/../Frameworks" /tmp/jfjv.load; then echo "no rpath into the bundle"; exit 1; fi
# ... and nothing in the bundle may point at this machine's Qt (or anything else under /Users)
find "$app" -type f \( -perm +111 -o -name '*.dylib' \) -exec otool -L {} + > /tmp/jfjv.libs 2>/dev/null || true
if grep -q "/Users/" /tmp/jfjv.libs; then grep "/Users/" /tmp/jfjv.libs; echo "absolute path into the build machine"; exit 1; fi
# the (ad-hoc) signature must survive everything installed into the bundle
codesign --verify --deep --strict "$app"
echo "self-contained and validly signed"
# Notarization - needs an Apple Developer ID, which the project does not have yet. Once it
# does, the app has to be signed with it (hardened runtime, secure timestamp) BEFORE cpack
# packs it, e.g. by passing to macdeployqt, through qt_generate_deploy_app_script's
# DEPLOY_TOOL_OPTIONS in viewer/CMakeLists.txt:
# -codesign="Developer ID Application: <NAME> (<TEAMID>)" -hardened-runtime -timestamp
# and the DMG then notarized and stapled here, with an app-specific password stored once on
# the runner (xcrun notarytool store-credentials jfjoch-notary ...):
# xcrun notarytool submit build-macos/jfjoch-viewer-*.dmg --keychain-profile jfjoch-notary --wait
# xcrun stapler staple build-macos/jfjoch-viewer-*.dmg
# Until then a downloaded DMG is blocked by Gatekeeper on first open (right-click > Open).
- name: Upload DMG to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names it jfjoch-viewer-<version>-macos-arm64.dmg
files=(build-macos/jfjoch-viewer-*.dmg)
if [ ${#files[@]} -eq 0 ]; then
echo "No DMG found in build-macos/"
exit 1
fi
for file in "${files[@]}"; do
python3 gitea_upload_file.py "$file"
done
# ---- Rugnux --------------------------------------------------------------------------------------
build-rugnux-tgz:
name: build:rugnux:linux-x86_64:cuda
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-rugnux # self-hosted runners reuse the workspace volume
- name: Configure rugnux build
shell: bash
run: |
cmake -G Ninja -S . -B build-rugnux -DJFJOCH_RUGNUX_ONLY=ON -DJFJOCH_USE_CUDA=ON \
${{ env.LINUX_CMAKE_FLAGS }} -DCMAKE_BUILD_TYPE=Release
- name: Build rugnux tgz
shell: bash
run: cmake --build build-rugnux --target package -j16
- name: Verify the artifact is self-contained
shell: bash
run: |
set -euo pipefail
rm -rf /tmp/rgx && mkdir -p /tmp/rgx
tar xf build-rugnux/rugnux-*.tar.gz -C /tmp/rgx
bin=$(find /tmp/rgx -name rugnux -type f)
# Each producer writes to a file and the greps read files, rather than piping into
# `grep -q`: grep exits at the first match, the producer then dies of SIGPIPE, and under
# `set -o pipefail` that becomes the pipeline's status -- which a leading `!` would turn
# into a pass, hiding the very thing being tested.
ldd "$bin" > /tmp/rgx.ldd 2>&1 || true
if grep -q cufft /tmp/rgx.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi
find /tmp/rgx -name 'libcufft*' > /tmp/rgx.cufft
if [ -s /tmp/rgx.cufft ]; then echo "a libcufft is shipped in the tarball"; exit 1; fi
echo "self-contained: static cuFFT, nothing to ship beside it"
- name: Upload rugnux tgz to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names the archive rugnux-<version>-linux-{cuda<major>|cpu}.tar.gz;
# rename to .tgz before upload, as the viewer archive does.
files=(build-rugnux/rugnux-*.tar.gz)
if [ ${#files[@]} -eq 0 ]; then
echo "No rugnux archive found in build-rugnux/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
build-rugnux-arm:
name: build:rugnux:linux-aarch64:cuda # cross-compiled on x86_64
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
# Cross-compiled on an x86_64 runner: nvcc, gcc and cmake all run native and only emit aarch64,
# so this costs about what a normal build costs. Uses the ordinary ubuntu2404 image, which now
# also carries the aarch64 toolchain and the CUDA sbsa cross tree.
#
# The toolchain file comes from the CHECKOUT, not from the image. It describes how to build this
# source, so it belongs with the source: baked into the image it could only be changed by
# rebuilding and re-pushing the image, and a fix to it would appear to have no effect.
runs-on: jfjoch_ubuntu2404
container:
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-aarch64 # self-hosted runners reuse the workspace volume
- name: Configure (cross to aarch64 SBSA)
shell: bash
# NOT ${{ env.LINUX_CMAKE_FLAGS }}: that pins -march=x86-64-v3, which an aarch64 compiler
# rejects. The aarch64 baseline already has NEON and a vectorised round, so the Eigen gap
# that flag closes on x86 does not exist here.
# sm_90 = GH200 (Grace Hopper, e.g. ALPS), sm_121 = GB10 (DGX Spark). One tarball, both.
run: |
cmake -G Ninja -S . -B build-aarch64 \
-DCMAKE_TOOLCHAIN_FILE="$PWD/docker/ubuntu2404/aarch64-sbsa.cmake" \
-DCMAKE_BUILD_TYPE=Release \
-DJFJOCH_RUGNUX_ONLY=ON \
-DJFJOCH_USE_CUDA=ON \
-DJFJOCH_CUDA_ARCHITECTURES="90;121"
- name: Build and package
shell: bash
run: cmake --build build-aarch64 --target package -j16
# A cross build that silently produced x86_64 still packages cleanly, so assert the artifact.
- name: Verify the tarball is really aarch64
shell: bash
run: |
set -euo pipefail
tgz=$(ls build-aarch64/rugnux-*.tar.gz | head -1)
echo "checking $tgz"
rm -rf /tmp/tgzcheck && mkdir -p /tmp/tgzcheck
tar xf "$tgz" -C /tmp/tgzcheck
bin=$(find /tmp/tgzcheck -name rugnux -type f | head -1)
# Everything is captured to a file first and the greps read files. Piping a large
# producer into `grep -q` makes it die of SIGPIPE once grep exits, which `set -o pipefail`
# turns into a failure -- and on a negated check into a spurious PASS.
file "$bin" > /tmp/tc.bin
cat /tmp/tc.bin
if ! grep -q "ARM aarch64" /tmp/tc.bin; then echo "not an aarch64 binary"; exit 1; fi
# nothing x86 may have leaked in (libjpeg-turbo's ExternalProject is the classic culprit)
find /tmp/tgzcheck -type f -exec file {} + > /tmp/tc.arch
if grep -q "x86-64" /tmp/tc.arch; then echo "an x86-64 file leaked into the tarball"; exit 1; fi
# cuFFT must be linked statically: nothing to ship beside the binary, nothing to find
ldd "$bin" > /tmp/tc.ldd 2>&1 || true
if grep -q cufft /tmp/tc.ldd; then echo "cuFFT is linked dynamically"; exit 1; fi
# both GPU targets present in the fatbin: sm_90 = GH200, sm_121 = GB10
cuobjdump --list-elf "$bin" > /tmp/tc.elf
for arch in sm_90 sm_121; do
if ! grep -q "$arch" /tmp/tc.elf; then echo "fatbin is missing $arch"; exit 1; fi
done
echo "aarch64, self-contained, both GPU targets present"
- name: Upload tarball to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
files=(build-aarch64/rugnux-*.tar.gz)
if [ ${#files[@]} -eq 0 ]; then
echo "No rugnux archive found in build-aarch64/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
build-rugnux-windows:
name: build:rugnux:windows-x86_64:cuda
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: windows-11-cuda-qt
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
# cmd, not bash: the Windows runner has no bash, and every cmake invocation needs the MSVC
# environment from vcvars64.bat first -- same shape as build-windows above. No Qt in
# CMAKE_PREFIX_PATH: JFJOCH_RUGNUX_ONLY is the portable subtree minus viewer/, and every
# library it needs is now fetched and built by the configure, so C:/deps is only a leftover.
- name: Configure rugnux build
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake -G Ninja -B build-rugnux -DJFJOCH_RUGNUX_ONLY=ON -DJFJOCH_USE_CUDA=ON -DCMAKE_BUILD_TYPE=Release ${{ env.MSVC_ARCH_CMAKE_FLAGS }} -DCMAKE_PREFIX_PATH="C:/deps"
- name: Build rugnux
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cmake --build build-rugnux
- name: Build rugnux zip
shell: cmd
run: |
for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath`) do set "VSPATH=%%i"
call "%VSPATH%\VC\Auxiliary\Build\vcvars64.bat"
cd build-rugnux
cpack
- name: Upload rugnux zip to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: powershell
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
# Zip named rugnux-<version>-win64-{cuda<major>|cpu}.zip (see CMakeLists.txt).
$files = Get-ChildItem -Path build-rugnux -Filter 'rugnux-*-win64-*.zip'
if ($files.Count -eq 0) { throw 'No rugnux zip found in build-rugnux/' }
foreach ($file in $files) {
python gitea_upload_file.py $file.FullName
if ($LASTEXITCODE -ne 0) { throw "Upload failed for $($file.Name)" }
}
build-rugnux-macos:
name: build:rugnux:macos-arm64:nocuda
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: macos-arm64 # see build-macos-viewer
timeout-minutes: 120
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build-rugnux-macos
- name: Configure rugnux build
shell: bash
run: cmake -S . -B build-rugnux-macos -DJFJOCH_RUGNUX_ONLY=ON -DJFJOCH_USE_CUDA=OFF -DCMAKE_BUILD_TYPE=Release
- name: Build rugnux tgz
shell: bash
run: cmake --build build-rugnux-macos --target package -j "$(sysctl -n hw.ncpu)"
- name: Verify the artifact is self-contained
shell: bash
run: |
set -euo pipefail
rm -rf /tmp/rgxm && mkdir -p /tmp/rgxm
tar xf build-rugnux-macos/rugnux-*.tar.gz -C /tmp/rgxm
bin=$(find /tmp/rgxm -name rugnux -type f)
# Only the system's own libraries may be linked dynamically; everything else is static.
otool -L "$bin" | tail -n +2 > /tmp/rgxm.libs
if grep -v -e "/usr/lib/" -e "/System/Library/" /tmp/rgxm.libs; then echo "links a non-system library"; exit 1; fi
# and it runs: with no arguments rugnux prints its usage
"$bin" > /tmp/rgxm.out 2>&1 || true
if ! grep -qi "usage" /tmp/rgxm.out; then cat /tmp/rgxm.out; echo "rugnux did not start"; exit 1; fi
echo "self-contained and runs"
- name: Upload rugnux tgz to release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
shopt -s nullglob
# CMakeLists.txt names it rugnux-<version>-macos-arm64-cpu.tar.gz; rename to .tgz before
# upload, as the Linux archives are.
files=(build-rugnux-macos/rugnux-*.tar.gz)
if [ ${#files[@]} -eq 0 ]; then
echo "No rugnux archive found in build-rugnux-macos/"
exit 1
fi
for file in "${files[@]}"; do
tgz="${file%.tar.gz}.tgz"
mv "$file" "$tgz"
python3 gitea_upload_file.py "$tgz"
done
# ---- Jungfraujoch --------------------------------------------------------------------------------
unit-tests:
name: Unit tests
# A release built by hand does not re-run the suite: the push that put the commit on a branch
# already ran it on that very commit, and so does the merge to main. Nothing is skipped that
# was not already run, and a release build is the one case where the wait is pure delay.
if: "!(github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true))"
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
options: --gpus all
timeout-minutes: 90
env:
CTEST_OUTPUT_ON_FAILURE: '1'
steps:
- uses: actions/checkout@v4
- name: Configure auth and fetch LFS
shell: bash
env:
GITEA_TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
git lfs install --local
AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
git config --local --unset http.${{ github.server_url }}/.extraheader
git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
git lfs pull
- name: Build tests
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ${{ env.LINUX_CMAKE_FLAGS }} ..
ninja -j16 jfjoch_test jfjoch_hdf5_test jfjoch_hdf5_enospc_test enospc_shim
- name: Run unit tests
shell: bash
run: |
cd build/tests
./jfjoch_test
- name: HDF5 ENOSPC test
shell: bash
run: |
cd build/tests
LD_PRELOAD=./enospc_shim.so ./jfjoch_hdf5_enospc_test HDF5File_enospc
LD_PRELOAD=./enospc_shim.so ./jfjoch_hdf5_enospc_test FileWriter_enospc
- name: Run hdf5 test
shell: bash
run: |
cd build/tools
./jfjoch_hdf5_test ../../tests/test_data/compression_benchmark.h5
build-rpm:
name: build:jfjoch:${{ matrix.platform }}:${{ matrix.variant }}
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: ${{ matrix.runner }}
container:
image: ${{ matrix.image }}
strategy:
fail-fast: false
matrix:
include:
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8
platform: rocky8
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet8-cuda12/upload
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8_sls9
platform: rocky8
variant: cuda-sls9
cmake_flags: -DJFJOCH_USE_CUDA=ON -DSLS9=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet9-cuda12/upload
- runner: jfjoch_rocky8
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
distro: rocky8_nocuda
platform: rocky8
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el8/slsdet8-nocuda/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9
platform: rocky9
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet8-cuda13/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9_sls9
platform: rocky9
variant: cuda-sls9
cmake_flags: -DJFJOCH_USE_CUDA=ON -DSLS9=ON
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet9-cuda13/upload
- runner: jfjoch_rocky9
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
distro: rocky9_nocuda
platform: rocky9
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.rpm"
upload_url: https://gitea.psi.ch/api/packages/mx/rpm/centos/el9/slsdet8-nocuda/upload
- runner: jfjoch_ubuntu2204
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2204:2609
distro: ubuntu2204
platform: ubuntu2204
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/jammy/cuda13/upload
- runner: jfjoch_ubuntu2204
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2204:2609
distro: ubuntu2204_nocuda
platform: ubuntu2204
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/jammy/nocuda/upload
- runner: jfjoch_ubuntu2404
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
distro: ubuntu2404
platform: ubuntu2404
variant: cuda
cmake_flags: -DJFJOCH_USE_CUDA=ON
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/noble/cuda13/upload
- runner: jfjoch_ubuntu2404
image: gitea.psi.ch/leonarski_f/jfjoch_ubuntu2404:2609
distro: ubuntu2404_nocuda
platform: ubuntu2404
variant: nocuda
cmake_flags: -DJFJOCH_USE_CUDA=OFF
pkg_glob: "*.deb"
upload_url: https://gitea.psi.ch/api/packages/mx/debian/pool/noble/nocuda/upload
steps:
- uses: actions/checkout@v4
- name: Clean previous build
shell: bash
run: rm -rf build # self-hosted runners reuse the workspace volume; a stale build dir
# can leave the viewer linking a system libcurl instead of the vendored one
- name: Setup build (cmake)
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DJFJOCH_INSTALL_DRIVER_SOURCE=ON -DJFJOCH_VIEWER_BUILD=ON ${{ env.LINUX_CMAKE_FLAGS }} -DCMAKE_BUILD_TYPE=Release ${{ matrix.cmake_flags }} ..
- name: Build packages
shell: bash
run: |
cd build
ninja frontend
ninja -j16 package
- name: Upload packages
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
set -euo pipefail
cd build
shopt -s nullglob
files=(${{ matrix.pkg_glob }})
if [ ${#files[@]} -eq 0 ]; then
echo "No package files found for pattern: ${{ matrix.pkg_glob }}"
exit 1
fi
url="${{ matrix.upload_url }}"
if [[ "$url" == *"/rpm/"* ]]; then
url="${url}?sign=true"
fi
for file in "${files[@]}"; do
echo "Uploading $file -> $url"
curl --fail --user __token__:"$TOKEN" --upload-file "$file" "$url"
done
cd ..
# The viewer is shipped to the release as a Linux-version-agnostic .tgz by the
# build-viewer-tgz job; only the writer (and the XDS plugin .so on rocky8) go to the
# release as distro packages here.
if [ "${{ matrix.distro }}" = "rocky8_nocuda" ]; then
for file in build/jfjoch-writer*rpm build/xds-plugin/libjfjoch_xds_plugin.so.*; do
python3 gitea_upload_file.py "$file"
done
elif [ "${{ matrix.distro }}" = "rocky9_nocuda" ]; then
for file in build/jfjoch-writer*rpm; do
python3 gitea_upload_file.py "$file"
done
elif [ "${{ matrix.distro }}" = "ubuntu2204_nocuda" ]; then
for file in build/jfjoch*writer*.deb; do
python3 gitea_upload_file.py "$file"
done
fi
hdf5-consumer-tests:
name: HDF5 consumer tests (DIALS, XDS)
runs-on: jfjoch_rocky9
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky9:2609
steps:
- uses: actions/checkout@v4
- name: Build processing test
shell: bash
run: |
mkdir -p build
cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ${{ env.LINUX_CMAKE_FLAGS }} ..
ninja -j16 jfjoch_hdf5_test jfjoch_xds_plugin
# One job for what used to be four, each of which configured and built the same
# jfjoch_hdf5_test on the same image. Every case still runs when an earlier one fails -- each
# is a subshell under `set -e`, so a case stops at its own first bad command without taking
# the rest down -- and the summary at the end decides the job's result.
- name: Run every consumer against every file layout
shell: bash
run: |
cd "$GITHUB_WORKSPACE"
HDF5_TEST=$PWD/build/tools/jfjoch_hdf5_test
DATA=$PWD/tests/test_data/compression_benchmark.h5
# Unquoted on use: these are two arguments, not one.
CELL="space_group=P43212 unit_cell=78.551,78.551,36.914,90.000,90.000,90.000"
# -- DIALS -------------------------------------------------------------------------
# dials.import before xia2.ssx, checking the image COUNT: dxtbx reports a short scan
# rather than failing when it cannot reach every image, and xia2.ssx still succeeds on
# the images it did get, so an exit code alone does not notice. Multi-file NXmxLegacy is
# exercised by the XDS cases below but deliberately not here: dxtbx reads only the first
# data file of a legacy master (it takes one dataset out of NXdata and indexes it
# globally), so that combination cannot pass until the limitation is fixed upstream.
dials_case() { # <dir> <prefix> <jfjoch_hdf5_test flags...>
local dir=$1 prefix=$2; shift 2
source /opt/dials-v3-27-0/dials_env.sh
rm -rf "$dir"; mkdir -p "$dir"; cd "$dir"
"$HDF5_TEST" "$DATA" "$@" -o "$prefix"
dials.import "${prefix}_master.h5" | tee import.log
grep -q "num images: 100" import.log || { echo "DIALS did not read all 100 images"; return 1; }
xia2.ssx image="${prefix}_master.h5" $CELL
}
dials_legacy() { dials_case test01 legacy -n100; }
dials_vds() { dials_case test02 vds -n100 -f25 -V; }
# -S writes <prefix>_master.h5, not <prefix>.h5 - naming this wrong meant xia2.ssx was
# pointed at a file that never existed, so the integrated format went untested.
dials_single() { dials_case test03 single -n100 -S; }
# Everything above writes an UNTILTED detector, where every plausible encoding of the tilt
# agrees exactly - so none of it constrains the NXmx transformation chain. This does.
dials_tilted() {
source /opt/dials-v3-27-0/dials_env.sh
rm -rf test04; mkdir -p test04; cd test04
"$HDF5_TEST" "$DATA" -n1 -S -o geom
dials.python "$GITHUB_WORKSPACE/tests/nxmx_geometry_dials_test.py" geom_master.h5
}
# -- XDS, once per read plugin -----------------------------------------------------
# Each case clears its directory first. The runner reuses the workspace volume, so a
# leftover .h5/.LP from an earlier run could let XDS "pass" on files this case never
# wrote -- which is what the first case of two of these jobs used to risk.
xds_case() { # <dir> <jfjoch_hdf5_test flags...>
local dir=$1; shift
cd "tests/$dir"
rm -f ./*.h5 ./*.LP ./*.HKL
"$HDF5_TEST" "$DATA" "$@"
/opt/xds/xds_par
test -f IDXREF.LP
}
durin_legacy() { xds_case xds_durin -n25 -f10; }
durin_vds() { xds_case xds_durin -n25 -f10 -V; }
durin_single() { xds_case xds_durin -n25 -S; }
jfjoch_legacy() { xds_case xds -n25 -f10; }
jfjoch_vds() { xds_case xds -n25 -f10 -V; }
jfjoch_single() { xds_case xds -n25 -S; }
# Neggia cannot read a VDS master, so it gets the two layouts it supports.
neggia_legacy() { xds_case xds_neggia -n25 -f10; }
neggia_single() { xds_case xds_neggia -n25 -S; }
fail=0
summary=""
for t in dials_legacy dials_vds dials_single dials_tilted \
durin_legacy durin_vds durin_single \
jfjoch_legacy jfjoch_vds jfjoch_single \
neggia_legacy neggia_single; do
echo "::group::$t"
# Run the subshell as a command of its own and test $? afterwards. A subshell used
# directly as an `if` condition has its `set -e` ignored, so the case would carry on
# past its first failure and report only the last command's status.
( set -e; cd "$GITHUB_WORKSPACE"; "$t" )
rc=$?
if [ "$rc" -eq 0 ]; then
summary="$summary PASS $t"$'\n'
else
summary="$summary FAIL $t (exit $rc)"$'\n'; fail=1
fi
echo "::endgroup::"
done
echo "===== HDF5 consumer tests ====="
printf '%s' "$summary"
exit "$fail"
python-client:
name: Generate python client
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
- name: Build python client
shell: bash
run: bash gen_python_client.sh
- name: Upload Package to Gitea PyPI
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
run: |
twine upload -u __token__ -p ${{ secrets.CI_PYPI_TOKEN }} --skip-existing dist/*
twine upload -u __token__ -p ${{ secrets.PIP_REPOSITORY_API_TOKEN }} --repository-url https://gitea.psi.ch/api/packages/mx/pypi dist/*
documentation:
name: Build documentation
# Uploads to the release, so it waits for the job that creates it.
needs: create-release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
- name: Build documentation
shell: bash
run: bash make_doc.sh
- name: Upload documentation
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
run: |
git config --global user.email "filip.leonarski@psi.ch"
git config --global user.name "Filip Leonarski (Gitea)"
git checkout --orphan gitea-pages
git reset --hard
# Remove everything except .git and public
git clean -fdx -e public -e .git
# Move public contents to root and remove the now-empty public dir
shopt -s dotglob nullglob
mv public/* .
rm -rf public
git add .
git commit -m "Deploy site"
git push -f https://${{secrets.GITHUB_TOKEN}}@gitea.psi.ch/${{ github.repository }}.git gitea-pages
# ---- Release -------------------------------------------------------------------------------------
# Runs on every event, but only creates something when asked. Keeping it unconditional is what
# lets every other job depend on it with a plain `needs:` -- a skipped dependency would otherwise
# skip its dependents too. When it creates nothing it is just a checkout.
create-release:
name: Create release
runs-on: jfjoch_rocky8
container:
image: gitea.psi.ch/leonarski_f/jfjoch_rocky8:2609
steps:
- uses: actions/checkout@v4
with:
persist-credentials: 'true' # Optional; should be the default
# The LFS objects are pulled only to attach the FPGA images below, so this stays off the
# critical path of an ordinary push -- where this job is a bare checkout and nothing else.
- name: Configure auth and fetch LFS
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
shell: bash
env:
GITEA_TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
run: |
git lfs install --local
AUTH=$(git config --local http.${{ github.server_url }}/.extraheader)
git config --local --unset http.${{ github.server_url }}/.extraheader
git config --local http.${{ github.server_url }}/${{ github.repository }}.git/info/lfs/objects/batch.extraheader "$AUTH"
git lfs pull
- name: Create release
if: github.event_name == 'workflow_dispatch' && (github.event.inputs.create_release == 'true' || github.event.inputs.create_release == true)
env:
TOKEN: ${{ secrets.PIP_REPOSITORY_API_TOKEN }}
shell: bash
run: |
python3 gitea_create_release.py
tools/bitshuffle_bench/bs_bench.cpp
+131
View File
@@ -0,0 +1,131 @@
// Bitshuffle block-decode benchmark: classic NEON (malloc per block), classic NEON with the caller's
// scratch (S1), the NEON port of hperf, hperf itself (scalar fallback on arm64), and the selector
// rugnux uses (JFJochBitUnshuffleBlock). Optionally the full bitshuffle/LZ4 chunk decode.
//
// Build: ./build.sh /path/to/nextgendcu [clang] [clang++] (checkout of branch arm64-bitshuffle), or by hand:
// REPO=/path/to/nextgendcu
// S="$REPO/compression/bitshuffle/bitshuffle_core.c $REPO/compression/bitshuffle/iochain.c \
// $REPO/compression/bitshuffle_hperf/bitshuffle.c $REPO/compression/bitshuffle_hperf/bitshuffle_neon.c \
// $REPO/compression/lz4/lz4.c"
// for f in $S; do clang -O3 -std=gnu11 -I$REPO/compression -c $f -o $(basename $f .c).o; done
// clang++ -O3 -std=c++20 -I$REPO/compression bs_bench.cpp *.o -o bs_bench
// ./bs_bench [-m MB] [chunk.bslz4 ...] # MB of synthetic blocks (default 64); 4-byte chunks,
// # e.g. eiger2_0.bslz4 pilatus4_0.bslz4
// (on Linux use gcc/g++ the same way; the same objects also build bs_exact_test.cpp, the
// byte-identity check, which is worth running natively once: ./bs_exact_test *.bslz4)
#include <chrono>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <cstdlib>
#include <fstream>
#include <functional>
#include <string>
#include <vector>
#include <BitShuffleBlock.h>
#include <lz4/lz4.h>
extern "C" uint64_t bshuf_read_uint64_BE(const void *buf);
#if defined(__aarch64__)
#include <bitshuffle_hperf/bitshuffle_neon.h>
#endif
using Decoder = std::function<void(char *out, const char *in, char *scratch, size_t size, size_t elem)>;
static double best_seconds(const std::function<void()> &f, int reps) {
double best = 1e30;
for (int r = 0; r < reps; r++) {
auto t0 = std::chrono::steady_clock::now();
f();
best = std::min(best, std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count());
}
return best;
}
static std::vector<std::pair<std::string, Decoder>> decoders() {
std::vector<std::pair<std::string, Decoder>> d;
d.push_back({"classic bshuf_untrans_bit_elem", [](char *o, const char *i, char *, size_t n, size_t e) {
bshuf_untrans_bit_elem(i, o, n, e);
}});
d.push_back({"hperf bitshuf_decode_block", [](char *o, const char *i, char *s, size_t n, size_t e) {
bitshuf_decode_block(o, i, s, n, e);
}});
#if defined(__aarch64__)
d.push_back({"S1 classic NEON, caller scratch", [](char *o, const char *i, char *s, size_t n, size_t e) {
bshuf_trans_byte_bitrow_NEON(i, s, n, e);
bshuf_shuffle_bit_eightelem_NEON(s, o, n, e);
}});
d.push_back({"NEON port of hperf", [](char *o, const char *i, char *s, size_t n, size_t e) {
bitshuf_decode_block_neon(o, i, s, n, e);
}});
#endif
d.push_back({"JFJochBitUnshuffleBlock (shipped)", [](char *o, const char *i, char *s, size_t n, size_t e) {
JFJochBitUnshuffleBlock(o, i, s, n, e);
}});
return d;
}
// Block transform alone: 8192-byte blocks (the EIGER/rugnux default) over `total` bytes.
static void bench_blocks(size_t total) {
uint64_t x = 12345;
for (size_t elem : {1, 2, 4}) {
const size_t block = 8192 / elem;
std::vector<char> raw(total), enc(total), out(total), scratch(block * elem);
for (size_t i = 0; i < total / elem; i++) { // sparse photon counts
x = x * 6364136223846793005ULL + 1442695040888963407ULL;
const uint32_t v = (x >> 60) < 3 ? (uint32_t) ((x >> 40) % 17) : 0;
memcpy(&raw[i * elem], &v, elem);
}
for (size_t off = 0; off < total; off += block * elem)
bshuf_trans_bit_elem(&raw[off], &enc[off], block, elem);
printf("block decode, elem_size %zu, block %zu elements, %zu MB:\n", elem, block, total >> 20);
for (auto &[name, dec] : decoders()) {
memset(out.data(), 0, total);
const double t = best_seconds([&] {
for (size_t off = 0; off < total; off += block * elem)
dec(&out[off], &enc[off], scratch.data(), block, elem);
}, 5);
printf(" %-36s %8.0f MB/s %s\n", name.c_str(), total / t / 1e6,
memcmp(out.data(), raw.data(), total) == 0 ? "" : " MISMATCH");
}
}
}
// Full bitshuffle/LZ4 chunk decode (LZ4 + block transform), as JFJochDecompressHperfPtr does it.
static void bench_chunk(const std::string &path, size_t elem) {
std::ifstream f(path, std::ios::binary);
std::vector<char> src((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
if (src.size() < 12) { printf("cannot read %s\n", path.c_str()); return; }
const uint8_t *p = (const uint8_t *) src.data();
const size_t total = bshuf_read_uint64_BE(p), block = bshuf_read_uint32_BE(p + 8) / elem, nelem = total / elem;
std::vector<char> image(total), dec(block * elem), scratch(block * elem), ref;
printf("chunk %s (%zu MB decoded):\n", path.c_str(), total >> 20);
for (auto &[name, fn] : decoders()) {
const double t = best_seconds([&] {
size_t pos = 12, done = 0;
while (nelem - done >= 8) {
size_t cur = std::min(block, nelem - done);
cur -= cur % 8;
const size_t csize = bshuf_read_uint32_BE(p + pos);
pos += 4;
LZ4_decompress_safe(src.data() + pos, dec.data(), (int) csize, (int) (cur * elem));
pos += csize;
fn(&image[done * elem], dec.data(), scratch.data(), cur, elem);
done += cur;
}
memcpy(&image[done * elem], src.data() + pos, (nelem - done) * elem);
}, 5);
if (ref.empty()) ref = image;
printf(" %-36s %8.0f MB/s %s\n", name.c_str(), total / t / 1e6, image == ref ? "" : " MISMATCH");
}
}
int main(int argc, char **argv) {
int first = 1;
size_t mb = 64;
if (argc > 2 && strcmp(argv[1], "-m") == 0) { mb = strtoul(argv[2], nullptr, 10); first = 3; }
bench_blocks(mb << 20);
for (int i = first; i < argc; i++)
bench_chunk(argv[i], 4);
}
tools/bitshuffle_bench/bs_exact_test.cpp
+165
View File
@@ -0,0 +1,165 @@
// Exactness test for the bitshuffle block decoders: classic (malloc), S1 (caller scratch),
// hperf-NEON port, hperf (x86 SIMD or scalar), and the shipped JFJochBitUnshuffleBlock selector.
// Prints a hash over every encoded input and decoded output so x86 and aarch64 runs can be compared.
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <cstdlib>
#include <vector>
#include <string>
#include <fstream>
#include <BitShuffleBlock.h>
#include <lz4/lz4.h>
extern "C" uint64_t bshuf_read_uint64_BE(const void* buf);
#if defined(__aarch64__)
#include <bitshuffle_hperf/bitshuffle_neon.h>
#endif
static uint64_t rng_state = 0x1234567887654321ULL;
static uint64_t next_rand() {
uint64_t z = (rng_state += 0x9e3779b97f4a7c15ULL);
z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9ULL;
z = (z ^ (z >> 27)) * 0x94d049bb133111ebULL;
return z ^ (z >> 31);
}
static uint64_t hash = 1469598103934665603ULL;
static void hash_bytes(const char *p, size_t n) {
for (size_t i = 0; i < n; i++) { hash ^= (uint8_t) p[i]; hash *= 1099511628211ULL; }
}
static long failures = 0, checks = 0;
static void expect_same(const std::vector<char> &ref, const std::vector<char> &got, const char *what,
size_t size, size_t elem, int pattern) {
checks++;
if (memcmp(ref.data(), got.data(), ref.size()) != 0) {
if (failures < 20)
printf("MISMATCH %s size=%zu elem=%zu pattern=%d\n", what, size, elem, pattern);
failures++;
}
}
static void fill(std::vector<char> &v, size_t elem, int pattern) {
const size_t n = v.size() / elem;
for (size_t i = 0; i < v.size(); i++) {
switch (pattern) {
case 0: v[i] = (char) next_rand(); break; // random
case 1: v[i] = 0; break; // zeros
case 2: v[i] = (char) 0xff; break; // ones
case 3: v[i] = (char) ((i % elem == 0) ? (next_rand() % 7 == 0 ? next_rand() % 5 : 0) : 0); break; // sparse counts
case 4: v[i] = (char) (i & 1 ? 0xaa : 0x55); break; // alternating
case 5: v[i] = (char) (((i / elem) % (8 * elem)) / 8 == i % elem ? 1u << ((i / elem) % 8) : 0); break; // walking bit
case 6: v[i] = (char) ((i % elem == elem - 1) ? 0x80 : (char) next_rand()); break; // sign bit set + noise
}
}
(void) n;
}
static void run_block(size_t size, size_t elem, int pattern) {
std::vector<char> orig(size * elem), enc(size * elem), enc_h(size * elem), scratch(size * elem);
fill(orig, elem, pattern);
if (bshuf_trans_bit_elem(orig.data(), enc.data(), size, elem) < 0) { printf("encode failed\n"); failures++; return; }
if (bitshuf_encode_block(enc_h.data(), orig.data(), scratch.data(), size, elem) != 0) { printf("hperf encode failed\n"); failures++; return; }
expect_same(enc, enc_h, "encode classic vs hperf", size, elem, pattern);
hash_bytes(enc.data(), enc.size());
std::vector<char> out(size * elem);
std::fill(out.begin(), out.end(), 0x5a);
bshuf_untrans_bit_elem(enc.data(), out.data(), size, elem);
expect_same(orig, out, "classic bshuf_untrans_bit_elem", size, elem, pattern);
std::fill(out.begin(), out.end(), 0x5a);
bitshuf_decode_block(out.data(), enc.data(), scratch.data(), size, elem);
expect_same(orig, out, "hperf bitshuf_decode_block", size, elem, pattern);
std::fill(out.begin(), out.end(), 0x5a);
std::fill(scratch.begin(), scratch.end(), 0x33);
if (JFJochBitUnshuffleBlock(out.data(), enc.data(), scratch.data(), size, elem) < 0) failures++;
expect_same(orig, out, "JFJochBitUnshuffleBlock", size, elem, pattern);
#if defined(__aarch64__)
std::fill(out.begin(), out.end(), 0x5a);
bshuf_trans_byte_bitrow_NEON(enc.data(), scratch.data(), size, elem);
bshuf_shuffle_bit_eightelem_NEON(scratch.data(), out.data(), size, elem);
expect_same(orig, out, "S1 two-stage NEON with scratch", size, elem, pattern);
if (elem == 1 || elem == 2 || elem == 4) {
std::fill(out.begin(), out.end(), 0x5a);
if (bitshuf_decode_block_neon(out.data(), enc.data(), scratch.data(), size, elem) != 0) failures++;
expect_same(orig, out, "hperf NEON port", size, elem, pattern);
} else if (bitshuf_decode_block_neon(out.data(), enc.data(), scratch.data(), size, elem) != -1)
failures++;
#endif
hash_bytes(out.data(), out.size());
}
// A bitshuffle/LZ4 chunk as HDF5 filter 32008 stores it, decoded block by block through every path.
static void run_chunk(const std::string &path, size_t elem) {
std::ifstream f(path, std::ios::binary);
std::vector<char> src((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
const uint8_t *p = (const uint8_t *) src.data();
const size_t total = bshuf_read_uint64_BE(p);
const size_t block = bshuf_read_uint32_BE(p + 8) / elem;
const size_t nelem = total / elem;
size_t pos = 12, done = 0;
std::vector<char> image(total);
std::vector<char> dec(block * elem), scratch(block * elem), ref(block * elem), out(block * elem);
long nblocks = 0, npartial = 0;
while (nelem - done >= 8) {
size_t cur = std::min(block, nelem - done);
cur -= cur % 8;
const size_t csize = bshuf_read_uint32_BE(p + pos);
pos += 4;
const int r = LZ4_decompress_safe(src.data() + pos, dec.data(), (int) csize, (int) (cur * elem));
if (r != (int) (cur * elem)) { printf("LZ4 error %s\n", path.c_str()); failures++; return; }
pos += csize;
if (cur != block) npartial++;
nblocks++;
bshuf_untrans_bit_elem(dec.data(), ref.data(), cur, elem);
ref.resize(cur * elem); out.resize(cur * elem);
bitshuf_decode_block(out.data(), dec.data(), scratch.data(), cur, elem);
expect_same(ref, out, "chunk hperf", cur, elem, -1);
JFJochBitUnshuffleBlock(out.data(), dec.data(), scratch.data(), cur, elem);
expect_same(ref, out, "chunk JFJochBitUnshuffleBlock", cur, elem, -1);
#if defined(__aarch64__)
bshuf_trans_byte_bitrow_NEON(dec.data(), scratch.data(), cur, elem);
bshuf_shuffle_bit_eightelem_NEON(scratch.data(), out.data(), cur, elem);
expect_same(ref, out, "chunk S1", cur, elem, -1);
bitshuf_decode_block_neon(out.data(), dec.data(), scratch.data(), cur, elem);
expect_same(ref, out, "chunk NEON port", cur, elem, -1);
#endif
memcpy(image.data() + done * elem, ref.data(), cur * elem);
ref.resize(block * elem); out.resize(block * elem);
done += cur;
}
const size_t left = (nelem - done) * elem;
memcpy(image.data() + done * elem, src.data() + pos, left);
pos += left;
if (pos != src.size()) { printf("chunk not fully consumed %s\n", path.c_str()); failures++; }
uint64_t h = 1469598103934665603ULL;
for (char c : image) { h ^= (uint8_t) c; h *= 1099511628211ULL; }
printf("chunk %s: %zu elements, %ld blocks (%ld partial), %zu leftover bytes, image hash %016llx\n",
path.c_str(), nelem, nblocks, npartial, left, (unsigned long long) h);
hash_bytes(image.data(), image.size());
}
int main(int argc, char **argv) {
const size_t elems[] = {1, 2, 3, 4, 8};
std::vector<size_t> sizes;
for (size_t s = 8; s <= 1040; s += 8) sizes.push_back(s); // every row-length tail 1..130 bytes
for (size_t s : {2048, 2056, 4096, 8192, 8200, 16384, 1544, 1536, 2040})
sizes.push_back(s);
for (size_t elem : elems)
for (size_t size : sizes)
for (int pattern = 0; pattern < 7; pattern++)
run_block(size, elem, pattern);
for (int i = 1; i < argc; i++)
run_chunk(argv[i], 4);
printf("%ld checks, %ld failures, total hash %016llx\n", checks, failures, (unsigned long long) hash);
return failures ? 1 : 0;
}
tools/bitshuffle_bench/build.sh
Executable
+10
View File
@@ -0,0 +1,10 @@
#!/bin/bash
# ./build.sh <REPO> [CC] [CXX] - builds bs_bench and bs_exact_test next to this script
set -e
REPO=$1; CC=${2:-cc}; CXX=${3:-c++}
cd "$(dirname "$0")"
for f in bitshuffle/bitshuffle_core.c bitshuffle/iochain.c bitshuffle_hperf/bitshuffle.c bitshuffle_hperf/bitshuffle_neon.c lz4/lz4.c; do
$CC -O3 -std=gnu11 -I$REPO/compression -c $REPO/compression/$f -o $(basename $f .c).o
done
$CXX -O3 -std=c++20 -I$REPO/compression bs_bench.cpp *.o -o bs_bench
$CXX -O2 -std=c++20 -I$REPO/compression bs_exact_test.cpp *.o -o bs_exact_test
tools/bitshuffle_bench/eiger2_0.bslz4
BIN
View File
Binary file not shown.
tools/bitshuffle_bench/pilatus4_0.bslz4
BIN
View File
Binary file not shown.